This is probably due to differences in the size of the individual directories within the two directory trees. Each directory is represented on disk by some disk blocks which store the names of the files in the directory. These blocks are sometimes called the "directory file". When you run "ls -l" or "ls -s", the size field for a directory is the size of this directory file.
When you create a file within a directory, the file's name is added to the directory file. If the directory file doesn't contain enough space, it might have to be enlarged by adding another disk block .When you delete a file from a directory, the file's name is removed from the directory file leaving some unused space. The unused space can be used to store another filename. But it isn't returned to the OS, so the directory file can grow in size but it never shrinks.
If you were to compare the directory trees side-by-side, you'd probably find that some of the matching directories were different sizes. That generally means that one directory used to have more files in it, or files with bigger names, or had a lot of file creation/deletion "churn" in the past.
When diff compares two directories, it only checks whether both directories have the same names in them. It doesn't check the unused space within the respective directory files, and it doesn't care if the two directory files are actually the same size.