When refreshing group policy, the list of GPOs on the server is compared against a list of applied GPOs on the local machine. This list includes the GUID, version number and path. By comparing this information, it can be determined which policies have changed and need to be reapplied. It cannot determine what has changed within the policy. Therefor the entire policy will have to be downloaded, cached locally and applied again.
The size of the GPO is not determined by the number of settings configured in the GPO, but the number of administrative templates. This is not as much of an issue with the new ADMX/ADML format and the use of a central store. However the number of settings that need to be applied, and the number of conflicting settings with other policies can affect startup and logon times. I personally prefer multiple smaller GPOs as this tends to make troubleshooting easier. I do not find that having multiple smaller GPOs affects performance.