Experienced this little gem when we upgraded from VMware 4.0 to 5.5. We had a few servers with some custom memory settings in the 4.0 environment. When we migrated we suddenly had issues with the memory use on these servers shooting up to 100%, and memory being paged causing performance issues. Needless to say this was not a good situation.
The issue is caused by the Balloon driver when there is an inconsistency between the VM's allocated memory and the reservation settings, and in our case didn't show up until we upgraded from 4.0 to 5.5.
The solution to clean up this issue is to disable the balloon driver, which can be accomplished via the vSphere client or in the Windows registry of the VM. Both methods to disable the driver are described here.