This post is in response to Gabe’s recent post:
and then Frank Denneman’s reply here: Re: Impact of Large Pages on Consolidation ratios
Finished reading? Ok, let’s continue…
Let me try to summarize very briefly what happens. When Large Pages are not considered, Transparent Page Sharing (TPS) runs across a host’s VMs periodically (every 60 minutes by default) and reclaims identical 4KB pages – think of it like de-duping your memory. When using Large Pages (2MB), TPS does not try to reclaim those identical pages because of the “cost” of comparing these much larger pages. That is until there is memory contention, at which point the Large Pages are broken down into 4KB blocks and identical pages are shared.
Large Pages have been proven to offer performance improvements. The strategy that VMware are following is technically advantageous – why incur the expense of figuring out which Large Pages are identical if the host doesn’t need to reclaim memory. If it can back all VM memory requests with physical memory, then it doesn’t need to worry about reclamation yet.
However, the problem I see is that of perception. If you run lots of VMs with very similar memory usage on a host, you would expect to see the advantage of TPS kicking in and suitable memory savings. However if the Large Pages don’t get shared until the host thinks it is under pressure, you won’t see those savings until there appears to be a problem. The host will wait until it hits 94% memory usage (6% free), before it deems itself under memory contention and starts to break those Large Pages into smaller 4KB ones. So in a VM environment with very similar VMs, you are consistently going to run your host at around 94% memory used. This isn’t a technical issue. All those identical memory pages can still be reclaimed, just as before, and you are gaining a performance gain of Large Pages. This is a perception issue.
Most vSphere administrators probably don’t realize this, their manager almost certainly don’t realize this. All they see is their hosts running out of memory – they have less than 10% memory free! Time for some more hosts. And even for those that understand that when they get to this level there is the potential that there is still more potential memory available, it’s not that apparent how much saving they could expect and how far to push it. Do we hide all memory usage in the vSphere client until it hits only 4% or 2% free? Obviously not. I think this is a measurement issue.