Here at Bitcube we use KVM as our preferred virtualisation platform. It has the best Linux support and is the standard for all Linux distributions. Whilst it still shows a bit of immaturity at times, there is an upside - rapid development and improvements.

The latest of these is KSM or Kernel Samepage Merging. KSM "merges" identical chunks of memory which frees up real physical memory which in turn improves performance (due to more caching and less swapping) or allows you to run more virtual machines on a given host. Naturally the effectiveness of this merging depends upon what your VMs are doing and how similar their codebases are.

This is how we set this up on our Debian testbed:
* Linux 2.6.33 from Debian experimental
* libvirt-0.7.6 from Debian Lenny backports
* qemu-kvm 0.12.3 from Michael Tokarev

These are bleeding edge versions - you need a kernel with "CONFIG_KSM" set (check /boot/config*) and also a KVM with KSM support. Note that if you compile your own KVM you'll need to define madv-mergeable yourself on older Linux systems (Debian Lenny for example).

On older kernels you may need to set /sys/kernel/mm/ksm/max_kernel_pages, however on newer kernels all you need to do is to run:

echo 1 > /sys/kernel/mm/ksm/run


Now if you start running some programs (i.e. KVM) which use mergeable memory, you'll see /sys/kernel/mm/pages_shared start to go up. Note that you don't have to start ksm running prior to starting KVM, however the KVM does need KSM support (obviously).

On our little testbed (5 VMs - two firewalls, a webserver, a database/email server and a monitoring server) we get these numbers from /sys/kernel/mm/ksm.

full_scans: 1458
pages_shared: 50784
pages_sharing: 56894
pages_to_scan: 100
pages_unshared: 353582
pages_volatile: 18829
run: 1
sleep_millisecs: 20


See this page for details of the numbers (the website is currently unresponsive). We are saving 56894 pages, at 4KB per page - 222MB or 11% of the used memory (excluding buffers and cache). For reference, "free" on the testbed currently shows:

total used free shared buffers cached
Mem: 4054924 2897912 1157012 0 166152 802264
-/+ buffers/cache: 1929496 2125428
Swap: 2203640 0 2203640