haxm much slower than kvm?

[mifritscher]

PC with haxm: i7 6820HQ, 16 GB RAM, NVMe SSD - Windows 7, haxm 7.1, qemu 2.11 - 64 bit
PC with kvm: i7-5500U, 16 GB RAM, SATA SSH - Ubuntu 18.04, kernel 4.15, qemu 2.11 - 64 bit

A simple Ubuntu 16.04 installation lasts 16 minutes on the haxm-PC, but only 7 minutes on the kvm-PC...
Is there a technical reason for that or more a bug / inefficiency somewhere?

[mifritscher]

Also I noticed that haxm is much slower than VMWare on the 6820HQ machine. At least in the use case "let 7 java8 programs run with light network activity themself and to outside with Ubuntu 16.04 - without any graphics, not even a framebuffer, only strict textmode". On VMware, there is about 10% cpu utlisation (if not less), while on haxm I've 60-90% cpu utilization on the 2 virtual cores.

[AlexAltea]

Recent patches have changed the instruction emulator (#42) and reduced the number of feature checks while load/saving states while entering/exiting the VM (#63). Although the emulator won't probably improve performance, the feature check patches most certainly will.

Out of sheer curiosity: If you still have the same hardware around, and building+installing the drivers is easy for you, could you rerun the tests? Just to see if we are below 16 minutes now.

---

@raphaelning We could really use some set of microkernels to do benchmarks of HAXM. I assume the sources of performance penalties are:

• VMX exit handlers.
• QEMU exit handlers.
• Context switches: both VM-HAXM switches and HAXM-QEMU switches.

These three are always triggered due to a VM-exit event. So we could have a set of microkernels, each repeatedly triggering a VM-exit with a specific exit reasons, and comparing average execution time against other hypervisors.

[mifritscher]

Sadly I've no suitable build environment :-( But I've still access to the same hardware, so if you give me the binaries I could test them.

[wcwang]

Thanks for your reply. The driver binary based on the latest code has been provided as below. You may follow the steps to test the performance.

1. Download HAXM driver and HaxmLoader, then extract them in a folder;
2. Run 'Command Prompt (Admin)' on Windows 7 64-bit and execute following commands:

   sc stop IntelHaxm
   HaxmLoader.exe -u
   HaxmLoader.exe -i IntelHaxm.sys

[raphaelning]

@mifritscher We really appreciate your time to run these tests. Note that the test driver installed by HaxmLoader.exe is "temporary" and will disappear when you reboot the machine. You can also manually unload it:

HaxmLoader.exe -u

and then re-enable the original/official driver:

sc start intelhaxm

@AlexAltea Thanks for the inspiration. I just had a look at the kvm-unit-tests project and found this:

https://github.com/rhdrjones/kvm-unit-tests/blob/master/x86/vmexit.c

It seems to implement the same idea, except that the same microkernel image can be used to test all VM exit reasons. Although we can't reuse the code, we can definitely learn a lot from its framework, which also enables writing other microkernel-based unit tests that can be run in QEMU, e.g.:

https://github.com/rhdrjones/kvm-unit-tests/blob/master/x86/emulator.c

[mifritscher]

Ok, I exectued the testcase "install 16.04 - in a fully automated environment (via preseed)".
haxm 7.1 on the i7 6820HQ machine: 12:25:13 until 12:37:36 (12:23)
new test version on the same machine: 12:42:00 until 12:53:18 (11:18)

Big problem: The system seemed to work (loaded the preseed.conf, exited after reasonable time, the size of the image and the extracted files are ok as well), but no graphical output after "loading from rom".

Command line:

qemu -m 1024 -smp 2 -rtc base=utc -drive file=temp.vmdk,if=virtio,detect-zeroes=unmap,discard=unmap -drive file=fat:rw:fat-type=32:label=kernel:output.dir,format=raw,if=virtio -vga std virtio-net,netdev=natted -netdev user,id=natted -kernel linux_initrd/linux -initrd linux_initrd/initrd.gz -append "nofb fb=false pti=off interface=auto auto=true priority=critical preseed/url=http://10.0.2.2:8000/preseed.cfg" -nodefaults

[AlexAltea]

haxm 7.1 on the i7 6820HQ machine: 12:25:13 until 12:37:36 (12:23)
new test version on the same machine: 12:42:00 until 12:53:18 (11:18)

Great to see a near 10% performance improvement. :-)

Big problem: The system seemed to work (loaded the preseed.conf, exited after reasonable time, the size of the image and the extracted files are ok as well), but no graphical output after "loading from rom".

That's bad news. I'll look into it later as there's a non-zero chance that my emulator might have broken something: Just recently I found a movzx bug, I'll submit a patch soon enough.

EDIT: Patch is available at #67.

---

@raphaelning Oh, I totally forgot about kvm-unit-tests, I'll definitely run some benchmarking tests with it. Thank you!

[mifritscher]

If you update the binaries I would be keen on retesting ;-)

[mifritscher]

Some fresh data (Setup of Ubuntu 18.04, additonal cleanup, additional creation of a squashfs-image):
kvm on the Broadwell machine: 13:46 <- squashfs alone took 3:54
haxm (7.3.2) on the Skylake machine: 17:04 <-squashfs alone took 3:43

So it seems haxm got faster over the last months :-)

[raphaelning]

Thanks, that's good to know! We (@junxiaoc for the most part) have identified some inefficient VMCS read/write logic that slows down the VM exit code path. After we fix that, hopefully you'll see even better performance data.

[mifritscher]

Using Qemu 4.0.0 and haxm 7.5.1, it took 19:38, squashfs took 7:47. Note: This was done with only 1 guest CPU, as SMP 2 crashes early (see #205).