Permit MMIO exits to bypass the emulation.

[cracyc]

It appears to be impossible for a program to handle a mmio exit in userspace rather than round tripping to kernel every time. If the registers are reloaded after HAX_EXIT_FAST_MMIO they will likely then be corrupted in em_emulate_insn. I also tried to use HAX_EPT_PERM_NONE but it seems to not support small memory regions.

[raphaelning]

I guess what you want is something like the macOS Hypervisor.framework API, which allows user space to access most VMCS registers, so the QEMU HVF accelerator (which runs in user space) can fetch the MMIO instruction, emulate it, and then resume the guest from the next instruction.

The HAXM API provides a higher-level abstraction by design, which is why HAX_EXIT_FAST_MMIO isn't suitable for your purpose (it assumes most of the instruction emulation work is done in the kernel).

HAX_EXIT_PAGEFAULT (assuming the MMIO region can be protected with HAX_RAM_PERM_NONE) might be a better alternative. However, right now the GPA protection mechanism is limited to non-MMIO regions, and even if it worked on MMIO regions, you would still need to make additional "round trips" to the kernel:

1. One for reading the vCPU register state to get the current instruction pointer (RIP), before you can fetch the MMIO instruction.
2. One for writing the vCPU register state to make RIP point to the next instruction, after you emulate the MMIO instruction.

With the HAXM API, you can't read/write the guest RIP alone, but have to sync a larger set of vCPU registers (again that's by design), so these round trips will be very costly.

In fact, if the instruction only accesses one MMIO address (which is the most common case), I think the HAX_EXIT_FAST_MMIO approach already guarantees the minimum number of kernel/user context switches (kernel => user => kernel), with pretty low overhead.

[cracyc]

To be clear I'm trying to emulate VGA which requires an MMIO region 64K in size. HAX_EXIT_FAST_MMIO requires a round trip for every vram read and write and there'll be thousands while a frame is drawn then none for a long period. Try a vga program that uses unchained mode such as doom in qemu with haxm and you'll see the performance is horrid, far slower than emulation so I was trying to do it like dosemu which exit kvm at the first vram access and emulates until it appears the program is finshed. As you say HAX_EXIT_PAGEFAULT would likely be the best solution but it appears that it has a 2MB granularity which is far to large for the 64k vga vram window and will put the entire vm address space within 2MB when running in real mode.

[raphaelning]

[...] so I was trying to do it like dosemu which exit kvm at the first vram access and emulates until it appears the program is finshed.

I see, so you don't need to sync vCPU state very often. Is "kvm" a typo? I wonder if dosemu uses KVM at all, since it predates KVM. And I'm curious whether KVM API (KVM_EXIT_MMIO, etc.) meets your requirements, since it's similar to HAXM.

As you say HAX_EXIT_PAGEFAULT would likely be the best solution but it appears that it has a 2MB granularity which is far to large for the 64k vga vram window and will put the entire vm address space within 2MB when running in real mode.

True. The 2MB granularity stems from the fact that HAXM divides each RAM block (host buffer backing guest memory) into 2MB chunks:

haxm/core/include/memory.h

Lines 37 to 38 in 0d3922d

	#define HAX_CHUNK_SHIFT 21
	#define HAX_CHUNK_SIZE (1U << HAX_CHUNK_SHIFT) // 2MB

Theoretically, it's possible to switch to 4KB chunks, but last time we tried that, we ran into some stability issue. Maybe you could try to make it work?

[cracyc]

Is "kvm" a typo? I wonder if dosemu uses KVM at all, since it predates KVM. And I'm curious whether KVM API (KVM_EXIT_MMIO, etc.) meets your requirements, since it's similar to HAXM.

Dosemu2 supports KVM but the original dosemu presumably had the same problem but with v86 mode instead. I think dosemu2 uses v86 mode in KVM for compatibility with the original dosemu and then they use regular page faults to trap vga access. That makes it basically impossible to support any programs which need their own paging.

Maybe you could try to make it work?

Well, I can take a stab at it I suppose.

[krytarowski]

@cracyc I wanted to port/update dosemu support for NetBSD some time ago but I've faced rather very legacy code that needed modernization in regards of used interfaces (e.g. switch to mcontext from sigcontext). We have also dropped v86 support from the NetBSD kernel and dosemu was using it.

Was this situation changed? Is dosemu(2) switching to HAXM now? KVM restricts users to Linux while HAXM works now on 4 major Operating Systems (including NetBSD). Unfortunately I had to resign from my porting efforts previously as it demanded too many generic improvements beyond adding compatibility code.

@raphaelning I got a report from @polprog that OpenBSD in HAXM is terribly slow.. is this VGA related?

[cracyc]

Not that I know of, what I'm working on is a bit different.

[raphaelning]

@raphaelning I got a report from @polprog that OpenBSD in HAXM is terribly slow.. is this VGA related?

I have no idea. Other desktop OSes seem to run smoothly. Since VGA rendering is probably done with MMIO, you may be able to identify the bottleneck by observing exits to QEMU of type HAX_EXIT_FAST_MMIO.

[krytarowski]

@raphaelning NetBSD with X Window is also unusable due to slowness... I will file a dedicated PR for it in future once all the booting issues will be solved.

[cracyc]

If it's using basic vga mode (4 plane/16 color) it's (at least from my experimentation) going to be very slow. It appears the svga cirrus emulation in linear framebuffer mode doesn't mmio exit for vram writes which would be far faster.

[polprog]

From what I have noticed, the apparent VGA slowness is because of general emulation slowness in case of having extremely verbose logging enabled. On NetBSD, caused by setting the following debug level variable to 0 (That is not the default value!).

int default_hax_log_level = 0; in platforms/netbsd/hax_wrapper.c L47

[krytarowski]

From what I have noticed, the apparent VGA slowness is because of general emulation slowness in case of having extremely verbose logging enabled. On NetBSD, caused by setting the following debug level variable to 0 (That is not the default value!).

int default_hax_log_level = 0; in platforms/netbsd/hax_wrapper.c L47

Are you sure that performance bottlenecks disappear if we set default_hax_log_level to 0? I don't see many messages in dmesg(8) myself..

[krytarowski]

OK, @polprog explained off-list that if we add extra logging performance is reduced.. but reverting to HEAD it's back to the current state. Unfortunately this doesn't solve our primary issue with MMIO bottleneck.

[polprog]

ad "off-list explanation":
Setting the log level to 0 makes it print every message, including hax_debug() calls - those printf-s are blocking so they stall the whole VM for the time they print the message, and there's several of them per VM exit, so they slow things down.

To add a little context - when I was debugging, I would change that value back and forth as I needed to see more or less info in dmesg(8) and the VM would run either slow or fast.

My test box is not the fastest, and with loglevel set to zero (most verbose) I could literally see QEMU's BIOS print the messages line-by-line as if it was a 9600bps terminal! And that also caused longer kernel load times - just as if the emulated CPU clock was a couple times slower. - This is not a bug, since implementing those debug messages in a non-blocking way would be just plain over-engineering and they are suspressed in most cases anyway.

This is a proof that it's not MMIO related, I just feel that it should be mentioned :)

[krytarowski]

Do you mean that your report of slowness of OpenBSD was caused by debug level and not MMIO bottleneck?

[polprog]

Yes, exactly.