FEMU Best Practice

Some tips to run FEMU for more stable performance:

Pinning to avoid performance unpredictability:

• Set host CPU to "performance" mode: use the set_cpu_perf_mode.sh script under femu-scripts for this purpose. Furthermore, it would be even better to run host CPUs at max/fixed frequency by disabling C/P-states.

• Pin vCPUs to physical CPUs (1 vCPU maps to 1 physical CPU). You can do this using the pin.sh script under femu-scripts to do this.

• Pin FEMU polling threads: TOADD

• Utilize huge pages as FEMU memory back-end: ToADD

• Eliminate all vMMIO operations:

Additional (Optional) Tweaks

1. Disable doorbell writes in your guest Linux NVMe driver:

**Note: Linux kernel version less than 4.14 has a wrong implementation over the doorbell buffer config support bit. (Fixed in this commit: 223694b9ae8bfba99f3528d49d07a740af6ff95a). FEMU has been updated to fix this problem accordingly. Thus, in order for FEMU polling to work properly out of box, please use guest Linux >= 4.14.

Otherwise, if you want to stick to 4.12/4.13, please make sure NVME_OACS_DBBUF = 1 << 7 in hw/block/nvme.h as this is what was wrongly implemented in 4.12/4.13**

In Linux 4.14 source code, file drivers/nvme/host/pcie.c, around line 293, you will find below function which is used to indicate whether to perform doorbell write operations.

What we need to do is to add one sentence (return false;) after *dbbuf_db = value;, as shown in the code block below.

After this, recompile your guest Linux kernel.

/* Update dbbuf and return true if an MMIO is required */
static bool nvme_dbbuf_update_and_check_event(u16 value, u32 *dbbuf_db,
                                              volatile u32 *dbbuf_ei)
{
        if (dbbuf_db) {
                u16 old_value;

                /*
                 * Ensure that the queue is written before updating
                 * the doorbell in memory
                 */
                wmb();

                old_value = *dbbuf_db;
                *dbbuf_db = value;

                /* Disable Doorbell Writes for FEMU: We only need to 
                 * add the following statement */
                return false;
                /* End FEMU modification for NVMe driver */

                if (!nvme_dbbuf_need_event(*dbbuf_ei, value, old_value))
                        return false;
        }

        return true;
}