Memory patcher conflicts with OMPI and libfabric

[rajachan]

We had ported OMPI’s patcher code to libfabric not too long ago to get notifications about memory events for its registration cache. Libfabric also has a userfaultfd notifier, but it defaults to the patcher-based “memhooks” notifier given the additional coverage it provides. Given how patcher uses jump statements to patch the various calls, we can not really stack multiple patcher hooks atop each other. When testing the OFI BTL with EFA, we observed silent data corruption with some benchmarks and root caused it to libfabric using stale registrations having failed to invalidate entries after OMPI’s patcher taking over the hooks.

UCX has a mechanism to take in external events from applications, ucm_set_external_event(). OMPI uses this mechanism in the UCX PML to invoke ucm_vm_munmap() from a unmap callback based on its internal memory hook events. We can achieve a similar workflow with libfabric's FI_MR_MMU_NOTIFY mode. While this mode was designed with a different use-case in mind (allowing registrations that are not backed by physical pages), we should be able to use it in conjunction with fi_mr_refresh() to take external events from applications like OMPI. To be specific, libfabric providers can make FI_MR_MMU_NOTIFY a soft requirement. If an application does not support it, they can continue using memhooks as is. If an application like OMPI does support that mode, providers can rely on fi_mr_refresh() notifications in place of the internal monitor (or perhaps in addition to userfaultfd) to determine when to evict an entry from the cache.

On the Open MPI side, the provider query logic will have to set FI_MR_MMU_NOTIFY mode and call fi_mr_refresh() on unmap events. Rcache uses the OPAL memory hooks, so it registers a callback to react to memory events (mca_rcache_base_mem_cb). For what the OFI components need, we can register another OFI-specific callback from common OFI code used both by the BTL and the MTL (which will eventually want to use a cache for CUDA buffers given the cost of querying CUDA buffer attributes). This OFI-specific callback can then directly pass on the notification to libfabric via fi_mr_refresh(). This will all be provider-agnostic.

@open-mpi/ofi @hppritcha @shefty, thoughts?

[shefty]

Conceptually, I think this could work, and I like the proposed flow.

The problem I see is fi_mr_refresh() takes a mr as input, which requires that the application register the region.

[wzamazon]

When MR cache is used, we can find the MR from cache?

[shefty]

The app needs to pass the MR into fi_mr_refresh(). To do that it needs to initiate the registration call.

The MR is only really needed here in order to route to the correct control() function. We could call the domain->control() operation instead, but that would be through fi_control() directly. Plus that behavior does not relate directly with FI_MR_MMU_NOTIFY as it's currently defined.

We're really wanting the app to hook into the memory monitor.

[wzamazon]

It seems we can have a new fi_control command on an fi_domain object, and each provider can choose how or whether to implement it. Does it look right?

[shefty]

It's more than that. We need a way to for the memhooks to be disabled for all providers. Imaging if libfabric is being used by 2 (or more) different middleware libraries. The hooks can only be installed once. It can't be set per domain or provider or fabric. And even the middleware might need to coordinate this with each other.

We can use an environment variable to force it off. That's okay. But we should also be able to disable it when it's not needed. The question is how to convey that it's not needed. That's the FI_MR_MMU_NOTIFY path Raghu was discussing. On the surface, it sounded good to me, but thinking about it more, I don't think it works.

Something needs to tell libfabric that it doesn't need to perform this task, and that some sort of callback will be invoked for it.

There have been several times recently when it would be convenient if there were something like a fid_core or fid_ofi or fid_global. A singular fabric object that's not associated with any provider that can be used to invoke some functionality at the libfabric core.

[rajachan]

Wei had raised the point about fids when I discussed the proposal with him. I agree, having a global fid would be useful for things like this. IIRC, we'd discussed using fi_control() in the past for core component controls (like disabling/enabling the cache); which would also need a special fid of some sort.

[awlauria]

@bwbarrett @hppritcha this looks like a blocker for v5.0.0. Agree?

[shefty]

libfabric added the ability for ompi to import its memory patcher support through libfabric commit 7759f8c. There's no guarantee that the code will work as-is, since there's no implementation that uses it yet. But in theory the updates needed on the ompi side shouldn't be difficult.

[gpaulsen]

Is anyone looking at this?

[bwbarrett]

AWS is suppose to be looking at this, but we're a bit behind.

[wckzhang]

Going to start working on this