Support dynamic allocation of sys_{mutex,sem}

[stephanosio]

Is your enhancement proposal related to a problem? Please describe.
The sys_mutex and sys_sem mechanisms, unlike the kernel equivalent (k_mutex and k_sem), do not support run-time dynamic allocation, and must be statically allocated at the build time to work properly.

These mechanisms exist to provide abstraction for the synchronisation primitives such that their performance impact can be minimised (i.e. it internally uses k_mutex and k_sem when CONFIG_USERSPACE=n, and atomic integer and k_futex when CONFIG_USERSPACE=y).

Being unable to dynamically allocate these locks imposes severe restrictions for their practical use and forces applications to use k_mutex and k_sem instead: for instance, any OOP class-like dynamically-instantiable primitives containing synchronisation objects (e.g. dynamically instantiated MQTT client containing a mutex) will have to use k_mutex and k_sem with k_object_alloc instead of using sys_mutex and sys_sem.

This leads to severe performance impact and defeats the purpose of having the sys_mutex and sys_sem abstractions in the first place.

Describe the solution you'd like

Background information
In theory, both sys_mutex and sys_sem should be using k_futex; but, due to the limitations of the current Zephyr kernel implementation, only sys_sem does.

Prior to implementing one of the following solutions, the sys_mutex should be updated to use k_futex first (corresponds to the issue #15138). The following solutions assume that sys_mutex has been updated to use k_futex.

Solution 1 (ideal)

• Rework the k_futex mechanism such that struct z_futex_data resides in user-accessible memory location (arbitrary futex pointers #25497).
• Embed struct z_futex_data into struct sys_mutex and struct sys_sem.

Solution 2 (if we don't make progress on #25497)

• Keep kernel-side struct z_futex_data (i.e. assuming arbitrary futex pointers #25497 is not implemented).
• Make struct z_futex_data kernel object dynamically allocable.
• Add a kernel API to link a dynamically allocated kernel object (i.e. z_futex_data) to a dynamically allocated user mode object (i.e. k_futex).

DISCLAIMER: The "solutions" suggested above are meant serve as only general guideline for resolving this issue. Some very in-depth details may not be fully correct.

Additional context
See #36201 (comment) for the analysis on the status quo.

[stephanosio]

cc @dcpleung @andyross @ioannisg @carlescufi

[andyross]

It might be a good idea to fold this in with a lot of other requirements/issues around the synchronization primitives. It's been mentioned in a few places but really what I think we need to do is condense all the gadgets to a shared backend. Locks within the same memory domain should always become atomic_cas operations. The kernel should take a syscall/API entry only on contention. Priority inheritance vs. semaphore counting should be a runtime/backend decision (and only on contention). Spinning (and SMP relaxation!) should probably be part of the same framework, etc...

[zephyrbot]

Hi @andyross, @dcpleung,

This issue, marked as an Enhancement, was opened a while ago and did not get any traction. Please confirm the issue is correctly assigned and re-assign it otherwise.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

@stephanosio you are also encouraged to help moving this issue forward by providing additional information and confirming this request/issue is still relevant to you.

Thanks!