LTTNG-UST 2.13 changes soname breaking loading of libcoreclrtraceptprovider

[hoyosjs]

There's an ABI breaking change between LTTNG-ust 2.12 and 2.13, so they've reved their soname from liblttng-ust.so.0 to liblttng-ust.so.1. We link against the former. Given some distros will start supporting newer versions, customers will likely end up upgrading packages and realize they can't load libcoreclrtraceptprovider.so. We should investigate avenues to support multiple versions as needed using compilation tricks like the ones used in System.Security for openssl. Other possibility is to only support 2.13 and that brings us come code sanity advantages - this would only happen in .NET 7+ realistically.

@brianrob @dotnet/dotnet-diag

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

There's an ABI breaking change between LTTNG-ust 2.12 and 2.13, so they've reved their soname from liblttng-ust.so.0 to liblttng-ust.so.1. We link against the former. Given some distros will start supporting newer versions, customers will likely end up upgrading packages and realize they can't load libcoreclrtraceptprovider.so. We should investigate avenues to support multiple versions as needed using compilation tricks like the ones used in System.Security for openssl. Other possibility is to only support 2.13 and that brings us come code sanity advantages - this would only happen in .NET 7+ realistically.

@brianrob @dotnet/dotnet-diag

Author:	hoyosjs
Assignees:	-
Labels:	area-Diagnostics-coreclr
Milestone:	-

[am11]

using compilation tricks like the ones used in System.Security for openssl

Note, same (shim) tricks are used for libnuma.so, which is closer to where coreclrtraceptprovider lives. 🙂

[hoyosjs]

Thanks. The shim approach is tempting. We sort of codegen the trace points, so this one will be interesting if we decide to support both.

[am11]

After some digging, it turned out lttng/tracepoint.h header (the one we use) does things quite differently than libssl, libnuma ones. It differs in the following manner:

• We use TACEPOINT_* macros which (during the macro expansion) binds to __tracepoints__init method and the related code chunks.
  • We stay consistent and do not directly use any non-macro APIs.
• The __tracepoints__init itself uses dlopen(3) and uses the ABI-versioned flavor liblttng-ust.so.0, liblttng-ust.so.1 (For 2.13).
  • They even have explicit validation checks in 2.13 to ensure that the current process does not have so.0 loaded. e.g. https://github.com/lttng/lttng-ust/blob/d2a010d/src/lib/lttng-ust-tracepoint/tracepoint.c#L611-L642

Here is the expansion of macros for our generated tpdotnetruntimemonoprofiler.h:

# on Ubuntu x64
$ ./build.sh clr
$ clang-9 -E \
    artifacts/obj/coreclr/Linux.x64.Debug/pal/src/eventprovider/lttngprovider/lttng/tpdotnetruntimemonoprofiler.h \
    -Isrc/coreclr/pal/inc/rt -Isrc/coreclr/pal/inc > dump

the dump looks like this: http://sprunge.us/J5rsv3 (and file tpdotnetruntimemonoprofiler.h: http://sprunge.us/LPmIuJ). Search for __tracepoints__init or dlopen("liblttng. It is quite noisy, so I made a copy, removed #include "palrt.h" etc. and kept only one probe, here is the shorter version of this expansion: http://sprunge.us/BK0q1V, at the very end of this file is the code of interest.

I will defer to @janvorli as to the best approach for lttngshim. One option is that we install both versions on the official build machine, and compile both v0 and v1 implementations in our code with our pseudo init; which will probe for .so.1, and if not found, .so.0, then select the implementation based on the result. This approach will prevent us from stamping out symbols of liblttng-ust{-tracepoint}.so and we will still be using their init mechanism (which seem to take care of "shared access" scenarios).

[janvorli]

@am11 thank you for the analysis! I'll take a look into the possibilities of shimming lttng and consider your suggestion.

[am11]

Thanks. I actually found about dlopen/dlsym pattern in their code when I commented out these lines (to see what breaks)

runtime/src/coreclr/pal/src/eventprovider/lttngprovider/CMakeLists.txt

Lines 63 to 68 in 57bfe47

	find_library(LTTNG NAMES lttng-ust)
	target_link_libraries(coreclrtraceptprovider
	${LTTNG}
	)

just to find out it had no effect on the build result (still succeeds) other than we (unnecessarily) link to liblttng-ust{-tracepoint}.so.0. 🙂 All public APIs in their header are already shim-like macro (without multi-version support):

• https://lttng.org/man/3/lttng-ust/v2.12/
• https://lttng.org/man/3/lttng-ust/v2.13/

[janvorli]

It seems that we will need to create and distribute two flavors of the libcoreclrtraceptprovider.so, one compiled with one version of lttng and the other with the other one. Then we would try to load one and if we fail, try to load the other. It seems that's essentially what @am11 has suggested, right @am11?

[hoyosjs]

Yes, and that also seems to be what needs to happen from my understanding.

[am11]

@janvorli, yes, that's what I was thinking. It will be cleaner, as it will save us from using macro post-expansion calls. The quickest way to get there is to duplicate python code (as macro names are changed in 2.13).

Part II - python free implementation

However, in a separate project, we can replace the python scripts with lttngshim which will dynamically define those macros (at build time) and we can expand it without needing to build code. It is more work as python scripts read the XML (ETL manifest) file and generates multiple header and code files. If we think replacing python is a good idea, then I was imagining for:

• input: we define a static header with our events and groups and keep it in sync with src/coreclr/vm/ClrEtwAll.man (manually at first, as it's only a few times a year we add/modify events; if we have more time, we can use XmlPeek msbuild task to update the header files when msbuild is present .. because cmake doesn't support xml parsing)
• output, instead of creating too many header and code files (this is half of what we produce: https://gist.github.com/am11/404d99384eaae7259bcabdd06ff5e307), we can make one and ifdef out the flavors: coreclr, mono, rundom, private and stress.