ROOT loads unneeded PCMs

[ktf]

Check duplicate issues.

• Checked for duplicates

Describe the bug

When doing a simple TFile::Open(), ROOT initialises the interpreter on the first TEnv::GetValue needed by the aforementioned method. This makes it load all the pcms which it can find, regardless of weather they are needed to read the file or not (in our case we only have branches with basic types, e.g. floats or ints or vector of thereof).
This has the net result of adding 70MB of RSS for no particular reason to any process which uses ROOT. You can find more details in the attached instruments screenshot.
Moreover, single stepping with the debugger and printing out the LoadModule argument, it seems that the pcms for Sofie, Proof and 3D graphics are particularly honours, all of which we do not need and we do not link.

What is the expected behaviour?

The best would be for ROOT to be smart enough to lazily load the pcms when they are actually needed, not upfront. A second best option would be to be able to start the interpreter in a clean slate state and load by hand only the pcms which are needed to read a file.

How to reproduce?

On macOS using Instruments one can do:

xcrun xctrace record --output root.trace --time-limit 30s --template Allocations --launch -- /Users/ktf/src/sw/osx_arm64/ROOT/alice-v6-28-04-local2/bin/root.exe -q -b http://alimonitor.cern.ch/train-workdir/testdata/LFN/alice/data/2015/LHC15o/000246392/pass2/PWGZZ/Run3_Conversion/320_20220701-0813/AOD/019/AO2D.root
open root.trace

and get the report in question. Using the alice environment should not matter, but in principle you should be able to load it from CVMFS.

ROOT version

6.28.04, but it's a while we see it.

How did you install ROOT?

ALICE build environment

Which operating system are you using?

macOS, linux

Additional context

No response

[Axel-Naumann]

Info from @ktf : with 100 processes linked against ROOT running on these machines that's an extra 70MB*100=7GB used by (unneeded?) PCMs.

[vgvassilev]

The best would be for ROOT to be smart enough to lazily load the pcms when they are actually needed, not upfront.

Yes, and unfortunately that is as good as it gets with the current infrastructure. Here I've put a bit more details about what could be improved to further reduce the memory footprint: cms-sw/cmssw#41810 (comment)

Note that the loading of the pcm files is mmap-ed and is a no-op. The problem is that some sections of the PCMs are not lazy and they are eagerly deserialized. Using a PCH won't make the things much better since the PCH still deserializes eagerly some sections. That is, the deficiency is not in ROOT but in Clang itself.

Could you rerun using export ROOTDEBUG=7 and attach the output. Sometimes we lookup an identifier like I or something which is not found and it triggers deserialization of entities. That sometimes could be avoided.

[ktf]

I will provide you the output for ROOTDEBUG=7.

Could you elaborate a bit on the mmap? I would expect that using MAP_SHARED would be enough of a workaround for our (ALICE) use case, because what is really killing us is the fact the memory is not shared among the many processes we have. Any reason why that cannot be done? Did I misunderstand something?

[vgvassilev]

I will provide you the output for ROOTDEBUG=7.

Could you elaborate a bit on the mmap?

The pcm files are "mmapped" when being opened. That practically uses some virtual memory with the promise it won't increase your rss, unless something is needed from that file. Here is how this is achieved:

root/interpreter/llvm/src/tools/clang/lib/Serialization/ModuleManager.cpp

Line 216 in 6eb7f64

Buf = FileMgr.getBufferForFile(NewModule->File,

I believe your issue is that due to some identifier lookup we start loading pcm files which have sections that require eager deserialization where the mmap manifests into a real rss increase.

I would expect that using MAP_SHARED would be enough of a workaround for our (ALICE) use case, because what is really killing us is the fact the memory is not shared among the many processes we have. Any reason why that cannot be done? Did I misunderstand something?

The reason is that the serialization in Clang has deficiencies and reads from disk when a pcm file is loaded. I've been hunting down these cases and sometimes we could avoid them. That's why I was looking for some output that could help us do that.

That being said, we could make some effort to split the startup phase of ROOT into loading and initialization. Then we could move the registration of pcm files and setting up the ROOT's runtime as part of the initialization process but realistically, if you use ROOT for anything you'd probably need these both...

[ktf]

Ok, so you are saying that it's not the mmaping to cause the problem, but the subsequent allocations due to some deserialisation. Ok, I checked and indeed the mmap is called with MAP_SHARED...

Find in attachment the ROOTDEBUG=7 log. This is only for the component which reads the flat root files (i.e. only basic types).

log.txt

[vgvassilev]

According to the log ROOT is looking up Experimental (which is a namespace) and pulls in all modules that have namespace partitions with the name Experimental. We have tried to improve that with #10969 but probably missed some bit. Would applying it help?

[ktf]

The behavior seems to have indeed changed, yet I still see the same amount of RSS being allocated in the same codepath. See attached log.

log.txt.gz

[ktf]

Anything else which I can try here?

[vgvassilev]

@ktf,

cat /Users/vvassilev/Downloads/log\ 2.txt | grep 'on demand'
[89725:internal-dpl-aod-reader]: Loading 'Core' on demand for 'TArrayF'
[89725:internal-dpl-aod-reader]: Loading 'Cling_Runtime' on demand for 'T'
[89725:internal-dpl-aod-reader]: Loading 'Smatrix' on demand for 'MatRepStd'
[89725:internal-dpl-aod-reader]: Loading 'MathCore' on demand for 'R'
[89725:internal-dpl-aod-reader]: Loading 'std' on demand for 'complex'
[89725:internal-dpl-aod-reader]: Loading 'GenVector' on demand for 'Cartesian2D'
[89725:internal-dpl-aod-reader]: Loading 'ROOT_Foundation_Stage1_NoRTTI' on demand for 'TVirtualIsAProxy'
[89725:internal-dpl-aod-reader]: Loading 'Tree' on demand for 'TIOFeatures'
[89725:internal-dpl-aod-reader]: Loading 'MultiProc' on demand for 'ReadBuffer'
[89725:internal-dpl-aod-reader]: Loading 'Hist' on demand for 'TH1'
[89725:internal-dpl-aod-reader]: Loading 'TreePlayer' on demand for 'TTreeFormula'
[89725:internal-dpl-aod-reader]: Loading 'RIO' on demand for 'TMapFile'
[89725:internal-dpl-aod-reader]: Loading 'ROOT_Rtypes' on demand for 'UChar_t'

Looks quite good. However, for some reason we pre-load a lot more modules than I expect. Do you have an up-to-date ROOTSYS/lib/modules.idx file?

[ktf]

I have whatever is generated by building ROOT 6.28.4. I attached it. Is there a way to trim it down for a specific proxy?

modules.idx.zip

[ktf]

Any further idea?

[ktf]

Anything else we can try here? I would really love to get rid of the overhead from ROOT.