Clang memory usage is massive when invoking multiple TUs

[erichkeane]

Bugzilla Link	44823
Resolution	FIXED
Resolved on	Feb 13, 2020 02:07
Version	trunk
OS	Windows NT
Blocks	#43900
CC	@thebrandre,@topperc,@zmodem,@hfinkel,@RKSimon,@nico,@zygoloid,@rnk,@rgal

Extended Description

After this patch: https://reviews.llvm.org/D69825

multiple TUs cause really high memory usage. This is because each invocation of the CC1Command isn't cleaning up after itself thanks to -disable-free. I believe that in cases where there will be more than 1 CC1 invocation we need to disable the driver's addition of the disable-free flag for all but the last CC1 invocation.

[llvmbot]

Thanks Erich for the report! So I guess the repro is to simply write the following?

$ clang -c a.c b.c c.c ?

I'll take a look today.

[erichkeane]

Yes, exactly! just multiple .c files on a command line will cause multiple invocations of the CC1Command and they don't clean up after themselves :)

[topperc]

Might want to consider the case where you compile a single TU and invoke the linker to produce an executable in one command. Previously the cc1 process would die and clean up its memory before the linker ran. But now the cc1 memory would stick around.

[rnk]

I had assumed the in-process cc1 code path only fired when compiling a single input, and not invoking other tools like the linker. I think that's what MSVC does, but maybe it's conditional on passing /MP.

We could address this issue by disabling the in-process job when cc1 is not the only job, or by cleaning up when the in-process cc1 job is not the last job. I don't feel strongly about the direction.

Either way, in the case of a single, in-process compilation, we should retain the -disable-free behavior: there is no point to freeing heap memory etc before process exit.

[rnk]

*** Bug llvm/llvm-bugzilla-archive#44810 has been marked as a duplicate of this bug. ***

[rnk]

Given how this interacts with MSBuild projects, we should probably fix this before releasing.

[llvmbot]

Yes, it is on my todo list for today :-)

[nico]

(My https://reviews.llvm.org/D69848 sketch made the in-process cc1 thing only fire if there's just a single TU; maybe you can crib something from there.)

[llvmbot]

Please take a look: https://reviews.llvm.org/D74447

[llvmbot]

Committed as https://reviews.llvm.org/rG20f1abe306d0

[zmodem]

And pushed to 10.x as 9c9e46d.