Previously passing configuration times out on GH actions

[zakkak]

Description

mandrel 21.3-dev appears to be failing the last few days.

The last passing run was https://github.com/graalvm/mandrel/runs/3901801751?check_suite_focus=true with the following configuration:

component	version
mandrel	2d827b9
mandrel-packaging	9bbf83a0a813c778762279faa0460661810ec554
quarkus	a2c81efd6607d270dc7eaa9b7c38acb849c85030
jdk	11.0.13-ea+7
mx	5.309.2

Re-running the exact same configuration on GH runners results in timeouts or failures (see https://github.com/graalvm/mandrel/runs/3947878999?check_suite_focus=true)

How To Reproduce

Steps to reproduce the behavior:

gh workflow run base.yml \
  -f quarkus-version=a2c81efd6607d270dc7eaa9b7c38acb849c85030 \
  -f name="rerun last working config for 21.3" \
  -f jdk=ea \
  -f version=2d827b9970ac5e45dc1bb3e997aea4e3333d6939 \
  -f mandrel-packaging-version=9bbf83a0a813c778762279faa0460661810ec554  

[zakkak]

GraalVM CE builds with the same configuration seem to be working fine (see https://github.com/graalvm/mandrel/runs/3948735963?check_suite_focus=true) so this issue shouldn't have to do with changes in the GH runner.

Compiling Quarkus' integration test main (with -Dquarkus.native.native-image-xmx=5g) also results in the following:

	GraalVM CE 21.3	Mandrel 21.3-dev
Comp time	~16.5m	~24.5m
# of GCs	275	301
# of Full GCs	125	153
Time spend in Full GCs	~11.5m	~19.5m

Relates to: #288

[zakkak]

As the above numbers indicate main results in both Mandrel and GraalVM CE allocating enough data to make the heap usage reach close to the heap capacity and thus in excessive number of GCs

Removing -Dquarkus.native.native-image-xmx=5g the performance is much better and the number of Full GCs falls to only 11 (for both Mandrel and GraalVM CE).

That said, it appears that Mandrel is performing slightly more allocations than GraalVM CE resulting in higher heap usage and thus stressing the GC even more.

What's even more interesting is that building GraalVM CE from source without libgraal [1] appears to be performing better than both the GraalVM CE and Mandrel.

	GraalVM CE 21.3	Mandrel 21.3-dev	GraalVM CE nolibgraal
Comp time	~16.5m	~24.5m	~15m
# of GCs	275	301	245
# of Full GCs	125	153	82
Time spend in Full GCs	~11.5m	~19.5m	~9.8m

[1]

mx --primary-suite-path substratevm --components='Native Image' --native-images=native-image build

[zakkak]

FWIW: Building mandrel with labsjdk (to eliminate the chance of this being an artifact of using different base JDKs) and compiling with -Dquarkus.native.native-image-xmx=5g results in:

Fatal error:java.lang.OutOfMemoryError: GC overhead limit exceeded

[jerboaa]

FWIW: Building mandrel with labsjdk (to eliminate the chance of this being an artifact of using different base JDKs) and compiling with -Dquarkus.native.native-image-xmx=5g results in:

Fatal error:java.lang.OutOfMemoryError: GC overhead limit exceeded

It could be a symptom of metaspace issues.

@zakkak Could you do a run with this patch added to show RSS? This gives us some clue about non heap memory in addition to heap.

[zakkak]

(to eliminate the chance of this being an artifact of using different base JDKs)

It turns out there is indeed at least one difference causing more heap allocations to happen depending on the underlying JDK.

One of them is due to the implementation of HotSpotResolvedJavaFieldImpl.getAnnotation which in the upstream OpenJDK 11 results in the field being copied while in labsjdk it looks like no heap allocations are happening (probably due to caching).

HotSpotResolvedJavaFieldImpl.getAnnotation is invoked from AnalysisConstantReflectionProvider.filterInjectedAccessor

Experimenting further I tried building GraalVM CE without libgraal and without jlinking [1], the results are the following:

	GraalVM CE 21.3	Mandrel 21.3-dev	GraalVM CE nolibgraal	GraalVM CE nolibgraal nojlinking
Comp time	~16.5m	~24.5m	~15m	~18.5m
# of GCs	275	301	245	285
# of Full GCs	125	153	82	113
Time spend in Full GCs	~11.5m	~19.5m	~9.8m	~13.2m

Judging by the results it looks like not-jlinking has a big impact in the number of allocations. Looking into why...From my so far understanding this is due to AbstractNativeImageClassLoaderSupport.handleClassFileName being invoked more times and resulting in more allocations.

[1]

mx --primary-suite-path substratevm --no-jlinking --components='Native Image' build

[jerboaa]

Experimenting further I tried building GraalVM CE without libgraal and without jlinking [1], the results are the following:
[...]

Judging by the results it looks like not-jlinking has a big impact in the number of allocations.

This doesn't explain why we are seeing this now, though. Does it?

[zakkak]

This doesn't explain why we are seeing this now, though. Does it?

No, that's true.

Since you got me back on track I performed some more tests and eventually re-ran the exact same run that was failing (which led me to opening this issue) and now it passed (see https://github.com/graalvm/mandrel/runs/3968009738?check_suite_focus=true)

Additionally, I found out that the same test fails (on GH actions) with GraalVM CE 21.3 as well, so it's not a Mandrel thing after all... (see https://github.com/quarkusio/quarkus/runs/3966293297?check_suite_focus=true)

graal/master (i.e. latest graal) seems to work fine for both Mandrel and GraalVM CE builds (see https://github.com/graalvm/mandrel/runs/3962138022?check_suite_focus=true and https://github.com/graalvm/mandrel/runs/3959205365?check_suite_focus=true)

Unfortunately further testing indicates that this might be a transient error, I have seen the same mandrel version (a3ad15c) build the native image within ~11m, ~17m, and ~25m.

I'll try to run more experiments on an idling laptop to see if they are more stable.

[zakkak]

I'll try to run more experiments on an idling laptop to see if they are more stable.

The lack of stability appears to be the same on my idling laptop making this really hard to pin down.

logs/mandrel-21.2.0.2-Final-main-1:[INFO] Total time:  18:14 min
logs/mandrel-21.2.0.2-Final-main-2:[INFO] Total time:  17:48 min
logs/mandrel-21.2.0.2-Final-main-3:[INFO] Total time:  18:31 min
logs/mandrel-21.2.0.2-Final-main-4:[INFO] Total time:  17:58 min
logs/mandrel-21.2.0.2-Final-main-5:[INFO] Total time:  18:10 min

logs/mandrel-21.3.0.0-Final-main-1.txt:[INFO] Total time:  18:22 min
logs/mandrel-21.3.0.0-Final-main-2.txt:[INFO] Total time:  17:39 min
logs/mandrel-21.3.0.0-Final-main-3.txt:[INFO] Total time:  29:56 min
logs/mandrel-21.3.0.0-Final-main-4.txt:[INFO] Total time:  19:35 min
logs/mandrel-21.3.0.0-Final-main-5.txt:[INFO] Total time:  33:06 min

Passing -H:-ParseOnce seems to:

1. Improve compilation time by ~3x (In contrast to what I wrongfully observed when testing without constraining the memory usage on a9a2f40)
2. Increase stability
3. Decrease the maximum live data set (as observed by the gc logs) from 3.9G to 2.7G. The live data increase in the "universe" phase.

logs/mandrel-21.2.0.2-Final-main-noParseOnce-1.txt:[INFO] Total time:  09:59 min
logs/mandrel-21.2.0.2-Final-main-noParseOnce-2.txt:[INFO] Total time:  04:45 min
logs/mandrel-21.2.0.2-Final-main-noParseOnce-3.txt:[INFO] Total time:  04:45 min
logs/mandrel-21.2.0.2-Final-main-noParseOnce-4.txt:[INFO] Total time:  04:48 min
logs/mandrel-21.2.0.2-Final-main-noParseOnce-5.txt:[INFO] Total time:  04:45 min

logs/mandrel-21.3.0.0-Final-main-noParseOnce-1.txt:[INFO] Total time:  05:54 min
logs/mandrel-21.3.0.0-Final-main-noParseOnce-2.txt:[INFO] Total time:  05:42 min
logs/mandrel-21.3.0.0-Final-main-noParseOnce-3.txt:[INFO] Total time:  05:36 min
logs/mandrel-21.3.0.0-Final-main-noParseOnce-4.txt:[INFO] Total time:  05:38 min
logs/mandrel-21.3.0.0-Final-main-noParseOnce-5.txt:[INFO] Total time:  05:28 min

Some open questions:

1. How did this not pop up earlier in our CI? Quarkus main has not been using -H:-ParseOnce since 0be0406

2. What causes the large variance in compilation times?

3. Why does this only affect integration-tests/main to that degree?

[Karm]

@zakkak I might be bringing owls to Athens with this "hacker news" style pro tip....but have you disabled turbo boost to get more consistent perf? I guess fiddling around /sys/devices/system/cpu/intel_pstate/no_turbo on my workstation would do. Not sure about your system...

[zakkak]

@zakkak I might be bringing owls to Athens with this "hacker news" style pro tip....but have you disabled turbo boost to get more consistent perf? I guess fiddling around /sys/devices/system/cpu/intel_pstate/no_turbo on my workstation would do. Not sure about your system...

That's probably worth a try. Although the fact that the big variance is consistent on both my laptop and GH actions reduces my expectations (and is the main reason I didn't bother trying it yet). Note also that even in 21.2 we see excessive GC (which would result in throttling as well), but the variance only appears in 21.3...

PS: That's definitely not a "hacker news" style pro tip. I assure you there are plenty of academic papers (not to mention blog posts) out there that report X% of improvement without factoring in DVFS. Thanks for bringing it up :)