CI flaky in Android build as out of memory

[gnprice]

We're continuing to occasionally see the symptom of #1330 — CI runs sometimes fail with a "Java heap space" error in the android suite, as Gradle runs out of memory. Here's an example today:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:compileDebugKotlin'.
> Error while evaluating property 'friendPathsSet$kotlin_gradle_plugin_common' of task ':app:compileDebugKotlin'.
   > Could not resolve all files for configuration ':app:debugCompileClasspath'.
      > Failed to transform armeabi_v7a_debug-1.0.0-1e035cc693f067ed5033f180e0b6169d2c8a3e58.jar (io.flutter:armeabi_v7a_debug:1.0.0-1e035cc693f067ed5033f180e0b6169d2c8a3e58) to match attributes {artifactType=android-classes-jar, org.gradle.category=library, org.gradle.libraryelements=jar, org.gradle.status=release, org.gradle.usage=java-api}.
         > Execution failed for JetifyTransform: /home/runner/.gradle/caches/modules-2/files-2.1/io.flutter/armeabi_v7a_debug/1.0.0-1e035cc693f067ed5033f180e0b6169d2c8a3e58/1d4de013e75bd976543eba257de04ed6657c9c28/armeabi_v7a_debug-1.0.0-1e035cc693f067ed5033f180e0b6169d2c8a3e58.jar.
            > Java heap space

Before #1324, this failure was happening in 100% of CI runs — that was #1330.

With #1324 we bumped the Gradle JVM max heap size from 1.5 GiB to 3 GiB. That made CI mostly pass, but it continued to sometimes fail:
https://chat.zulip.org/#narrow/channel/243-mobile-team/topic/Gradle.20out.20of.20memory/near/2082072

With #1331 we bumped it further to 4 GiB. The failure today, quoted above, was with 4 GiB.

[gnprice]

The obvious next thing to try, and which I guess we should try, is to bump the max heap size further.

It's not clear that will solve the problem, though — so we'll leave this issue open, and monitor for a while before declaring it fixed. Some builds of example apps within the Flutter test suite were flaking with a similar error a few months ago, flutter/flutter#156063; they bumped the heap size from 4 GiB to 8 GiB although not confident the heap size was the real issue:
flutter/flutter#156063 (comment)
and indeed the flakes continued after the heap size was increased:
flutter/flutter#156063 (comment)

In particular I checked the latest example flake output from that thread, from 2024-11-06:
flutter/flutter#156063 (comment)
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8731992401931736721/+/u/step_is_flaky:_run_native_assets_android/stdout
and it indeed has "Java heap space" failures. So it's not that some unrelated flake appeared around the same time as the attempted fix.

Those flakes did eventually stop, as seen in that thread. The thread has no theories on what else might have fixed them.

[gnprice]

… Hmm, never mind: that failed run I linked above was actually from before #1331 and even #1324.

What happened is that I got got by the classic GitHub bug where it shows a page that's a mix of old and new data: the tab where I had #1327 open was showing the new version I'd pushed earlier today, and my comment today, and below that was saying that checks had failed. And the link for details led to that CI run I mentioned above. And this was a couple of hours after I'd pushed that new version, so if checks had completed and failed that meant the checks on the new version had failed, right?

But in reality, that part of the page was just stale. It was still showing the old status from before my new revision — the revision I'd pushed hours earlier, and which was shown on the very same page right above the checks status. Reloading the page shows that CI succeeded, as expected.

I guess if we do at some point see a flake of this kind post-#1331, we can re-open this thread.