[Bug]: yarn install silently stops if memory is set too low in a CI environment

[macrozone]

Self-service

• I'd be willing to implement a fix

Describe the bug

i migrated a project to yarn 2 with nodeLinker: node-modules

running yarn install on gitlab-ci silently stops:

the log just ends there (followed by the next command)

I had a memory limit of 500Mi, increasing it to 700Mi fixes the problem.

Problem is though that there is no indication of a lack of memory, yarn just silently crashes.

To reproduce

I used node:14 as a base image in gitlab-ci

I set these env vars to limit the memory of the container:

  KUBERNETES_MEMORY_REQUEST: "350Mi"
  KUBERNETES_MEMORY_LIMIT: "500Mi"

running yarn install --immutable in such a container silently stops.

You have to lower the limit probably depending on the size of the project

Environment

System:
    OS: macOS 12.0.1
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
  Binaries:
    Node: 14.18.3 - /private/var/folders/h2/6_301c1x0qdbh_3gsyyjyvrr0000gn/T/xfs-d36c185a/node
    Yarn: 3.1.1 - /private/var/folders/h2/6_301c1x0qdbh_3gsyyjyvrr0000gn/T/xfs-d36c185a/yarn
    npm: 6.14.15 - ~/.nvm/versions/node/v14.18.3/bin/npm
  npmPackages:
    jest: ^27.4.5 => 27.4.5

Additional context

No response

[arcanis]

Yarn can't do much, it's your kernel that kills the process with no opportunity to react (if it was Node, you'd have a page fault error with memory addresses etc).

[clhuang]

the issue is not that yarn crashes, it's that it exits with no indication that it fails due to a lack of memory, making it quite difficult to debug. Why would yarn not be able to indicate this?

[clhuang]

if the kernel kills the process, it exits with code 137; yarn instead exits with code 0 so it seems it's not yarn itself that was killed, but one of yarn's child processes, and the overarching yarn process should be able to recognize this and surface the error to the caller.

[arcanis]

Yarn catches error codes from its subprocesses (and it's quite uncommon to have subprocesses during the fetch step), so that's very unlikely to be the case. I don't know exactly why the system would return 0, but my past investigations led me to believe it didn't come from Yarn itself.

[clhuang]

I added the following code to /opt/yarn-v1.22.17/bin/yarn.js:

process.on('exit', code => console.log('HIII not a kernel exit', code))

and it does display the text HIIII not a kernel exit 0, so at the very least, the yarn cli is not being killed by the kernel

[clhuang]

so in the docker container (running with host PID mode), I ran yarn install > /dev/null & sleep 1 && ps aux
which output:

root     25445 10.0  0.1 545852 64328 pts/0    Sl   22:43   0:00 node /opt/yarn-v1.22.17/bin/yarn.js install
root     25464  272  0.7 97015244 241620 pts/0 Rl   22:43   0:02 /usr/local/bin/node /app/.yarn/releases/yarn-3.1.1.cjs install

and then from the host's dmesg I see:

[Thu Mar  3 22:43:49 2022] Memory cgroup out of memory: Killed process 25464 (node) total-vm:97565800kB, anon-rss:994316kB, file-rss:36848kB, shmem-rss:0kB, UID:0 pgtables:7748kB oom_score_adj:0
[Thu Mar  3 22:43:49 2022] oom_reaper: reaped process 25464 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

so it seems like yarn-3.1.1 is being killed, and the parent yarn-v1.22.17 is the one not recognizing the OOM condition

[clhuang]

so it seems like the issue is here: https://github.com/yarnpkg/yarn/blob/3119382885ea373d3c13d6a846de743eca8c914b/src/cli/index.js#L639
exitCode is null if the child was killed with a SIGKILL, which happens in an OOM condition, so this error just gets swallowed

[ohana54]

Hi, encountered the same issue, could we please reopen this and possibly look at the PR?