Regression in Julia 1.11: Cannot create fresh base sysimage (`incremental=false`) containing Pkg

[DilumAluthge]

MWE

import PackageCompiler

sysimage_stdlibs = ["Pkg"]

base_sysimage = PackageCompiler.create_fresh_base_sysimage(
    sysimage_stdlibs;
    cpu_target = PackageCompiler.default_app_cpu_target(),
    sysimage_build_args = ``,
)

Expected behavior: no exception is thrown.

Actual behavior:

✔ [01m:35s] PackageCompiler: creating compiler .ji image (incremental=false)
⣄ [00m:55s] PackageCompiler: compiling fresh sysimage (incremental=false)error during bootstrap:
⠇ [00m:55s] PackageCompiler: compiling fresh sysimage (incremental=false)

LoadError("/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_QNNqc5/sysimage_packagecompiler_fdb8261a-8ca⠋ [00m:55s] PackageCompiler: compiling fresh sysimage (incremental=false)57, LoadError("/myjuliapath/julia/1.11.0/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl", 3, LoadError("/myjuliapath/julia/1.11.0/share/julia/stdlib/v1.11/Pkg/src/precompile.jl", 101, Base.IOError("could not spawn setenv(`'' a registries/Registry.tar.gz -tgzip registries/Registry.tar`,[... long list of environment variables ...]): permission denied (EACCES)", -13))))

_spawn_primitive at ./process.jl:140
[... very long stack trace ...]

✖ [00m:56s] PackageCompiler: compiling fresh sysimage (incremental=false)
ERROR: failed process: Process(setenv(`/myjuliapath/julia/1.11.0/bin/julia --color=yes --startup-file=no --pkgimages=no --cpu-target generic --sysimage=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_QNNqc5/corecompiler.ji --output-o=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_QNNqc5/sys-o.a /var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_QNNqc5/sysimage_packagecompiler_fdb8261a-8cae-11ef-3b3f-f5debedf65d6.jl`,[... long list of environment variables]), ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:598 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:482
  [3] macro expansion
    @ /mydir/PackageCompiler.jl/src/PackageCompiler.jl:270 [inlined]
  [4] (::PackageCompiler.var"#12#17"{String, Cmd, Vector{String}, String, String, String, String})()
    @ PackageCompiler /mydir/PackageCompiler.jl/ext/TerminalSpinners.jl:157
  [5] spin(f::PackageCompiler.var"#12#17"{…}, s::PackageCompiler.TerminalSpinners.Spinner{…})
    @ PackageCompiler.TerminalSpinners /mydir/PackageCompiler.jl/ext/TerminalSpinners.jl:164
  [6] macro expansion
    @ /mydir/PackageCompiler.jl/ext/TerminalSpinners.jl:157 [inlined]
  [7] (::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String})()
    @ PackageCompiler /mydir/PackageCompiler.jl/src/PackageCompiler.jl:256
  [8] cd(f::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String}, dir::String)
    @ Base.Filesystem ./file.jl:112
  [9] create_fresh_base_sysimage(stdlibs::Vector{String}; cpu_target::String, sysimage_build_args::Cmd)
    @ PackageCompiler /mydir/PackageCompiler.jl/src/PackageCompiler.jl:243
 [10] top-level scope
    @ REPL[2]:1
Some type information was truncated. Use `show(err)` to see complete types.

The actual error message is difficult to see above, because the line is really long and thus you have to scroll so far to the right. So here is the error message itself as a Markdown quote:

LoadError("/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_QNNqc5/sysimage_packagecompiler_fdb8261a-8ca⠋ [00m:55s] PackageCompiler: compiling fresh sysimage (incremental=false)57, LoadError("/myjuliapath/julia/1.11.0/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl", 3, LoadError("/myjuliapath/julia/1.11.0/share/julia/stdlib/v1.11/Pkg/src/precompile.jl", 101, Base.IOError("could not spawn setenv('' a registries/Registry.tar.gz -tgzip registries/Registry.tar,[... long list of environment variables ...]): permission denied (EACCES)", -13))))

Julia versioninfo:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2b (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M2
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

I can reproduce this bug on both Linux and macOS (Apple Silicon).

[DilumAluthge]

I have bisected this bug to JuliaLang/julia@5fc5556.

Bisect output:

5fc5556173684f5f54616f4ffa57f4539755df36 is the first bad commit
commit 5fc5556173684f5f54616f4ffa57f4539755df36
Author: Kristoffer Carlsson <kcarlsson89@gmail.com>
Date:   Thu Sep 21 04:42:33 2023 +0200

    move Pkg out of the sysimage (take 2) (#51189)

    Co-authored-by: Jonas Schulze <jonas.schulze@st.ovgu.de>
    Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com>
    Co-authored-by: Tim Besard <tim.besard@gmail.com>

 base/sysimg.jl                                     |  3 ---
 contrib/generate_precompile.jl                     | 13 +----------
 .../md5                                            |  1 +
 .../sha512                                         |  1 +
 .../md5                                            |  1 -
 .../sha512                                         |  1 -
 pkgimage.mk                                        |  2 +-
 stdlib/Pkg.version                                 |  2 +-
 stdlib/REPL/src/REPL.jl                            | 25 ++++++++++++++++++++++
 9 files changed, 30 insertions(+), 19 deletions(-)
 create mode 100644 deps/checksums/Pkg-cf0019fbbaf3ab1e606d4e973c1908bbd899a9d5.tar.gz/md5
 create mode 100644 deps/checksums/Pkg-cf0019fbbaf3ab1e606d4e973c1908bbd899a9d5.tar.gz/sha512
 delete mode 100644 deps/checksums/Pkg-ff6d645cd8bd18f4d22f6b5a24ff0234cf193fa5.tar.gz/md5
 delete mode 100644 deps/checksums/Pkg-ff6d645cd8bd18f4d22f6b5a24ff0234cf193fa5.tar.gz/sha512
bisect found first bad commit

Git bisect log

Here is the bisect log (click to expand):

git bisect start
# status: waiting for both good and bad commits
# bad: [501a4f25c2b7626c5e368cd01d1c050b70bafdb9] set VERSION to 1.11.0 (#55953)
git bisect bad 501a4f25c2b7626c5e368cd01d1c050b70bafdb9
# status: waiting for good commit(s), bad commit known
# good: [7fe6b16f4056906c99cee1ca8bbed08e2c154c1a] Set VERSION to 1.10.0-DEV (#47222)
git bisect good 7fe6b16f4056906c99cee1ca8bbed08e2c154c1a
# good: [382ba35647c7a44915ed2c0df99c04ce0f4b35a8] Support `convert(LineNumberNode, ::LineInfoNode)` (#50676)
git bisect good 382ba35647c7a44915ed2c0df99c04ce0f4b35a8
# bad: [d336a3e15db141cee054d90c7927c36807dbec81] Fix negation for an immutable unit triangular (#52510)
git bisect bad d336a3e15db141cee054d90c7927c36807dbec81
# bad: [4d36da7768ea403284cb7364366a941cfba40642] Improve dostring for error(msg...) (#51721)
git bisect bad 4d36da7768ea403284cb7364366a941cfba40642
# good: [39a53168a824a9a223adc6642da31e7a26b6890a] optimize: fix `effect_free` refinement in post-opt dataflow analysis (#51185)
git bisect good 39a53168a824a9a223adc6642da31e7a26b6890a
# bad: [f7d8bfb0113fad531bb8ac5b1714b8b8a89da61a] deal with remaining race conditions in these APIs
git bisect bad f7d8bfb0113fad531bb8ac5b1714b8b8a89da61a
# good: [ee371a1ab17e3656753e506ef925738ef46063ca] mention `getindex()` shouldn't return a "view" (#47099)
git bisect good ee371a1ab17e3656753e506ef925738ef46063ca
# good: [3290bd7baeb5a4a8f52cd2c55b182141251a3aae] inference: handle `Vararg` in `abstract_call_unionall` for `argtypes` computed by `abstract_apply` (#51393)
git bisect good 3290bd7baeb5a4a8f52cd2c55b182141251a3aae
# bad: [9f0676c87c9c8b7c3364d34f7439c61e9a5393e0] don't print task backtrace for GC threads (#51413)
git bisect bad 9f0676c87c9c8b7c3364d34f7439c61e9a5393e0
# good: [15f34aa649dbbb34e53ff6d16db15cd11ae4a887] [NFC] rng_split: some elaboration and clarification (#50680)
git bisect good 15f34aa649dbbb34e53ff6d16db15cd11ae4a887
# bad: [85c96b9c4597b6f3c51e879acd7ed990cf5cbe4e] define `show` instead of `repr` in BinaryPlatforms (#51398)
git bisect bad 85c96b9c4597b6f3c51e879acd7ed990cf5cbe4e
# bad: [5fc5556173684f5f54616f4ffa57f4539755df36] move Pkg out of the sysimage (take 2) (#51189)
git bisect bad 5fc5556173684f5f54616f4ffa57f4539755df36
# good: [602337034f7729f8b5893f3b5adea03130274497] Clarify `searchsortedfirst`/`searchsortedlast` docstrings (#50941)
git bisect good 602337034f7729f8b5893f3b5adea03130274497
# first bad commit: [5fc5556173684f5f54616f4ffa57f4539755df36] move Pkg out of the sysimage (take 2) (#51189)