1.11 regression: REPL error if starting julia with `-O3`

[cgeoga]

Hello,

This issue also has a little discussion at this discourse link.

After upgrading to v1.11, I have an issue where if I start a REPL with -O3 I get an error when I press ] to enter Pkg mode:

cg:~> julia -O3
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> Unhandled Task ERROR: ArgumentError: Package REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2423
  [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2300
  [3] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
  [4] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
  [5] _require_prelocked
    @ ./loading.jl:2287 [inlined]
  [6] _require_prelocked
    @ ./loading.jl:2286 [inlined]
  [7] macro expansion
    @ ./loading.jl:2587 [inlined]
  [8] macro expansion
    @ ./lock.jl:273 [inlined]
  [9] require_stdlib(package_uuidkey::Base.PkgId, ext::String)
    @ Base ./loading.jl:2542
 [10] macro expansion
    @ ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/Pkg_beforeload.jl:8 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] load_pkg()
    @ REPL ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/Pkg_beforeload.jl:7
 [13] (::REPL.var"#119#137"{REPL.LineEdit.MIState, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ REPL ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:1233

If I instantiate a REPL without the flag, it works fine:

cg:~> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> 

(@v1.11) pkg> # pressed enter, no error

I have recreated this issue on two machines with the following versioninfo:

julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2b (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × 12th Gen Intel(R) Core(TM) i7-1260P
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
  LD_LIBRARY_PATH = :/opt/knitro-13.2.0-Linux-64/lib:/opt/knitro-13.2.0-Linux-64/lib:/opt/knitro-13.2.0-Linux-64/lib

and

julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2b (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 9554P 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver4)
Threads: 1 default, 0 interactive, 1 GC (on 128 virtual cores)
Environment:
  LD_LIBRARY_PATH = :/u/g/e/geoga/private/knitro-13.2.0-Linux-64/lib

It seems like several other commenters on the discourse thread are experiencing something similar.

Is there a simple fix to this, or is this the result of a mistake on my end? I installed the updated version with juliaup, if that is relevant.

Thanks so much!

[DilumAluthge]

Does the bug occur on Julia 1.10.5?

[cgeoga]

Not for me, no. I've had the bash alias

alias jp = 'julia -O3 --banner=no --project -C"native"'

for a very long time (maybe since 1.6 or 1.7?) and have never seen this issue before.

[DilumAluthge]

On 1.11, what do other values for -O give you? E.g. -O0, -O1, and -O2?

[cgeoga]

Ah, good idea! It is just -O3 that makes the error for me.

[PallHaraldsson]

I confirm precompiling but not error on 1.11.0 (and rc4), i.e. without -C"native"' but then an error with it see B.

A.
First when entering Julia, then for the REPL and more e.g. StyledStrings, then even more when entering package mode:

(@v1.11) pkg> Precompiling REPLExt...
  Progress [>                                        ]  0/27
  ◒ NetworkOptions
  ◑ UUIDs
  ◓ Logging
  ◒ Zlib_jll
  ◑ MbedTLS_jll
  ◑ ArgTools
  ◑ MozillaCACerts_jll
  ◒ nghttp2_jll
  ◓ p7zip_jll

[And this takes quite a long time, still running for me... After no progress for ca. 10 min I did CTRL-C.]

I feel responsible, I got lowered opt in for the REPL, and suspect it related. If that's the case then there possibly needs to be a max. opt. option for packages (at least used by stdlibs).

B.

$ julia -O3 --project -C"native"
Precompiling REPL...
Info Given REPL was explicitly requested, output will be shown live 
ERROR: LoadError: ArgumentError: Package REPL does not have StyledStrings in its dependencies:
- Note that the following manifests in the load path were resolved with a different
  julia version, which may be the cause of the error. Try to re-resolve them in the
  current version, or consider deleting them if that fails:
    /home/pharaldsson/Manifest.toml (v1.10.5)
    /home/pharaldsson/.julia/environments/v1.11/Manifest.toml (v1.11.0-rc3)
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have REPL checked out for development and have
  added StyledStrings as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with REPL
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:2211 [inlined]
 [2] macro expansion
   @ ./lock.jl:273 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2183
 [4] #invoke_in_world#3
   @ ./essentials.jl:1088 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:1085 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2176
 [7] include
   @ ./Base.jl:557 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2806
 [9] top-level scope
   @ stdin:4
in expression starting at /home/pharaldsson/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:3
in expression starting at stdin:4
  ✗ REPL
  0 dependencies successfully precompiled in 2 seconds. 4 already precompiled.
[ Info: Precompiling REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] (cache misses: invalid header (8), mismatched flags (4), dep uuid changed (2))
ERROR: LoadError: ArgumentError: Package REPL does not have StyledStrings in its dependencies:
- Note that the following manifests in the load path were resolved with a different
  julia version, which may be the cause of the error. Try to re-resolve them in the
  current version, or consider deleting them if that fails:
    /home/pharaldsson/Manifest.toml (v1.10.5)
    /home/pharaldsson/.julia/environments/v1.11/Manifest.toml (v1.11.0-rc3)
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have REPL checked out for development and have
  added StyledStrings as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with REPL
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:2211 [inlined]
 [2] macro expansion
   @ ./lock.jl:273 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2183
 [4] #invoke_in_world#3
   @ ./essentials.jl:1088 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:1085 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2176
 [7] include
   @ ./Base.jl:557 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2806
 [9] top-level scope
   @ stdin:4
in expression starting at /home/pharaldsson/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:3
in expression starting at stdin:4
┌ Warning: Failed to import REPL
│   exception =
│    Failed to precompile REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] to "/home/pharaldsson/.julia/compiled/v1.11/REPL/jl_Y2QYsm".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64})
│        @ Base ./loading.jl:3089
│      [3] (::Base.var"#1081#1082"{Base.PkgId})()
│        @ Base ./loading.jl:2477
│      [4] mkpidlock(f::Base.var"#1081#1082"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
│      [5] #mkpidlock#6
│        @ ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
│      [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
│      [7] #invokelatest#2
│        @ ./essentials.jl:1056 [inlined]
│      [8] invokelatest
│        @ ./essentials.jl:1051 [inlined]
│      [9] maybe_cachefile_lock(f::Base.var"#1081#1082"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:3613
│     [10] maybe_cachefile_lock
│        @ ./loading.jl:3610 [inlined]
│     [11] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:2473
│     [12] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:2300
│     [13] #invoke_in_world#3
│        @ ./essentials.jl:1088 [inlined]
│     [14] invoke_in_world
│        @ ./essentials.jl:1085 [inlined]
│     [15] _require_prelocked
│        @ ./loading.jl:2287 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:2286 [inlined]
│     [17] macro expansion
│        @ ./loading.jl:2587 [inlined]
│     [18] macro expansion
│        @ ./lock.jl:273 [inlined]
│     [19] require_stdlib(package_uuidkey::Base.PkgId, ext::Nothing)
│        @ Base ./loading.jl:2542
│     [20] require_stdlib
│        @ ./loading.jl:2542 [inlined]
│     [21] load_REPL()
│        @ Base ./client.jl:410
│     [22] run_main_repl(interactive::Bool, quiet::Bool, banner::Symbol, history_file::Bool, color_set::Bool)
│        @ Base ./client.jl:425
│     [23] repl_main
│        @ ./client.jl:567 [inlined]
│     [24] _start()
│        @ Base ./client.jl:541
└ @ Base client.jl:412
┌ Warning: REPL provider not available: using basic fallback
│   LOAD_PATH = "@:@v#.#:@stdlib"
└ @ Base client.jl:451

julia> 

[vtjnash]

This is somewhat expected that the REPL might need to compile a new copy (unable to use require_stdlib) with non-default options. Though I thought our intent is to compile everything at O3 with g2, so those at least don't cause recompilation (and currently seems those aren't getting propagated though)

[PallHaraldsson]

See:

julia/stdlib/REPL/src/REPL.jl

Line 17 in ecf41b1

Base.Experimental.@optlevel 1

[I see REPLExt in the code, that's it loaded, but not from where...]

And this speed-optimization (for Pkg-related, actually for PythonCall indirectly) might be the cause (for the package mode):

https://github.com/JuliaLang/Pkg.jl/pull/4015/files

I thought our intent is to compile everything at O3 with g2

What is g2? And is compiling with O3 just to avoid recompilation, then doing julia -O3? [When you lower opt. from I suppose never recompiled for slower, only when asking for higher than in (stdlib) packages, why I suggested a max opt option, for those packages that never need more opt.]

[KristofferC]

If one uses a 1.10 manifest with 1.11 + -O3 things go bad:

❯ julia +1.11 -O3 --project
Precompiling InteractiveUtils...
  3 dependencies successfully precompiled in 3 seconds
Precompiling REPL...
Info Given REPL was explicitly requested, output will be shown live 
ERROR: LoadError: ArgumentError: Package REPL does not have StyledStrings in its dependencies:
- Note that the following manifests in the load path were resolved with a different
  julia version, which may be the cause of the error. Try to re-resolve them in the
  current version, or consider deleting them if that fails:
    /home/kc/JuliaTests/PkgExtComp/Manifest.toml (v1.10.5)
    /home/kc/.julia/environments/v1.11/Manifest.toml (v1.11.0-rc4)
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have REPL checked out for development and have
  added StyledStrings as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with REPL
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:2211 [inlined]
 [2] macro expansion
   @ ./lock.jl:273 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2183
 [4] #invoke_in_world#3
   @ ./essentials.jl:1088 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:1085 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2176
 [7] include
   @ ./Base.jl:557 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2806
 [9] top-level scope
   @ stdin:4
in expression starting at /home/kc/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:3
in expression starting at stdin:4
  ✗ REPL
  1 dependency successfully precompiled in 1 seconds. 3 already precompiled.
[ Info: Precompiling REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] (cache misses: mismatched flags (4))
ERROR: LoadError: ArgumentError: Package REPL does not have StyledStrings in its dependencies:
- Note that the following manifests in the load path were resolved with a different
  julia version, which may be the cause of the error. Try to re-resolve them in the
  current version, or consider deleting them if that fails:
    /home/kc/JuliaTests/PkgExtComp/Manifest.toml (v1.10.5)
    /home/kc/.julia/environments/v1.11/Manifest.toml (v1.11.0-rc4)
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have REPL checked out for development and have
  added StyledStrings as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with REPL
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:2211 [inlined]
 [2] macro expansion
   @ ./lock.jl:273 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2183
 [4] #invoke_in_world#3
   @ ./essentials.jl:1088 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:1085 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:2176
 [7] include
   @ ./Base.jl:557 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2806
 [9] top-level scope
   @ stdin:4
in expression starting at /home/kc/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:3
in expression starting at stdin:4
┌ Warning: Failed to import REPL
│   exception =
│    Failed to precompile REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] to "/home/kc/.julia/compiled/v1.11/REPL/jl_1z7rAD".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
...
│        @ ./client.jl:567 [inlined]
│     [24] _start()
│        @ Base ./client.jl:541
└ @ Base client.jl:412
┌ Warning: REPL provider not available: using basic fallback
│   LOAD_PATH = "@:@v#.#:@stdlib"
└ @ Base client.jl:451

[cgeoga]

Hm. That is interesting. A small clarification on my part though: I mention using -C"native" and -O3 in my alias for a long time just to suggest that I think this problem is new. But I also don't think I'm hitting it because of an old Manifest.toml file, unless I'm confusing myself about something:

cg:~> head .julia/environments/v1.11/Manifest.toml 
# This file is machine-generated - editing it directly is not advised

julia_version = "1.11.0"
manifest_format = "2.0"
project_hash = "da39a3ee5e6b4b0d3255bfef95601890afd80709"

[deps]
cg:~> cat .julia/environments/v1.11/Project.toml 
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: .julia/environments/v1.11/Project.toml   <EMPTY>
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
cg:~> .juliaup/bin/julia -O3
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> Unhandled Task ERROR: ArgumentError: Package REPLExt [e5eb5ef1-03cf-53a7-ae1d-5a66b08e832b] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2423
  [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2300
  [3] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
  [4] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
  [5] _require_prelocked
    @ ./loading.jl:2287 [inlined]
  [6] _require_prelocked
    @ ./loading.jl:2286 [inlined]
  [7] macro expansion
    @ ./loading.jl:2587 [inlined]
  [8] macro expansion
    @ ./lock.jl:273 [inlined]
  [9] require_stdlib(package_uuidkey::Base.PkgId, ext::String)
    @ Base ./loading.jl:2542
 [10] macro expansion
    @ ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/Pkg_beforeload.jl:8 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] load_pkg()
    @ REPL ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/Pkg_beforeload.jl:7
 [13] (::REPL.var"#119#137"{REPL.LineEdit.MIState, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ REPL ~/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia/stdlib/v1.11/REPL/src/REPL.jl:1233