Skip to content

Conversation

alexcrichton
Copy link
Member

This commit adds logic to the compiler to attempt to handle super long linker
invocations by falling back to the @-file syntax if the invoked command is too
large. Each OS has a limit on how many arguments and how large the arguments can
be when spawning a new process, and linkers tend to be one of those programs
that can hit the limit!

The logic implemented here is to unconditionally attempt to spawn a linker and
then if it fails to spawn with an error from the OS that indicates the command
line is too big we attempt a fallback. The fallback is roughly the same for all
linkers where an argument pointing to a file, prepended with @, is passed.
This file then contains all the various arguments that we want to pass to the
linker.

Closes #41190

@alexcrichton
Copy link
Member Author

r? @michaelwoerister

@rust-highfive
Copy link
Contributor

r? @pnkfelix

(rust_highfive has picked a reviewer for you, use r? to override)

@carols10cents carols10cents added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 28, 2017
@shepmaster
Copy link
Member

@alexcrichton I've heard that @michaelwoerister is on vacation; perhaps you want to select another reviewer?

@alexcrichton
Copy link
Member Author

Ah that's ok, this is in no rush regardless

@michaelwoerister
Copy link
Member

@bors r+

Nice! Thanks, Alex!

@bors
Copy link
Collaborator

bors commented Sep 4, 2017

📌 Commit c8c4657 has been approved by michaelwoerister

@bors
Copy link
Collaborator

bors commented Sep 4, 2017

⌛ Testing commit c8c4657 with merge e7074e71c010e390089c13a146f7d84aa868f4dc...

@bors
Copy link
Collaborator

bors commented Sep 4, 2017

💔 Test failed - status-appveyor

@michaelwoerister
Copy link
Member

Seems like a legitimate error: https://ci.appveyor.com/project/rust-lang/rust/build/1.0.4538/job/f3auqt7jdxn3icc3#L7729

error[E0308]: mismatched types
   --> src\librustc_trans\back\link.rs:131:10
    |
131 |         (tool.to_command(), envs)
    |          ^^^^^^^^^^^^^^^^^ expected struct `back::command::Command`, found struct `std::process::Command`
    |
    = note: expected type `back::command::Command`
               found type `std::process::Command`
error: aborting due to previous error
error: Could not compile `rustc_trans`.

@michaelwoerister
Copy link
Member

@kennytm :D

@alexcrichton
Copy link
Member Author

@bors: r=michaelweorister

@bors
Copy link
Collaborator

bors commented Sep 5, 2017

📌 Commit 98e25a4 has been approved by michaelweorister

@arielb1 arielb1 added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2017
@bors
Copy link
Collaborator

bors commented Sep 5, 2017

⌛ Testing commit 98e25a41de8a093188171d2d94e1357b355ab9d6 with merge f3457796c3c61c6507fd47da84947d260eb8dd73...

@bors
Copy link
Collaborator

bors commented Sep 5, 2017

💔 Test failed - status-appveyor

@kennytm
Copy link
Member

kennytm commented Sep 5, 2017

@bors retry

check i686-pc-windows-msvc failed to build stage1-std, "failed to parse process output" ... ?

Building stage1 std artifacts (i686-pc-windows-msvc -> i686-pc-windows-msvc)
   Compiling libc v0.0.0 (file:///C:/projects/rust/src/rustc/libc_shim)
   Compiling gcc v0.3.51
   Compiling unwind v0.0.0 (file:///C:/projects/rust/src/libunwind)
   Compiling core v0.0.0 (file:///C:/projects/rust/src/libcore)
   Compiling libc v0.2.30
   Compiling filetime v0.1.10
   Compiling build_helper v0.1.0 (file:///C:/projects/rust/src/build_helper)
   Compiling std v0.0.0 (file:///C:/projects/rust/src/libstd)
   Compiling compiler_builtins v0.0.0 (file:///C:/projects/rust/src/rustc/compiler_builtins_shim)
   Compiling std_unicode v0.0.0 (file:///C:/projects/rust/src/libstd_unicode)
   Compiling rand v0.0.0 (file:///C:/projects/rust/src/librand)
   Compiling panic_abort v0.0.0 (file:///C:/projects/rust/src/libpanic_abort)
   Compiling alloc v0.0.0 (file:///C:/projects/rust/src/liballoc)
   Compiling collections v0.0.0 (file:///C:/projects/rust/src/libcollections)
   Compiling alloc_system v0.0.0 (file:///C:/projects/rust/src/liballoc_system)
   Compiling panic_unwind v0.0.0 (file:///C:/projects/rust/src/libpanic_unwind)
error: Could not compile `std`.
Caused by:
  failed to parse process output: `C:\projects\rust\build\bootstrap/debug/rustc --crate-name std src\libstd\lib.rs --color always --error-format json --crate-type dylib --crate-type rlib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 --cfg feature="backtrace" --cfg feature="panic-unwind" --cfg feature="panic_unwind" -C metadata=22fc13253edadc9b -C extra-filename=-22fc13253edadc9b --out-dir C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps -L dependency=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\release\deps --extern libc=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\liblibc-57f9cc46233141d9.rlib --extern compiler_builtins=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libcompiler_builtins-af47b91f5665c739.rlib --extern alloc=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\liballoc-8edc0052224e78e2.rlib --extern panic_unwind=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libpanic_unwind-f30e8935180a24f8.rlib --extern rand=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\librand-1dd743f4fd629578.rlib --extern collections=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libcollections-fab767d670e933ef.rlib --extern alloc_system=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\liballoc_system-adc2cf430de98b54.rlib --extern std_unicode=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libstd_unicode-77636529a2640872.rlib --extern panic_abort=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libpanic_abort-ba285cc200027a78.rlib --extern unwind=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libunwind-dea68fc02821d416.rlib --extern core=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\deps\libcore-4b30724c9b3622b8.rlib -l advapi32 -l ws2_32 -l userenv -l shell32 -L native=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\atlmfc\lib\ -L native=C:\projects\rust\build\i686-pc-windows-msvc\stage1-std\i686-pc-windows-msvc\release\build\compiler_builtins-9759ac0b288d4496\out` (exit code: 0)
To learn more, run the command again with --verbose.
thread 'main' panicked at 'command did not execute successfully: "C:\\projects\\rust\\build\\i686-pc-windows-msvc\\stage0/bin\\cargo.exe" "build" "-j" "4" "--target" "i686-pc-windows-msvc" "--release" "--locked" "--color" "always" "--features" "panic-unwind backtrace" "--manifest-path" "C:\\projects\\rust\\src/libstd/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101', src\bootstrap\compile.rs:883:8
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: C:\projects\rust\build\bootstrap\debug\bootstrap test --host i686-pc-windows-msvc --target i686-pc-windows-msvc
Build completed unsuccessfully in 0:18:46

This commit adds logic to the compiler to attempt to handle super long linker
invocations by falling back to the `@`-file syntax if the invoked command is too
large. Each OS has a limit on how many arguments and how large the arguments can
be when spawning a new process, and linkers tend to be one of those programs
that can hit the limit!

The logic implemented here is to unconditionally attempt to spawn a linker and
then if it fails to spawn with an error from the OS that indicates the command
line is too big we attempt a fallback. The fallback is roughly the same for all
linkers where an argument pointing to a file, prepended with `@`, is passed.
This file then contains all the various arguments that we want to pass to the
linker.

Closes rust-lang#41190
@alexcrichton
Copy link
Member Author

@bors: r=michaelwoerister

@bors
Copy link
Collaborator

bors commented Sep 6, 2017

📌 Commit ed938f0 has been approved by michaelwoerister

@bors
Copy link
Collaborator

bors commented Sep 7, 2017

⌛ Testing commit ed938f0 with merge a6a9d4c...

bors added a commit that referenced this pull request Sep 7, 2017
rustc: Attempt to handle super long linker invocations

This commit adds logic to the compiler to attempt to handle super long linker
invocations by falling back to the `@`-file syntax if the invoked command is too
large. Each OS has a limit on how many arguments and how large the arguments can
be when spawning a new process, and linkers tend to be one of those programs
that can hit the limit!

The logic implemented here is to unconditionally attempt to spawn a linker and
then if it fails to spawn with an error from the OS that indicates the command
line is too big we attempt a fallback. The fallback is roughly the same for all
linkers where an argument pointing to a file, prepended with `@`, is passed.
This file then contains all the various arguments that we want to pass to the
linker.

Closes #41190
@bors
Copy link
Collaborator

bors commented Sep 7, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing a6a9d4c to master...

@Mark-Simulacrum
Copy link
Member

@alexcrichton How do you feel about backporting this? It's a fairly small, targeted patch, and people are hitting on the build of the Rust compiler (#44898)... I'm somewhat hesitant, myself.

@Mark-Simulacrum Mark-Simulacrum added beta-nominated Nominated for backporting to the compiler in the beta channel. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 29, 2017
@alexcrichton
Copy link
Member Author

@Mark-Simulacrum we're about 2 weeks away from a release and incremental only "sorta works" on beta right now, so I think I'd be inclined to not backport.

@michaelwoerister
Copy link
Member

With the state of incremental in the current beta (i.e. halfway through the big tracking system refactoring) I'd rather make ./x.py --incremental do a non-incremental build.

@alexcrichton
Copy link
Member Author

De-nominating for beta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants