After r371957 (06bb7dfbd): error: output file too large: 18446744073707016932 bytes

[DimitryAndric]

Bugzilla Link	44715
Resolution	WONTFIX
Resolved on	Feb 26, 2020 15:18
Version	unspecified
OS	All
Blocks	#43900
Attachments	Reproduction tarball for negative size problem
CC	@dwblaikie,@emaste,@zmodem,@MaskRay,@rui314,@smithp35

Extended Description

Trying to build one of FreeBSD's boot loader components with lld 10.0.0, we see the following error appear:

$ ld.lld --sysroot=/usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/tmp --eh-frame-hdr -Bstatic -m elf_i386_fbsd -o loader_lua.sym -L/usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/tmp/usr/lib -Ttext 0x0 --gc-sections --no-rosegment /usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/stand/i386/btx/lib/crt0.o main.o conf.o vers.o chain.o boot.o commands.o console.o devopen.o interp.o interp_backslash.o interp_parse.o ls.o misc.o module.o load_elf32.o load_elf32_obj.o reloc_elf32.o load_elf64.o load_elf64_obj.o reloc_elf64.o disk.o part.o vdisk.o dev_net.o bcache.o isapnp.o pnp.o interp_lua.o zfs_cmd.o /usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/stand/liblua32/liblua.a /usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/stand/i386/libi386/libi386.a /usr/obj/share/dim/src/freebsd/clang1000-import/amd64.amd64/stand/libsa32/libsa32.a
ld: error: output file too large: 18446744073707016908 bytes

18446744073707016908 = 0xffffffffffd952cc, so it is pretty clear that lld is somehow creating a negative size.

Bisecting shows that this was introduced with https://reviews.llvm.org/rL371957 ("[ELF] Map the ELF header at imageBase"). It seems that this change has a bad effect when both -Ttext 0x0 and --no-rosegment are used.

[MaskRay]

When --no-rosegment is specified, the ELF header, program headers and .text are in the same segment. When -Ttext=0 is specified, there will be no space to place the ELF header and program headers.

So this is an invalid use case.

If -Ttext is specified as some larger address, --image-base (lld specific) should also be specified when --no-rosegment is used.

[dwblaikie]

Seems like the error message could be better, though?

[DimitryAndric]

When --no-rosegment is specified, the ELF header, program headers and .text
are in the same segment. When -Ttext=0 is specified, there will be no space
to place the ELF header and program headers.

So this is an invalid use case.

Well, this is a rather particular case, since it is for a bootloader-type executable. Notet that we had to add the --no-rosegment option in the past, to work around another bad interaction between lld and our btxld tool: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225775

But just from upgrading from the previous version of lld, I would still consider this a regression, at least in user-visible behavior.

Basically, 9.0.1 and everything before worked, and produced an ELF layout like:

ELF Header:
Magic: 7f 45 4c 46 01 01 01 09 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - FreeBSD
ABI Version: 0
Type: EXEC (Executable file)
Machine: Intel 80386
Version: 0x1
Entry point address: 0x0
Start of program headers: 52 (bytes into file)
Start of section headers: 1662924 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 3
Size of section headers: 40 (bytes)
Number of section headers: 19
Section header string table index: 17

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 001000 046e45 00 AX 0 0 16
[ 2] .rodata PROGBITS 00046e48 047e48 00fa0c 00 AMS 0 0 8
[ 3] set_Xcommand_set PROGBITS 00056854 057854 00007c 00 A 0 0 4
[ 4] .data PROGBITS 00058000 058000 00504c 00 WA 0 0 4096
[ 5] .bss NOBITS 0005d050 05d04c 008a10 00 WA 0 0 8
[ 6] .gnu_debuglink PROGBITS 00000000 05d04c 000014 00 0 0 1
[ 7] .comment PROGBITS 00000000 05d060 003c08 01 MS 0 0 1
[ 8] .debug_str PROGBITS 00000000 060c68 0129f9 01 MS 0 0 1
[ 9] .debug_loc PROGBITS 00000000 073661 048a77 00 0 0 1
[10] .debug_abbrev PROGBITS 00000000 0bc0d8 010623 00 0 0 1
[11] .debug_info PROGBITS 00000000 0cc6fb 072228 00 0 0 1
[12] .debug_ranges PROGBITS 00000000 13e923 005010 00 0 0 1
[13] .debug_frame PROGBITS 00000000 143934 00c068 00 0 0 4
[14] .debug_line PROGBITS 00000000 14f99c 037d03 00 0 0 1
[15] .debug_aranges PROGBITS 00000000 18769f 0000a0 00 0 0 1
[16] .symtab SYMTAB 00000000 187740 0080a0 10 18 1425 4
[17] .shstrtab STRTAB 00000000 18f7e0 0000c3 00 0 0 1
[18] .strtab STRTAB 00000000 18f8a3 006729 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0x00000000 0x00000000 0x568d0 0x568d0 R E 0x1000
LOAD 0x058000 0x00058000 0x00058000 0x0504c 0x0da60 RW 0x1000
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0

Section to Segment mapping:
Segment Sections...
00 .text .rodata set_Xcommand_set
01 .data .bss
02

So it placed .text at offset 0x1000 all by itself, and its address at 0.

If -Ttext is specified as some larger address,

Which larger address should it be then? I thought it was the linker's job to figure out how to put stuff in ELF files? :)

--image-base (lld specific)
should also be specified when --no-rosegment is used.

Hm, so maybe we should simply not bother with -Ttext at all, and only specify --image-base? I wonder if that works, and produces the same layout as I pasted above...

[MaskRay]

Seems like the error message could be better, though?

First, I don't think this --no-rosegment -Ttext=0 case is common enough that deserves a warning. Many -Ttext=0 use cases actually need careful scrutiny.
Second, we can probably generalize the failing case and add a diagnostic for the general case. Not that I don't want to improve it, but it is difficult. They are delicate file offset assignment rules.

But just from upgrading from the previous version of lld, I would still consider this a regression, at least in user-visible behavior.

Before I committed this change, I had analyzed the failing cases. I built all of our internal targets and just one file needed a --image-base fix. I also wrote about the -Ttext --image-base thing in the description. It may be hard to understand because of the nature of the subtle change.

I need this change to push my "[ELF][X86] Allow PT_LOAD to have overlapping p_offset ranges on EM_X86_64" patch.
It concluded the patch series that decreased the binary size by 0~3*maxpagesize bytes.

I think bugzilla is really not a good place to discuss this issue which requires questions and answers in several rounds. IRC may be better. I am on #bsddev@efnet and you can catch me there. Or you can ask Bdragon.

btxld does not correctly handle input with other than 2 PT_LOAD
segments. Passing --no-rosegment lets lld produce output eqivalent to
ld.bfd: 2 PT_LOAD segments and no PT_GNU_RELRO.

This looks very like NetBSD ld.so limitation. I hope it can be fixed.
This is a size motivation to do that: objcopy --only-keep-debug can produce smaller debug files for a binary with both R and RX segments.

Which larger address should it be then? I thought it was the linker's job to figure out how to put stuff in ELF files? :)

As my description says, (-Ttext < default image base) is a subtle case in GNU ld. lld had some support before rL371957 and it might appear to work but it was brittle. We can discuss it on IRC.

[MaskRay]

Basically, 9.0.1 and everything before worked, and produced an ELF layout like:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 001000 046e45 00 AX 0 0 16

Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0x00000000 0x00000000 0x568d0 0x568d0 R E 0x1000

So it placed .text at offset 0x1000 all by itself, and its address at 0.

I played with a simple example with GNU ld. Its layout is similar.

This layout is weird. The ELF header and program headers are not mapped.

Hm, so maybe we should simply not bother with -Ttext at all, and only specify --image-base? I wonder if that works, and produces the same layout as I pasted above...

s/-Ttext=0/--image-base=0/ definitely deserves a try.

For use cases like -Ttext=0x7f8000000000,
I will suggest s/-Ttext=0x7f8000000000/--image-base=0x7f8000000000 -Ttext=0x7f8000001000/

[DimitryAndric]

Basically, 9.0.1 and everything before worked, and produced an ELF layout like:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 001000 046e45 00 AX 0 0 16

Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0x00000000 0x00000000 0x568d0 0x568d0 R E 0x1000

So it placed .text at offset 0x1000 all by itself, and its address at 0.

I played with a simple example with GNU ld. Its layout is similar.

This layout is weird. The ELF header and program headers are not mapped.

Hm, so maybe we should simply not bother with -Ttext at all, and only specify --image-base? I wonder if that works, and produces the same layout as I pasted above...

s/-Ttext=0/--image-base=0/ definitely deserves a try.

For offsets other than 0, I can't get it to work, as our btxld complains "Not a BTX kernel". This is because the btx binary is linked with (effectively):

-nostdlib -e start -Ttext 0x9000 -Wl,-N,-S,--oformat,binary -Wl,--build-id=none -Wl,--no-rosegment -o btx btx.o

If the option is changed to use --image-base, like the following:

-nostdlib -e start -Wl,--image-base=0x9000 -Wl,-N,-S,--oformat,binary -Wl,--build-id=none -Wl,--no-rosegment -o btx btx.o

the resulting binary gets some sort of dynamic linker section prepended!

E.g., with -T text 0x9000, the binary starts with:

00000000 eb 0e 42 54 58 01 02 00 f7 0f b0 06 00 00 00 00 |..BTX...........|
00000010 fa 31 c0 8e d0 bc 00 18 8e c0 8e d8 66 6a 02 66 |.1..........fj.f|
00000020 9d bf 00 5e b9 00 19 f3 ab bb 48 95 b9 10 00 bf |...^......H.....|
00000030 80 00 89 1d 47 47 ab 83 c3 04 e2 f6 bf 00 5e be |....GG........^.|
00000040 02 96 ac 98 91 e3 1d ac 92 ad 93 ad b6 08 d1 eb |................|
00000050 73 0b 89 05 88 75 02 88 55 05 83 c0 04 8d 7d 08 |s....u..U.....}.|
00000060 e2 ec eb de c6 45 05 18 c6 45 08 10 c6 45 66 68 |.....E...E...Efh|
00000070 bb 20 28 e8 c5 00 67 0f 01 1d f6 95 00 00 67 0f |. (...g.......g.|
00000080 01 15 f0 95 00 00 0f 20 c0 40 0f 22 c0 ea 92 90 |....... .@."....|
00000090 08 00 31 c9 b1 10 8e d1 b1 38 0f 00 d9 ba 00 a0 |..1......8......|

but with -Wl,--image-base=0x9000, it becomes:

00000000 2f 6c 69 62 65 78 65 63 2f 6c 64 2d 65 6c 66 2e |/libexec/ld-elf.|
00000010 73 6f 2e 31 00 00 00 00 00 00 00 00 00 00 00 00 |so.1............|
00000020 eb 0e 42 54 58 01 02 00 f7 0f b0 06 00 00 00 00 |..BTX...........|
00000030 fa 31 c0 8e d0 bc 00 18 8e c0 8e d8 66 6a 02 66 |.1..........fj.f|
00000040 9d bf 00 5e b9 00 19 f3 ab bb 68 95 b9 10 00 bf |...^......h.....|
00000050 80 00 89 1d 47 47 ab 83 c3 04 e2 f6 bf 00 5e be |....GG........^.|
00000060 22 96 ac 98 91 e3 1d ac 92 ad 93 ad b6 08 d1 eb |"...............|
00000070 73 0b 89 05 88 75 02 88 55 05 83 c0 04 8d 7d 08 |s....u..U.....}.|
00000080 e2 ec eb de c6 45 05 18 c6 45 08 10 c6 45 66 68 |.....E...E...Efh|
00000090 bb 20 28 e8 c5 00 67 0f 01 1d 16 96 00 00 67 0f |. (...g.......g.|

Obviously some jump offsets here and there are also different. So in any case, --image-base works differently from -T text, it seems. Certainly so when the vale is nonzero.

I can't test the case now with -image-base=0, since the binaries involved depend on a correctly linked btx binary. I will need to do some more work to get that finished, but I'm out of time now.

For use cases like -Ttext=0x7f8000000000,
I will suggest s/-Ttext=0x7f8000000000/--image-base=0x7f8000000000
-Ttext=0x7f8000001000/

Most of the times this option is used, it is for bootloader-related binaries, which run at very small addresses, like 0x2000 or 0x7c00. They typically don't want an extra 4k page at the start.

[DimitryAndric]

After some messing around with the build system, I managed to build one of the loaders with --image-base=0, but the resulting ELF binary has a completely different layout now. :(

With -Ttext 0, the entrypoint becomes 0 (as intended), and there are only two LOAD segments (which was the original goal of the --ro-segment flag usage):

ELF Header:
Magic: 7f 45 4c 46 01 01 01 09 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: FreeBSD
ABI Version: 0
Type: EXEC (Executable file)
Machine: Intel i386
Version: 0x1
Entry point address: 0
Start of program headers: 52 (bytes into file)
Start of section headers: 1358756 (bytes into file)
Flags: 0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 3
Size of section headers: 40 (bytes)
Number of section headers: 20
Section header string table index: 18

Elf file type is EXEC (Executable file)
Entry point 0x0
There are 3 program headers, starting at offset 52

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0x00000000 0x00000000 0x4b184 0x4b184 R E 0x1000
LOAD 0x04d000 0x0004d000 0x0004d000 0x0604c 0x0ea68 RW 0x1000
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0

[...]
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 001000 03cda3 00 AX 0 0 16
[ 2] .rodata PROGBITS 0003cda8 03dda8 00e35c 00 AMS 0 0 8
[ 3] set_Xcommand_set PROGBITS 0004b104 04c104 00007c 00 A 0 0 4
[ 4] set_Xficl_compile PROGBITS 0004b180 04c180 000004 00 A 0 0 4
[ 5] .data PROGBITS 0004d000 04d000 00604c 00 WA 0 0 4096
[ 6] .bss NOBITS 00053050 05304c 008a18 00 WA 0 0 8

After, the entry point becomes 0xe4a0 (??), and the executable now also has a PHDR segment:

ELF Header:
Magic: 7f 45 4c 46 01 01 01 09 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: FreeBSD
ABI Version: 0
Type: EXEC (Executable file)
Machine: Intel i386
Version: 0x1
Entry point address: 0xe4a0
Start of program headers: 52 (bytes into file)
Start of section headers: 1354660 (bytes into file)
Flags: 0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 4
Size of section headers: 40 (bytes)
Number of section headers: 20
Section header string table index: 18

Elf file type is EXEC (Executable file)
Entry point 0xe4a0
There are 4 program headers, starting at offset 52

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000034 0x00000034 0x00000034 0x00080 0x00080 R 0x4
LOAD 0x000000 0x00000000 0x00000000 0x4b243 0x4b243 R E 0x1000
LOAD 0x04c000 0x0004d000 0x0004d000 0x0604c 0x0ea68 RW 0x1000
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0

[...]

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .rodata PROGBITS 000000b8 0000b8 00e35c 00 AMS 0 0 8
[ 2] set_Xcommand_set PROGBITS 0000e414 00e414 00007c 00 A 0 0 4
[ 3] set_Xficl_compile PROGBITS 0000e490 00e490 000004 00 A 0 0 4
[ 4] .text PROGBITS 0000e4a0 00e4a0 03cda3 00 AX 0 0 16
[ 5] .data PROGBITS 0004d000 04c000 00604c 00 WA 0 0 4096
[ 6] .bss NOBITS 00053050 05204c 008a18 00 WA 0 0 8

Furthermore, the .text and .rodata segments are now longer directly consecutive, but for some reason lld has moved .text to the back of the executable? This basically messes up the layout of the file completely.

I think that for the short term, I will simply keep r371957 reverted in the FreeBSD tree, since this issue is a blocker for importing 10.0.0, and I don't see it get solved soon without a lot more help from people who understand the intricacies of our boot loader building process.

[MaskRay]

I consider -Ttext=0 a shortcut of a linker scripts.

In this case, a linker case may work better.

% cat a.x
SECTIONS {
.text 0 : { (.text) } # RX
.rodata : { (.rodata) } # R
}

Other R, RX and RW sections will be placed in the appropriate place.
If you want to adjust the order, add more descriptions.

Instead of: ld.lld -Ttext=0 --no-rosegment a.o -o a
Use: ld.lld -T a.x --no-rosegment a.o -o a

The SECTIONS command currently forces --no-rosegment
https://github.com/llvm/llvm-project/blob/master/lld/ELF/ScriptParser.cpp#L531
I am considering changing it. -T and -T --no-rosegment may be different in the future.

[zmodem]

What's the current status here?

[DimitryAndric]

What's the current status here?

In FreeBSD's clang I just reverted 06bb7df, since I didn't manage to make any of the other options work at the time. I have not yet spent time figuring out how the linker script workaround could be applied. Even though the meaning of -Ttext 0 can probably considered to be invalid, there is the case that this used to work through quite a number of years, and now stopped working. :)

[zmodem]

Fangrui, can you please add to the lld release notes about the behaviour change in 06bb7df and what users should do to work around it?

[MaskRay]

Will mention -Ttext=0 in https://reviews.llvm.org/D75216 (some lld/ELF release notes).

I chatted on #bsdmips. dim has a workaround for FreeBSD and some other people (Bdragon) are helping. We can call it resolved.

It will also be nice to not rely on --no-rosegment (well, we have internal code using it. ChromeOS is using it... Other projects migrating from GNU ld/gold may use it...)