[native-image] Debug information not usable in gdb

[ctiedt]

Describe the issue
When building a native image binary with the -g flag, the debugging information is not recognized by gdb. The specific error is Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, 4 or 5) [in module /home/clemens/Studium/MA/re22/helloworld].

Steps to reproduce the issue
Please include both build steps as well as run steps

1. Write a Java program (e.g. just a Hello World) and compile it to bytecode
2. Compile it to a native binary with native-image -g HelloWorld
3. Load the binary in gdb: gdb helloworld

Describe GraalVM and your environment:

• GraalVM version: CE 22.1.0
• JDK major version: 17
• OS: Arch Linux
• Architecture: AMD64
• gdb: 12.1 (also tested on 10.2 and 11.2)

[fniephaus]

Thanks for the bug report, @ctiedt. I tried to reproduce this but it seems that I can debug a helloworld with gdb 12.1 on archlinux. Maybe @adinn has an idea what could be causing the Dwarf Error that you are seeing?

[adinn]

@ctiedt Hi Clemens

If you can provide me with a copy of the broken the binary and a dump of the debug info I may be able to to debug this and see what the broken compile unit is (I would prefer not to try dumping debug info on my own, non-Alpine box). To produce the dump output please run these commands

$ objdump --dwarf=info helloworld > helloworld.info.dw
$ objdump --dwarf=abbrev helloworld > helloworld.abbrev.dw
$ objdump -h helloworld > helloworld.headers.dw

I'm assuming that helloworld is the generated image and is in the current working directory. Also, that objdump is available on your build host.

I would be grateful if you could zip up the resulting output files, helloworld.info.dw, helloworld.abbrev.dw, helloworld.headers.dw plus the original binary helloworld and attach them to this issue.

[ctiedt]

Hi Andrew, thank you very much for looking into this. The files you requested are attached below. All the objdump commands also gave a warning Corrupt unit length (got 0x6c0000 expected at most 0xec2c9) in section .debug_info that was not included in the dumps.

helloworld-dbg.zip

[adinn]

@ctiedt Hi Clemens. Thanks for the files. Clearly something is wrong with the debug info layout as the info dump is empty. I am currently working my way through the binary parsing the records by eyeball. I'll let you know when I find the error.

[adinn]

Hi Clemens,

I have analysed the data present in the debug_info section of your binary and identified what is wrong with it. My analysis is available here.

In summary, the problem is not in the GraalVM debug info generator. It is in the linker that assembles the final ELF image. It's job is to merge the ELF object produced by GraalVM and various compiled C programs and that includes merging all of their debug info. Basically, it is getting a detail of the merge wrong, failing to update an offset. I explain the problem briefly below. Fixing it will require someone knowledgeable about the linker to take a look and diagnose what it is doing wrong and why.

I don't know whether this is a problem with the compiler chain you are using that is specific to Alpine Linux or a more general problem. Anyway, could you provide details of the compiler/linker installed on our system.

In order to explain what is going wrong I'll need to provide some context. The linker merges all the different sections from the various input objects to produxe the final image. As well as obvious sections like .text (code) and .data (read only or read write program memory values) it also merges sections containing debugger information. So, for example, the .debug_info sections from all the input objects are combined to produce an output .debug_info section.

These input .debug_info sections describe all the types, code and source code information for a specific compiled C file or library or for the Java classes in the image. The merged file includes all these descriptions in one .debug_info section of the executable. Some adjustment of the original content is needed because some of the data in the input sections uses offsets which get changed during merging.

Likewise the linker merges data for other sections such as the .debug_abbrev section. The abbrev section defines the record layouts used in the debug_info section. So each input info section has its own associated set of abbrev records. In order to match up info records and abbrev records before and after merging each block of records in the info section (a compile unit CU) identifies the point in the abbrev section where its records start. Before merging CUs usually specify abbrev section offset 0. That's standard because the abbrev section in their object file usually only has its own abbreviations. After merging each CU should be updated by the linker to specify the offset in the merged abbrev section where its own abbrev records start.

This is where the linker is going wrong. When it merges the GraalVM info section in with the info derived from compiled C and C++ files it is failing to update the abbrev section offset in the Graal CUs from 0 to 0x188. This means that objdump uses the abbrev layouts defined for some C/C++ file to parse the Java info records rather than using the ones the generator provided. As a result when it parses the first record it thinks there is one record of length 0x25 rather than a series of records with total length 0x6c. Since the header claims the length is 0x6c it blows up.

Parsing the record using the correct abbrev section offset shows that the content in the Graal CU is valid. So, the only real error with the GraalVM CUs is that the abbrev section offset has not been updated by the linker.

I am not clear why the linker is getting this wrong However, there are several things I noticed during the analysis that might be at play:

1. GraalVM is generating DWARF 4 records while the other compiled code is using DWARF 5. There ought to be no problem mixing sections with different DWARF levels but the linker may not be handling this correctly.
2. The headers for CUs 1 and 4 specify the associated source language with code 0x801. This is not a valid language code.
3. CU 4 includes content whose length matches the length specified in the CU header. However, there are 6 zero bytes of padding between the end of CU 4 and the start of CU 5. objdump appears to skip these padding bytes (treating them as empty compile units). However, their presence may indicate some error in the linker's merge algorithm that may be related to the failure to update the abbrev section offset.

[adinn]

@fniephaus @olpaw I am not clear what to do now. Essentially this is not a bug in GraalVM, it is a bug in the linker. What do you advise?

[ctiedt]

Hi Andrew, thank you so much for the in-depth analysis and explanation! I am using gcc 12.1 and ld 2.38. With --native-compiler-options and the -fuse-ld option in gcc, I also tried using mold and gold, both of which built working binaries but again with broken debug information.

[adinn]

@ctiedt Hi Clemens. Thanks for providing the extra details.

I am wondering if it might be possible to obtain debug output from the linker to help diagnose the problem. I'll consult my colleagues who work on gcc for advice and then play around with --native-compiler-options to see if any of the linker flags might generate useful info. If I find something I might ask you to do the same and ship me the build output.

[adinn]

@ctiedt I just took another look at the analysis and I noticed that I misread the language code for CUs 1 and 4. It is not 0x801 as I said but actually 0x8001. That's the code for 'MIPS assembler' input. I have no idea why your program links in two CUs that it thinks are derived frpm MIPS assembler sources. However, I recalled seeing another issue that involved a debug section with this as the source language type. A quick Google search led me to #3452 where essentially the same error was reported.

Your binary shows exactly the same error when I run the objdump command:

$ objdump --dwarf=info ~/tmp/debuginfo/helloworld
objdump: Warning: Corrupt unit length (0x6c0000) found in section .debug_info

helloworld:     file format elf64-x86-64

objdump: Warning: Corrupt unit length (0x6c0000) found in section .debug_info

The problem, as I showed in my dump analysis, is that CU 4 contains those extra 6 padding bytes. The first four are clearly treated as an empty CU (length = 0x0). The extra 2 cause the length of the Java base types CU to be read as

00 00 6c 00 = 0x6c0000

rather than

6c 00 00 00 = 0x6c

The root problem seems to be that whatever is generating the 'MIPS assembler' object is rounding up the debug info section to some power of 2 length. In the previous manifestation #3452 there were just two extra zero padding bytes. In this case there were 6. So, I suspect it is padding to an 8 byte boundary. Padding with 4 bytes would, of course, be innocuous.

The padding is harmless as far as any individual input object is concerned but it causes a problem when the linker merges the object before some other object since the zeros are then interpreted as belonging to the following CU and as a result the length field gets multiplied by 0x10000.