JVM crashes with grpc-netty-shaded 1.67.1 and alpine docker image

[jehervy]

An update of com.google.cloud dependency (which depends on grpc-netty-shaded:1.67.1) with a image built from eclipse-temurin:21-jre-alpine leaded to a segmentation fault:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000000000204b6, pid=1, tid=17
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_6417406470977119301460.so+0x2a154]  netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4
#
# Core dump will be written. Default location: /core.%e.1.%t
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log

As a workaround we had to downgrade to grpc-netty-shaded:1.65.1

Full error log: hs_err_pid1.log

[reinaldomjr]

Ths is happening on eclipse-temurin:17 also:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00000000000204b6, pid=1, tid=7
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.13+11 (17.0.13+11) (build 17.0.13+11)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.13+11 (17.0.13+11, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# C [libio_grpc_netty_shaded_netty_tcnative_linux_x86_643039151283623279840.so+0x2a154] netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4
#
# Core dump will be written. Default location: /core.%e.1.%t
#
# An error report file with more information is saved as:
# /app/hs_err_pid1.log

[ejona86]

eclipse-temurin:17?? That isn't Alpine. Alpine issues are common, but Ubuntu Temurin is very normal. I do see the address of the crash is the same, so likely related.

We either need to reproduce and try with various versions of netty-tcnative to try to find the change. Or we pull out objdump.

---

Investigation assuming the two errors are unrelated... (Spoiler: doesn't seem Alpine-specific.) Following the typical Alpine flow, it doesn't indeed seem related to glibc compatibility.

$ podman run --rm -it docker.io/eclipse-temurin:21-jre-alpine /bin/sh
/ # wget https://search.maven.org/remotecontent?filepath=io/netty/netty-tcnative-boringssl-static/2.0.65.Final/netty-tcnative-boringssl-static-2.0.65.Final-linux-
x86_64.jar
Connecting to search.maven.org (34.196.217.203:443)
Connecting to repo1.maven.org (199.232.192.209:443)
saving to 'netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar'
netty-tcnative-borin 100% |******************************************************************************************************************| 1187k  0:00:00 ETA
'netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar' saved
/ # 
/ # unzip netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar 
Archive:  netty-tcnative-boringssl-static-2.0.65.Final-linux-x86_64.jar
   creating: META-INF/
  inflating: META-INF/MANIFEST.MF
   creating: META-INF/license/
   creating: META-INF/maven/
   creating: META-INF/maven/io.netty/
   creating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/
   creating: META-INF/native/
  inflating: META-INF/LICENSE.txt
  inflating: META-INF/NOTICE.txt
  inflating: META-INF/license/LICENSE.aix-netbsd.txt
  inflating: META-INF/license/LICENSE.boringssl.txt
  inflating: META-INF/license/LICENSE.mvn-wrapper.txt
  inflating: META-INF/license/LICENSE.tomcat-native.txt
  inflating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.properties
  inflating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.xml
  inflating: META-INF/native/libnetty_tcnative_linux_x86_64.so
  inflating: META-INF/INDEX.LIST
/ # apk add gcompat
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz
(1/3) Installing musl-obstack (1.2.3-r2)
(2/3) Installing libucontext (1.2-r3)
(3/3) Installing gcompat (1.1.0-r4)
OK: 42 MiB in 73 packages
/ # LD_PRELOAD=/lib/libgcompat.so.0 ldd META-INF/native/libnetty_tcnative_linux_x86_64.so 
	/lib/ld-musl-x86_64.so.1 (0x7fb971f23000)
	/lib/libgcompat.so.0 => /lib/libgcompat.so.0 (0x7fb971f0b000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fb971f23000)
	libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7fb971f23000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fb971f23000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fb971f23000)
	libucontext.so.1 => /lib/libucontext.so.1 (0x7fb971f06000)
	libobstack.so.1 => /usr/lib/libobstack.so.1 (0x7fb971f01000)

Looking to see if the glibc symbols changed:

/ # ldd META-INF/native/libnetty_tcnative_linux_x86_64.so 
	/lib/ld-musl-x86_64.so.1 (0x7f50ccdf0000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f50ccdf0000)
	libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7f50ccdf0000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7f50ccdf0000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f50ccdf0000)
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __isnan: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __isinf: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __strdup: symbol not found

isnan and isinf are new. But no more need for libunwind:

/ # wget https://search.maven.org/remotecontent?filepath=io/netty/netty-tcnative-boringssl-static/2.0.61.Final/netty-tcnative-boringssl-static-2.0.61.Final-linux-
x86_64.jar
Connecting to search.maven.org (3.93.166.87:443)
Connecting to repo1.maven.org (199.232.196.209:443)
saving to 'netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar'
netty-tcnative-borin 100% |******************************************************************************************************************| 1218k  0:00:00 ETA
'netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar' saved
/ # unzip netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar 
Archive:  netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar
   creating: META-INF/
  inflating: META-INF/MANIFEST.MF
   creating: META-INF/license/
   creating: META-INF/maven/
   creating: META-INF/maven/io.netty/
   creating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/
   creating: META-INF/native/
  inflating: META-INF/LICENSE.txt
  inflating: META-INF/NOTICE.txt
  inflating: META-INF/license/LICENSE.aix-netbsd.txt
  inflating: META-INF/license/LICENSE.boringssl.txt
  inflating: META-INF/license/LICENSE.mvn-wrapper.txt
  inflating: META-INF/license/LICENSE.tomcat-native.txt
  inflating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.properties
  inflating: META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.xml
  inflating: META-INF/native/libnetty_tcnative_linux_x86_64.so
  inflating: META-INF/INDEX.LIST
/ # ldd META-INF/native/libnetty_tcnative_linux_x86_64.so 
	/lib/ld-musl-x86_64.so.1 (0x7ff60d448000)
	libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7ff60d448000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7ff60d448000)
	ld-linux-x86-64.so.2 => /lib/ld-linux-x86-64.so.2 (0x7ff60d43b000)
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_GetRegionStart: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_RaiseException: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_SetIP: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_GetLanguageSpecificData: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_GetTextRelBase: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_Resume_or_Rethrow: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __strdup: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_GetIPInfo: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_Resume: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_SetGR: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_DeleteException: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: _Unwind_GetDataRelBase: symbol not found

[lorand-lorincz-emnos]

We have the same issue. We are using eclipse-temurin:17.0.13_11-jdk-alpine.
I can confirm, forcing grpc-netty-shaded:1.65.1 works as workaround

[ejona86]

I've reproduced the problem on Alpine. I haven't reproduced it on eclipse-temurin:17; @reinaldomjr, can you double-check what image you are using?

The problem seems limited to grpc-netty-shaded. I couldn't reproduce it with grpc-netty.

Looking at objdump, it does point a finger at grpc-netty-shaded, with staticPackagePrefix which is different between the two (0x2a154 is the location of the crash):

   2a14c:       4c 89 ef                mov    %r13,%rdi
   2a14f:       e8 5c 63 ff ff          call   204b0 <__strdup@plt>
   2a154:       48 89 05 ed 1d 41 00    mov    %rax,0x411ded(%rip)        # 43bf48 <staticPackagePrefix>
   2a15b:       e9 4f fa ff ff          jmp    29baf <netty_internal_tcnative_SSLContext_JNI_OnLoad+0x41f>

$ cd grpc-java/examples/example-tls
$ git checkout v1.67.1
$ ../../gradlew installDist
$ ./build/install/example-tls/bin/hello-world-tls-server 50440 ../../testing/src/main/resources/certs/server1.pem ../../testing/src/main/resources/certs/server1.key ../../testing/src/main/resources/certs/ca.pem

## in another terminal
$ cd grpc-java
$ podman run --rm -it -v .:/files --network=host eclipse-temurin:21-jre-alpine /bin/sh
/ # cd files/examples/example-tls/
/files/examples/example-tls # ./build/install/example-tls/bin/hello-world-tls-client localhost 50440 ../../testing/src/main/resources/certs/ca.pem ../../testing/src/main/resources/certs/client.pem ../../testing/src/main/resources/certs/client.key
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000000000204b6, pid=6, tid=16
...

[ejona86]

Actually, my test was busted because I didn't use LD_PRELOAD. Because it crashed for me in the same place as you, this looks to be a regular "it crashes on Alpine if you don't use gcompat." Are y'all on alpine running with apk add gcompat and LD_PRELOAD=/lib/libgcompat.so.0? That's been needed for a while. I'm surprised it worked before. When I run with that it works fine:

/files/examples/example-tls # apk add gcompat
(1/3) Installing musl-obstack (1.2.3-r2)
(2/3) Installing libucontext (1.2-r3)
(3/3) Installing gcompat (1.1.0-r4)
OK: 55 MiB in 79 packages
/files/examples/example-tls # LD_PRELOAD=/lib/libgcompat.so.0 ./build/install/example-tls/bin/hello-
world-tls-client localhost 50440 ../../testing/src/main/resources/certs/ca.pem ../../testing/src/mai
n/resources/certs/client.pem ../../testing/src/main/resources/certs/client.key
Nov 11, 2024 11:23:24 PM io.grpc.examples.helloworldtls.HelloWorldClientTls greet
INFO: Will try to greet localhost ...
Nov 11, 2024 11:23:25 PM io.grpc.examples.helloworldtls.HelloWorldClientTls greet
INFO: Greeting: Hello localhost

Showing a bit more objdump context, it seems this is the source code location for the crash (0x2a154):

   2a131:       48 89 05 18 1e 41 00    mov    %rax,0x411e18(%rip)        # 43bf50 <sslPrivateKeyMet
hodDecryptTask_init>
   2a138:       0f 84 2d 01 00 00       je     2a26b <netty_internal_tcnative_SSLContext_JNI_OnLoad+
0xadb>
   2a13e:       4d 85 ed                test   %r13,%r13
   2a141:       bb 06 00 01 00          mov    $0x10006,%ebx
   2a146:       0f 84 63 fa ff ff       je     29baf <netty_internal_tcnative_SSLContext_JNI_OnLoad+
0x41f>
   2a14c:       4c 89 ef                mov    %r13,%rdi
   2a14f:       e8 5c 63 ff ff          call   204b0 <__strdup@plt>
   2a154:       48 89 05 ed 1d 41 00    mov    %rax,0x411ded(%rip)        # 43bf48 <staticPackagePrefix>
   2a15b:       e9 4f fa ff ff          jmp    29baf <netty_internal_tcnative_SSLContext_JNI_OnLoad+0x41f>

    NETTY_JNI_UTIL_GET_METHOD(env, sslPrivateKeyMethodDecryptTask_class, sslPrivateKeyMethodDecryptTask_init, "<init>", name, error);

    if (packagePrefix != NULL) {
        staticPackagePrefix = strdup(packagePrefix);
    }

Nothing there has changed in a long time. And that's the only reference to __strdup. So this just seems like a regular glibc compat issue solved by gcompat.

[ejona86]

Seems like this is resolved (use LD_PRELOAD=/lib/libgcompat.so.0 like normal). If not, comment, and it can be reopened.

[bcwhite-code]

Was a bug filed against the netty-tcnative library (as reported here that is the source of the problem?

[ejona86]

Filed for what? It has always been built with glibc. It used to not work with Alpine at all and then Alpine added gcompat.

[bcwhite-code]

The details of dynamic linking are beyond my experience. If netty is linked against "libc" then there is presumably some defined ABI for "libc" that it either netty is going beyond or musl is not fulfilling. If it's linked against "glibc" and musl is declaring itself as "glibc compatible" then it's definitely something that musl is declaring but not fulfilling.

The inclusion of "gcompat" could be interpreted either way.

I bring this up because I'm running on a 512MiB virtual machine and adding an additional 42MiB to apk add gcompat is a significant hit. Though the libgcompat.so.0 is only 66K so maybe there's a smaller way to install it...

[ejona86]

There is no "libc ABI" in a general sense. The only guarantee is running with the libc you compiled against, and often newer is made to work. There is code that detects broken loading of the .so and falls back to other approaches, but I think because Java uses RTLD_LAZY then the missing symbols aren't noticed at load time. (It is possible for a binary to mark symbols as lazy/deferred, but I'm not immediately seeing that being done in netty-tcnative.) In glibc when a deferred symbol fails to load it crashes, but also gives a message telling you which symbol was missing. musl just crashes, since it is icky for libc to print messages.

Note that "normal" usages of glibc-compiled code running on Alpine would fail to run immediately if gcompat is not installed, because the glibc linker is missing. gcompat implements a fake glibc linker that also brings in extra glibc symbols. But when using Java compiled for Alpine, the Java process itself uses the regular musl linker. Then when the netty-tcnative .so is loaded the linker is ignored and the extra symbols aren't added.

42 MiB seems way too much. I wonder if you updated a bunch of other packages when you installed it. I see it as around 3 MiB, without deleting the apk cache.

[bcwhite-code]

Thank you for the explanation. Some things are just the way they are.

After extracting only the 5 extra libs that apk add gcompat adds and installing only those (along with a half-dozen symlinks), the added content is only 140KiB, which is acceptable. If there is something I did not include, it isn't (yet) required for my app.

For reference, that is:

/lib/ld-linux-x86-64.so.2
/lib/libgcompat.so.0
/lib/libucontext.so.1
/lib/libucontext_posix.so.1
/usr/lib/libobstack.so.1.0.0

and

cd /lib
ln -s libgcompat.so.0 libc.so.6
ln -s libgcompat.so.0 libcrypt.so.1
ln -s libgcompat.so.0 libm.so.6
ln -s libgcompat.so.0 libpthread.so.0
ln -s libgcompat.so.0 libresolv.so.2
ln -s libgcompat.so.0 librt.so.1
ln -s libgcompat.so.0 libutil.so.1
cd /usr/lib
ln -s libobstack.so.1.0.0 libobstack.so.1