JVM crash with grpc-java 1.42.x and alpine docker image

[Spikhalskiy]

An attempt to upgrade from grpc-java 1.41.1 to 1.42.x ends with JVM crash.
It looks like the problem is specific to Alpine Linux. It reproduces on openjdk:15-jdk-alpine and openjdk:8-alpine and goes away with a switch on openjdk:X-slim [debian] images.
Maybe also affected by the fact that openjdk:X-alpine images are not maintained anymore, hence have no new JDK updates.
The first version of grpc-java with the problem is 1.42.0, the versions before work fine.

It may be related to grpc/grpc#27995

    #
  | # A fatal error has been detected by the Java Runtime Environment:
  | #
  | #  SIGSEGV (0xb) at pc=0x0000000000003efe, pid=372, tid=0x00007fbffbc9bb10
  | #
  | # JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
  | # Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
  | # Derivative: IcedTea 3.12.0
  | # Distribution: Custom build (Sat May  4 17:33:35 UTC 2019)
  | # Problematic frame:
  | # C  0x0000000000003efe
  | #
  | # Core dump written. Default location: /temporal-java-client/temporal-kotlin/core or core.372
  | #
  | # An error report file with more information is saved as:
  | # /temporal-java-client/temporal-kotlin/hs_err_pid372.log
  | #
  | # If you would like to submit a bug report, please include
  | # instructions on how to reproduce the bug and visit:
  | #   https://icedtea.classpath.org/bugzilla
  | #

hs_err_pid372.log

[ejona86]

Without the crash report or reproduction, there's not much we can do. The crash report would let us verify the failure is within netty-tcnative or netty's epoll, for example. Without knowing the source of crash, we can't really narrow much down. The problem may exist for non-Alpine as well, just doesn't result in a crash due to slightly different memory layout/library versions.

It is interesting that C core also started seeing a recent crash, but there's not much in that report either. It is strange that the program counter is at a very low address though, in both cases.

I fired up openjdk:8-alpine (a3562aa0b991), installed gcompat and libc6-compat, and ran grpc's interop-client[1] to hit an HTTPS server. No crashes.

/grpc-interop-testing-1.42.1 # ./bin/test-client --server_host=grpc-test.sandbox.googleapis.com --server_port=443
Running test empty_unary
Test completed.
/grpc-interop-testing-1.42.1 # # Running ./bin/test-server --use_tls=false outside the container
/grpc-interop-testing-1.42.1 # ./bin/test-client --server_host=my.host.ip.address --use_tls=false
Running test empty_unary
Test completed.

1. https://repo1.maven.org/maven2/io/grpc/grpc-interop-testing/1.42.1/grpc-interop-testing-1.42.1.tar

[Spikhalskiy]

@ejona86 This makes sense.
Here is the crash report: hs_err_pid372.log
Thank you for the quick turnaround and an attempt to reproduce.

[cfredri4]

I've started seeing this also, but only on Alpine in combination with Kubernetes (specifically I used azul/zulu-openjdk-alpine:17-jre-headless with no other changes, and Kubernetes 1.20). It works fine with an Ubuntu image on same Kubernetes environment, and it works fine with the Alpine image on a local Docker environment.

[ejona86]

@Spikhalskiy, are you also seeing this with k8s?

@cfredri4, do you know what OS your k8s Node is using?

[Spikhalskiy]

@ejona86 yeah.
This crash log that I attached is from build container spawned by buildkite CI/CD that uses agents deployed on aws k8s.
Another crash that we saw with openjdk:15-alpine is from a docker pod that uses the image (also aws k8s)

[djtuBIG-MaliceX]

Could this be related to netty/netty#11701 as a project I'm working in has been experiencing similar issues which appear to indicate the netty build being pulled in by grpc-netty-shaded breaking in alpine Docker bases.

[cfredri4]

Could this be related to netty/netty#11701 as a project I'm working in has been experiencing similar issues which appear to indicate the netty build being pulled in by grpc-netty-shaded breaking in alpine Docker bases.

Interesting. Changing from azul/zulu-openjdk-alpine:17-jre-headless (musl) to bellsoft/liberica-openjre-alpine:17 (glibc) made the problem go away for me.

[Spikhalskiy]

Could this be related to netty/netty#11701 as a project I'm working in has been experiencing similar issues which appear to indicate the netty build being pulled in by grpc-netty-shaded breaking in alpine Docker bases.

Not likely. We have this issue on an alpine image with installed glibc. Also, the image works with grpc-java (,1.42.0) and doesn't work with [1.42.0,). So, the problem doesn't look like just an absent glibc and not handling it by netty gracefully.

[ejona86]

TL;DR: Alpine doesn't have compatibility for the __strndup symbol. I don't know why the behavior is k8s-dependent, though. And it'll take some more research to determine appropriate next steps.

Looking at objdump output, it looks like the problem is happening in parsePackagePrefix.

libio_grpc_netty_shaded_netty_transport_native_epoll_x86_642526953976876250345.so+0xb487:

    b47c:	48 89 de             	mov    %rbx,%rsi
    b47f:	4c 89 e7             	mov    %r12,%rdi
    b482:	e8 59 fe ff ff       	call   b2e0 <parsePackagePrefix>
    b487:	83 7d dc ff          	cmpl   $0xffffffff,-0x24(%rbp)

But I don't see any obvious places the stack could get corrupted in parsePackagePrefix and there's no callout to a passed function. I'm suspicious though that the linker is broken. This is in parsePackagePrefix:

    b395:	e8 5e 8b ff ff       	call   3ef8 <__strndup@plt>

The address displayed is relative to the .so, so isn't the problem itself. It jumps to the PLT:

0000000000003ef8 <__strndup@plt>:
    3ef8:       ff 25 52 c2 20 00       jmp    *0x20c252(%rip)        # 210150 <__strndup@GLIBC_2.2.5>
    3efe:       68 36 00 00 00          push   $0x36
    3f03:       e9 80 fc ff ff          jmp    3b88 <.plt>

And the indirect jump goes to the GOT which should be filled with the adjusted address of 3efe. But maybe something is broken in the linker and it didn't get adjusted?

/ # apk add gcompat
/ # ldd libnetty_transport_native_epoll_x86_64-4.1.63.so 
	ldd (0x7fa4cfc2f000)
	librt.so.1 => ldd (0x7fa4cfc2f000)
	libdl.so.2 => ldd (0x7fa4cfc2f000)
	libc.so.6 => ldd (0x7fa4cfc2f000)
Error relocating libnetty_transport_native_epoll_x86_64-4.1.63.so: __strndup: symbol not found

Well, there we go... Older versions of epoll linked against strndup, not __strndup. This difference may have been caused by a glibc upgrade when compiling.

[ejona86]

TL;DR: Try setting the LD_PRELOAD=/lib/libgcompat.so.0 environment variable on old Alpine versions, so gcompat is actually used. Hopefully that won't break any of the musl binaries. Best approach is probably "upgrade Alpine to 3.13 or later" though. (Edit: New versions exhibit similar behavior, because this flow avoids libc6-compat glue)

openjdk:8-alpine is based on alpine 3.9.4, and using apline:3.9.4 directly produces similar ldd results. Interestingly apline:3.15.0 (latest) has more unresolved symbols:

/ # ldd libnetty_transport_native_epoll_x86_64-4.1.63.so 
	/lib/ld-musl-x86_64.so.1 (0x7f7bfe2f5000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f7bfe2f5000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7f7bfe2f5000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f7bfe2f5000)
Error relocating libnetty_transport_native_epoll_x86_64-4.1.63.so: __strdup: symbol not found
Error relocating libnetty_transport_native_epoll_x86_64-4.1.63.so: __strndup: symbol not found

But I think that isn't the full story. Looking at the older Alpine:

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.9.4
...
/ # apk info -L gcompat libc6-compat
gcompat-0.3.0-r0 contains:
lib/ld-linux-x86-64.so.2
lib/libgcompat.so.0

libc6-compat-1.1.20-r6 contains:
lib/libm.so.6
lib/libutil.so.1
lib/libc.so.6
lib/libpthread.so.0
lib/libcrypt.so.1
lib/librt.so.1
lib64/ld-linux-x86-64.so.2
/ # ls -l lib64/ld-linux-x86-64.so.2
lrwxrwxrwx    1 root     root            26 Dec 20 18:30 lib64/ld-linux-x86-64.so.2 -> /lib/libc.musl-x86_64.so.1

lib64/ld-linux-x86-64.so.2 is used for x86_64, not lib/ld-linux-x86-64.so.2. So I suspect gcompat is useless on this older version. Instead, libc6-compat is providing the linker which just forwards to the musl linker, and is missing the symbol. So I think ldd on this older Alpine is accurate.

/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
...
/ # apk info -L gcompat libc6-compat
gcompat-1.0.0-r4 contains:
lib/ld-linux-x86-64.so.2
lib/libgcompat.so.0
lib64/ld-linux-x86-64.so.2

libc6-compat-1.2.2-r7 contains:
lib/libc.so.6
lib/libcrypt.so.1
lib/libm.so.6
lib/libpthread.so.0
lib/librt.so.1
lib/libutil.so.1

/ # ls -l lib64/ld-linux-x86-64.so.2
lrwxrwxrwx    1 root     root            27 Dec 20 18:57 lib64/ld-linux-x86-64.so.2 -> ../lib/ld-linux-x86-64.so.2

Here though, gcompat is providing the linker which loads lib/libgcompat.so.0. That means I don't think the ldd output is accurate. I see that gcompat 0.3.0 (Alpine 3.9) and 1.0.0 (Alpine 3.15) have __strndup, so I think the wrong linker on older Alpine versions is the trouble. Trying the gcompat linker approach "manually" on the old Alpine seems to work:

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.9.4
...
/ # LD_PRELOAD=/lib/libgcompat.so.0 ldd libnetty_transport_native_epoll_x86_64-4.1.63.so 
	ldd (0x7f72fc25e000)
	/lib/libgcompat.so.0 => /lib/libgcompat.so.0 (0x7f72fc03c000)
	librt.so.1 => ldd (0x7f72fc25e000)
	libdl.so.2 => ldd (0x7f72fc25e000)
	libc.so.6 => ldd (0x7f72fc25e000)