[SPARK-51301][BUILD] Bump zstd-jni 1.5.7-3

[pan3793]

What changes were proposed in this pull request?

Bump zstd-jni to the latest version.

Why are the changes needed?

https://github.com/facebook/zstd/releases/tag/v1.5.7
luben/zstd-jni@v1.5.6-10...v1.5.7-3

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GHA, also update the benchmark results.

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

cc @dongjoon-hyun @LuciferYang

[LuciferYang]

cc @yaooqinn Will the results of this test also fluctuate?

[yaooqinn]

Results seem reasonable

[LuciferYang]

Got it ~

[beliefer]

The degradation looks obviously. cc @yaooqinn

[yaooqinn]

Can you run these benchmarks one or two times more? @pan3793

[pan3793]

3 times on this PR branch

• https://github.com/pan3793/spark/actions/runs/13496989883
• https://github.com/pan3793/spark/actions/runs/13496995223
• https://github.com/pan3793/spark/actions/runs/13496999794

3 times on master branch

• https://github.com/pan3793/spark/actions/runs/13497001661
• https://github.com/pan3793/spark/actions/runs/13497002797
• https://github.com/pan3793/spark/actions/runs/13497003849

[beliefer]

It looks worse on level 9.

[pan3793]

I managed to run the benchmark on a local Linux machine, it does have perf regression on level 9 parallel compression

master

OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.9.3-76060903-generic
Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz
Parallel Compression at level 9:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers                 162            164           1          0.0     1263562.4       1.0X
Parallel Compression with 1 workers                 203            205           1          0.0     1587511.9       0.8X
Parallel Compression with 2 workers                 121            124           2          0.0      944964.2       1.3X
Parallel Compression with 4 workers                 110            118           6          0.0      861307.6       1.5X
Parallel Compression with 8 workers                 130            135           4          0.0     1018503.5       1.2X
Parallel Compression with 16 workers                162            164           2          0.0     1263810.5       1.0X

this patch

OpenJDK 64-Bit Server VM 17.0.13+11-LTS on Linux 6.9.3-76060903-generic
Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz
Parallel Compression at level 9:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Parallel Compression with 0 workers                 224            226           2          0.0     1751380.5       1.0X
Parallel Compression with 1 workers                 255            257           2          0.0     1990519.8       0.9X
Parallel Compression with 2 workers                 146            147           1          0.0     1143136.1       1.5X
Parallel Compression with 4 workers                 116            124           8          0.0      905375.1       1.9X
Parallel Compression with 8 workers                 133            139           4          0.0     1042289.8       1.7X
Parallel Compression with 16 workers                161            164           2          0.0     1258236.8       1.4X

[beliefer]

Shall we give luben/zstd-jni/releases/tag/v1.5.7-1 a feedback ?

[pan3793]

I have opened an issue luben/zstd-jni#350

[luben]

Do you know what is the compression ratio achieved with both versions? Is it similar?

[pan3793]

A little busy these days, I will investigate this issue later this week.

for perf

[dongjoon-hyun]

Thank you for investigating and reporting back to the upstream, @pan3793 and all.

[LuciferYang]

@pan3793 v1.5.7-2 released

[luben]

@pan3793 v1.5.7-2 released

This adds only one feature on top on v1.5.7-1: Support decompression from byte array to ByteBuffer and vice-versa.

[dongjoon-hyun]

FYI, v1.5.7-3 is ready and looks much better to me than v1.5.7-1.

• https://github.com/luben/zstd-jni/releases/tag/v1.5.7-3
• luben/zstd-jni@v1.5.7-1...v1.5.7-3

[dongjoon-hyun]

Maybe, do you want to re-try with 1.5.7-3, @pan3793 ?

[dongjoon-hyun]

Thank you, @pan3793 . Please revise the PR title and description too.

[pan3793]

got similar benchmark result with zstd-jni:1.5.7-3, investigating

[luben]

got similar benchmark result with zstd-jni:1.5.7-3, investigating

I don't expect performance differences between 1.5.7-1 and 1.5.7-3 as they are based on the same zstd upstream code.

[pan3793]

Hi @luben, I mirrored the Spark benchmark to a single Java file, would be great if you can take a look

Main.java

import com.github.luben.zstd.BufferPool;
import com.github.luben.zstd.NoPool;
import com.github.luben.zstd.RecyclingBufferPool;
import com.github.luben.zstd.ZstdOutputStreamNoFinalizer;

import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;

public class Main {

    static boolean bufferPoolEnabled = false;

    static BufferPool bufferPool = bufferPoolEnabled ? RecyclingBufferPool.INSTANCE : NoPool.INSTANCE;

    static byte[] data = new byte[256 * 1024 * 1024];

    static {
        for (int i = 0; i < 256 * 1024 * 1024; i++) {
            data[i] = (byte) i;
        }
    }

    public static void main(String[] args) throws Exception {
        long start = System.nanoTime();
        for (int j = 0; j < 16; j++) {
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            ZstdOutputStreamNoFinalizer os = new ZstdOutputStreamNoFinalizer(baos, bufferPool)
                    .setLevel(9)
                    .setWorkers(0)
                    .setCloseFrameOnFlush(true);
            OutputStream zcos = new BufferedOutputStream(os, 32 * 1024);
            ObjectOutputStream oos = new ObjectOutputStream(zcos);
            for (int i = 0; i < 65535; i++) {
                oos.writeObject(data);
            }
            oos.close();
        }
        System.out.println("cost " + (System.nanoTime() - start) / 1000000L + "ms");
    }
}

➜  ~ java -version
openjdk version "17.0.15" 2025-04-15 LTS
OpenJDK Runtime Environment Zulu17.58+21-CA (build 17.0.15+6-LTS)
OpenJDK 64-Bit Server VM Zulu17.58+21-CA (build 17.0.15+6-LTS, mixed mode, sharing)

➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar Main.java
cost 2654ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar Main.java
cost 2643ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar Main.java
cost 2672ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar Main.java
cost 2668ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.6-10/zstd-jni-1.5.6-10.jar Main.java
cost 2589ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.7-3/zstd-jni-1.5.7-3.jar Main.java
cost 3630ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.7-3/zstd-jni-1.5.7-3.jar Main.java
cost 3689ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.7-3/zstd-jni-1.5.7-3.jar Main.java
cost 3682ms
➜  ~ java -cp ~/.m2/repository/com/github/luben/zstd-jni/1.5.7-3/zstd-jni-1.5.7-3.jar Main.java
cost 3706ms

[luben]

Thanks, I can reproduce it locally. I will look into the details

[luben]

I think it's some change in the underlying algorithm. I can reproduce that also with the zstd CLI in the data sample you are using:

$ time zstd-1.5.6 --single-thread -9 data_file_16 -c > /dev/null
zstd-1.5.6 --single-thread -9 data_file_16 -c > /dev/null  3,23s user 0,71s system 123% cpu 3,196 total

$ time zstd-1.5.7 --single-thread -9 data_file_16 -c > /dev/null
zstd-1.5.7 --single-thread -9 data_file_16 -c > /dev/null  3,79s user 0,64s system 116% cpu 3,794 total

the data_file_16 is 4 GiB file that is using the same data as the Java benchmark. BTW, you don't need that for (int i = 0; i < 65535; i++) loop there.

[luben]

To be fair, using the CLI I cannot reproduce that difference using real-world data (used some log files for testing).

[pan3793]

I think it's some change in the underlying algorithm. I can reproduce that also with the zstd CLI in the data sample you are using

@luben thank you for confirming and clarifying.

To be fair, using the CLI I cannot reproduce that difference using real-world data (used some log files for testing).

@yaooqinn @dongjoon-hyun since you wrote the ZStandardBenchmark, I'd like to hear your opinion.

[luben]

I will rise the issue with the upstream. BTW, here is minimized benchmark that still exhibits the same difference:

  import com.github.luben.zstd.ZstdOutputStreamNoFinalizer;
  import java.io.ByteArrayOutputStream;

  public class Main {

      static byte[] data = new byte[256 * 1024 * 1024];

      static {
          for (int i = 0; i < 256 * 1024 * 1024; i++) {
              data[i] = (byte) i;
          }
      }

      public static void main(String[] args) throws Exception {
          long start = System.nanoTime();
          for (int j = 0; j < 16; j++) {
              ByteArrayOutputStream baos = new ByteArrayOutputStream();
              ZstdOutputStreamNoFinalizer os = new ZstdOutputStreamNoFinalizer(baos)
                      .setLevel(9)
                      .setWorkers(0);
              os.write(data);
              os.close();
          }
          System.out.println("cost " + (System.nanoTime() - start) / 1000000L + "ms");
      }
  }

[luben]

There is explanation in facebook/zstd#4385 and it's what I suspected - the highly compressible data used in the benchmark is not representative of real world data.

[LuciferYang]

So should we first rewrite parts of this microbenchmark to make it more aligned with real-world scenarios before proceeding with the upgrades?

[pan3793]

@LuciferYang I'm trying to rewrite the ZStandardBenchmark with TPC-DS data, and already got a reasonable result on local test, will create a new PR soon.

[dongjoon-hyun]

+1, LGTM.