[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final #44384

panbingkun · 2023-12-17T08:53:35Z

What changes were proposed in this pull request?

The pr aims to upgrade Netty from 4.1.100.Final to 4.1.106.Final.

Why are the changes needed?

To bring the latest bug fixes
Automatically close Http2StreamChannel when Http2FrameStreamExceptionreaches end ofChannelPipeline (#13651)
Symbol not found: _netty_jni_util_JNI_OnLoad (#13695)
4.1.106.Final release note: https://netty.io/news/2024/01/19/4-1-106-Final.html
4.1.105.Final release note: https://netty.io/news/2024/01/16/4-1-105-Final.html
4.1.104.Final release note: https://netty.io/news/2023/12/15/4-1-104-Final.html
4.1.103.Final release note: https://netty.io/news/2023/12/13/4-1-103-Final.html
4.1.101.Final release note: https://netty.io/news/2023/11/09/4-1-101-Final.html

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun

Could you check the CI failures?

panbingkun · 2023-12-18T01:46:49Z

Could you check the CI failures?

Okay, let me check it, I'll temporarily change PR to draft first.

LuciferYang · 2023-12-18T02:46:28Z

[info] - applyInPandasWithState should require StatefulOpClusteredDistribution from children - without initial state *** FAILED *** (2 seconds, 622 milliseconds)
[info]   org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = 7231fcdb-98d1-4627-858e-55f020fb52bd, runId = 70165d3c-1a0e-4d20-a73f-1d4658698069] terminated with exception: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 7) (localhost executor driver): java.lang.NoSuchFieldError: chunkSize
[info] 	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.<init>(PooledByteBufAllocatorL.java:153)
[info] 	at io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:49)
[info] 	at org.apache.arrow.memory.NettyAllocationManager.<clinit>(NettyAllocationManager.java:51)
[info] 	at org.apache.arrow.memory.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
[info] 	at java.base/java.lang.Class.forName0(Native Method)
[info] 	at java.base/java.lang.Class.forName(Class.java:375)

java.lang.NoSuchFieldError: chunkSize... It seems that the change of Netty has caused compatibility issues with Arrow again ...

panbingkun · 2023-12-18T02:48:59Z

[info] - applyInPandasWithState should require StatefulOpClusteredDistribution from children - without initial state *** FAILED *** (2 seconds, 622 milliseconds)
[info]   org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = 7231fcdb-98d1-4627-858e-55f020fb52bd, runId = 70165d3c-1a0e-4d20-a73f-1d4658698069] terminated with exception: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 7) (localhost executor driver): java.lang.NoSuchFieldError: chunkSize
[info] 	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.<init>(PooledByteBufAllocatorL.java:153)
[info] 	at io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:49)
[info] 	at org.apache.arrow.memory.NettyAllocationManager.<clinit>(NettyAllocationManager.java:51)
[info] 	at org.apache.arrow.memory.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
[info] 	at java.base/java.lang.Class.forName0(Native Method)
[info] 	at java.base/java.lang.Class.forName(Class.java:375)

java.lang.NoSuchFieldError: chunkSize... It seems that the change of Netty has caused compatibility issues with Arrow again ...

Yes, I am fixing this issue.

panbingkun · 2023-12-18T02:56:43Z

A new pr for fix issued about arrow-memory-netty: apache/arrow#39266

dongjoon-hyun · 2023-12-18T19:51:43Z

Thank you for the info, @panbingkun .

LuciferYang · 2024-01-23T14:00:50Z

@panbingkun Please rebase this PR, and I think the upgrade target can be 4.1.106, we can reuse this ticket

dongjoon-hyun · 2024-01-23T15:57:17Z

+1 for @LuciferYang 's comment.

panbingkun · 2024-01-24T10:54:07Z

Done.

dongjoon-hyun

Please regenerate the dependency file. It seems that there is a new entry like the following.

netty-transport-native-epoll/4.1.106.Final/linux-riscv64/netty-transport-native-epoll-4.1.106.Final-linux-riscv64.jar

panbingkun · 2024-01-25T01:42:29Z

netty-transport-native-epoll-4.1.106.Final-linux-riscv64

Yeah, I analyzed the dependency using the following command

build/mvn -Phive-thriftserver -Pkubernetes -Pyarn -Phive -Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud -Phadoop-3 dependency:tree -pl assembly -am

and found that it is dependent on io.netty:netty-transport-classes-kqueue:jar:4.1.106.Final

So, let's add it.

LuciferYang · 2024-01-25T02:49:14Z

Can Spark run on the riscv64 machine? Can we exclude this dependency? If not, we still need to fix network-yarn/pom.xml:

spark/common/network-yarn/pom.xml

Lines 175 to 185 in b86053c

    
           <configuration> 
        
               <target> 
        
                   <echo message="Shade netty native libraries to ${spark.shade.native.packageName}" /> 
        
                   <unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/" /> 
        
                   <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so" 
        
                         tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" /> 
        
                   <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib" 
        
                         tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib" /> 
        
                   <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so" 
        
                         tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" /> 
        
                   <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"

panbingkun · 2024-01-25T11:02:21Z

common/network-yarn/pom.xml

+                  tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" />
+                <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
+                  tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib" />
+                <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_riscv64.so"


Only this entry has been added, and the rest has been adjusted for indentation, from 4 spaces to 2 spaces.

panbingkun · 2024-01-25T11:02:39Z

Can Spark run on the riscv64 machine? Can we exclude this dependency? If not, we still need to fix network-yarn/pom.xml:

spark/common/network-yarn/pom.xml

Lines 175 to 185 in b86053c

<configuration>

<target>

<echo message="Shade netty native libraries to ${spark.shade.native.packageName}" />

<unzip src="${shuffle.jar}" dest="${project.build.directory}/exploded/" />

<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so"

tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" />

<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"

tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib" />

<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so"

tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" />

<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"

Done.

LuciferYang · 2024-01-25T12:35:08Z

In fact, I am more inclined to exclude this dependency. Because we do not yet have the corresponding CI to verify the usability of Apache Spark on RISC-V, so I personally think it is not supported on this architecture for the time being.

dongjoon-hyun

+1, LGTM. I'm fine with new entry. It doesn't mean Apache Spark claims any new additional architecture support. Could you reconsider your decision, @LuciferYang ?

netty-transport-native-epoll/4.1.106.Final/linux-aarch_64/netty-transport-native-epoll-4.1.106.Final-linux-aarch_64.jar
netty-transport-native-epoll/4.1.106.Final/linux-riscv64/netty-transport-native-epoll-4.1.106.Final-linux-riscv64.jar
netty-transport-native-epoll/4.1.106.Final/linux-x86_64/netty-transport-native-epoll-4.1.106.Final-linux-x86_64.jar

LuciferYang

ok, let's add this new dependency.

LuciferYang · 2024-01-26T03:13:34Z

Merged into master for Spark 4.0. Thanks @panbingkun and @dongjoon-hyun ~

dongjoon-hyun · 2024-01-26T03:39:33Z

Thank you, @panbingkun and @LuciferYang !

The pr aims to upgrade `Netty` from `4.1.100.Final` to `4.1.106.Final`. - To bring the latest bug fixes Automatically close Http2StreamChannel when Http2FrameStreamExceptionreaches end ofChannelPipeline ([apache#13651](netty/netty#13651)) Symbol not found: _netty_jni_util_JNI_OnLoad ([apache#13695](netty/netty#13728)) - 4.1.106.Final release note: https://netty.io/news/2024/01/19/4-1-106-Final.html - 4.1.105.Final release note: https://netty.io/news/2024/01/16/4-1-105-Final.html - 4.1.104.Final release note: https://netty.io/news/2023/12/15/4-1-104-Final.html - 4.1.103.Final release note: https://netty.io/news/2023/12/13/4-1-103-Final.html - 4.1.101.Final release note: https://netty.io/news/2023/11/09/4-1-101-Final.html No. Pass GA. No. Closes apache#44384 from panbingkun/SPARK-46432. Lead-authored-by: panbingkun <[email protected]> Co-authored-by: panbingkun <[email protected]> Signed-off-by: yangjie01 <[email protected]>

[SPARK-46432][BUILD] Upgrade Netty to 4.1.104.Final

4d7f0f8

github-actions bot added the BUILD label Dec 17, 2023

dongjoon-hyun reviewed Dec 17, 2023

View reviewed changes

panbingkun marked this pull request as draft December 18, 2023 01:41

[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final

3e4d47c

panbingkun changed the title ~~[SPARK-46432][BUILD] Upgrade Netty to 4.1.104.Final~~ [SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final Jan 24, 2024

dongjoon-hyun reviewed Jan 24, 2024

View reviewed changes

panbingkun added 2 commits January 25, 2024 09:26

[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final

c073bd8

Merge branch 'master' into SPARK-46432

879cb46

panbingkun marked this pull request as ready for review January 25, 2024 01:42

panbingkun added 2 commits January 25, 2024 18:59

[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final

2beb166

Merge branch 'master' into SPARK-46432

aa0f416

panbingkun commented Jan 25, 2024

View reviewed changes

dongjoon-hyun approved these changes Jan 25, 2024

View reviewed changes

LuciferYang approved these changes Jan 26, 2024

View reviewed changes

LuciferYang closed this in 44b163d Jan 26, 2024

[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final #44384

[SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final #44384

Uh oh!

Conversation

panbingkun commented Dec 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Dec 18, 2023

Uh oh!

LuciferYang commented Dec 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panbingkun commented Dec 18, 2023

Uh oh!

panbingkun commented Dec 18, 2023

Uh oh!

dongjoon-hyun commented Dec 18, 2023

Uh oh!

LuciferYang commented Jan 23, 2024

Uh oh!

dongjoon-hyun commented Jan 23, 2024

Uh oh!

panbingkun commented Jan 24, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jan 25, 2024

Uh oh!

LuciferYang commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panbingkun Jan 25, 2024

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jan 25, 2024

Uh oh!

LuciferYang commented Jan 25, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Jan 26, 2024

Uh oh!

dongjoon-hyun commented Jan 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

panbingkun commented Dec 17, 2023 •

edited

Loading

LuciferYang commented Dec 18, 2023 •

edited

Loading

LuciferYang commented Jan 25, 2024 •

edited

Loading