HADOOP-18079. Upgrade Netty to 4.1.77. (#3977)

[jojochuang]

Upgrade netty to address

CVE-2019-20444,
CVE-2019-20445
CVE-2022-24823

Contributed by Wei-Chiu Chuang

cherrypicked from #3977
(cherry picked from commit a55ace7)
(cherry picked from commit c545341)

Description of PR

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	32m 0s		branch-3.2 passed
+1 💚	compile	0m 25s		branch-3.2 passed
+1 💚	mvnsite	0m 30s		branch-3.2 passed
+1 💚	javadoc	0m 36s		branch-3.2 passed
+1 💚	shadedclient	48m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 15s		the patch passed
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed
+1 💚	shadedclient	17m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 21s		hadoop-project in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		85m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4593/1/artifact/out/Dockerfile
GITHUB PR	#4593
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux b91fd4ea249c 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 0372229
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4593/1/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4593/1/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[jasonwzs]

@jojochuang , is there a special reason why we need add other netty artifacts dependency versions declaration in this PR, e.g. netty-handler-proxy? I don't see where those lib dependencies except netty-all are explicitly added in hadoop project. Is it ok to add have netty-all dependency only without adding those additional netty-* lib dependencies declaration?

[jojochuang]

It is my understanding that transitive dependencies should be included.
https://infra.apache.org/licensing-howto.html#deps-of-deps

Does it make sense to exclude transitive dependencies that are not used by Hadoop's use of Netty? i'm not sure what it implies for downstream applications that implicitly import Netty from Hadoop.

[jojochuang]

[INFO] +- io.netty:netty-all:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-buffer:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-dns:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-haproxy:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-http:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-http2:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-memcache:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-mqtt:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-redis:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-smtp:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-socks:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-stomp:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-codec-xml:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-common:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-handler:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-handler-proxy:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-resolver:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-resolver-dns:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-rxtx:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-sctp:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-udt:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-classes-epoll:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-native-unix-common:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-classes-kqueue:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-resolver-dns-classes-macos:jar:4.1.77.Final:compile
[INFO] | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.77.Final:runtime
[INFO] | +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.77.Final:runtime
[INFO] | +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.77.Final:runtime
[INFO] | +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.77.Final:runtime
[INFO] | +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.77.Final:runtime
[INFO] | - io.netty:netty-resolver-dns-native-macos:jar:osx-aarch_64:4.1.77.Final:runtime

[jasonwzs]

https://infra.apache.org/licensing-howto.html#deps-of-deps is about license, but this PR has only change to pom.xml.
In case that downstream application explicitly imports netty from hadoop dependency, the versions of other netty libs that netty-all are determined by netty-all version itself, so it's not necessary to list them in the dependenciesManagment section in pom file.

[jojochuang]

Ah i see what you're saying. I thought this was in the trunk where we upated LICENSE too.

IIRC the build breaks due to divergent transitive dependencies and a workaround was to define the versions explicitly. It could also work if we exclude those transitive dependencies. I didn't try that.