HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements. #6462
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@steveloughran Thank you for the information! I will add more information about log4j2 and java 17 soon. |
|
@ayushtkn @Hexiaoqiao @iwasakims @goiri @steveloughran @tasanuma Hadoop 3.4.0 includes numerous commits involving significant improvements such as RBF, EC, Yarn Federation, DN, AWS, and many more. I am unable to determine the specific highlights and improvements for this release. I need your assistance in collectively deciding the highlights for this release. Regarding the inclusion of Hadoop on Windows in this release, from my personal perspective, I believe it should be highlighted. I would also like to hear your opinions. @GauthamBanasandra |
|
🎊 +1 overall
This message was automatically generated. |
|
@slfan1989 Thanks for your works! Some nit comments,
|
|
Thanks @slfan1989 for putting this together. Sure, I agree that we should mention that Hadoop now fully builds and runs on Windows. Please let me know where you would like me to add some notes for it. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@steveloughran Can we now fully support Java 17 compilation? I didn't find jira. |
Do we have any further information about Hadoop on Windows? Thank you very much! |
|
@aajisaka I see that you have made progress in upgrading log4j and JDK versions. Can we can now support JDK 17 (I am not certain about this)? Also, can we announce full support for log4j2 (HADOOP-12956 is still pending completion) ? Looking forward to your reply. |
|
@slfan1989 I don't think JDK 17 & log4j2 is fully supported. From my understanding, HADOOP-15984 is still a huge blocker. |
|
Compile with JDK 8 and running in JDK 17 might work, but I haven't tested yet. |
@aajisaka Thank you for your reply! |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Hexiaoqiao
reviewed
Jan 24, 2024
There was a problem hiding this comment.
@slfan1989 Thanks for your updates. Leave some nit comments inline. Do we need backport to banch-3.4 and branch-3.4.1?
|
|
||
| All users of the abfs connector in hadoop releases 3.3.2+ MUST either upgrade | ||
| or disable prefetching by setting `fs.azure.readaheadqueue.depth` to `0` | ||
| This release of Hadoop moves the S3A connector to Amazon S3 to the V2 SDK. |
There was a problem hiding this comment.
This release of Hadoop moves the S3A connector to Amazon S3 to the V2 SDK.
->
This release upgrade Hadoop's AWS connector S3A from AWS SDK for Java V1 to AWS SDK for Java V2.
There was a problem hiding this comment.
Thanks for the suggestion! I will improve it.
|
|
||
| [HADOOP-18103](https://issues.apache.org/jira/browse/HADOOP-18103). | ||
| *High performance vectored read API in Hadoop* | ||
| [HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) Federation balance tool introduces a new HDFS federation balance tool to balance data across different federation |
There was a problem hiding this comment.
HDFS-15294 Federation balance tool introduces a new HDFS federation balance tool to balance data across different federation namespaces.
->
HDFS-15294 HDFS Federation balance tool introduces one tool to balance data across different namespace.
|
|
||
| Mapreduce: Manifest Committer for Azure ABFS and google GCS | ||
| ---------------------------------------------------------- | ||
| [HDFS-16663](https://issues.apache.org/jira/browse/HDFS-16663) Allow block reconstruction pending timeout refreshable to increase decommission performance. |
There was a problem hiding this comment.
Allow block reconstruction pending timeout refreshable to increase decommission performance.
->
EC: Allow block reconstruction pending timeout refreshable to increase decommission performance.
| [HDFS-15382](https://issues.apache.org/jira/browse/HDFS-15382) Split one FsDatasetImpl lock to volume grain locks. | ||
|
|
||
| Throughput is one of the core performance evaluation for DataNode instance. | ||
| However, it does not reach the best performance especially for Federation deploy all the time although there are different improvement, |
There was a problem hiding this comment.
Please remove the redundant blank space between 'performance' and 'especially'.
|
Compile time support for JDK-8 & Runtime is till JDK-11, JDK-17 runtime isn't there itself: |
@Hexiaoqiao Thank you very much for reviewing the code! I will improve |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
slfan1989
force-pushed
the
HADOOP-19039
branch
from
c5bc9be to
8ddc42c
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@Hexiaoqiao Thank you for reviewing the code! all compilations have passed. I will merge this pr into the trunk branch. |
slfan1989
added a commit
that referenced
this pull request
Jan 25, 2024
…6462) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>
slfan1989
added a commit
that referenced
this pull request
Jan 25, 2024
…6462) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>
jxhan3
pushed a commit
to jxhan3/hadoop
that referenced
this pull request
Jan 25, 2024
…pache#6462) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>
Thank you for the information! We plan to support JDK17 in the production environment (we will upgrade to Spark4.0 in the future). I will continue to follow up on the compilation of JDK11 and JDK17. I look forward to successfully completing this task together. |
jiajunmao
pushed a commit
to jiajunmao/hadoop-MLEC
that referenced
this pull request
Feb 6, 2024
…pache#6462) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao <[email protected]> Signed-off-by: Shilun Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements.
While preparing for the release of Hadoop-3.4.0, I've noticed the inclusion of numerous commits in this version. Therefore, highlighting significant features and improvements becomes crucial. I've completed the initial version and now seek the review of more experienced partner to ensure the finalization of the version's highlights.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?