Skip to content

Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 #5112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 7, 2022
Merged

Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 #5112

merged 4 commits into from
Nov 7, 2022

Conversation

@melissayou
Copy link
Contributor

Description of PR

As part of HADOOP-18518, this is the second PR to fix deprecation of PathChildrenCache/TreeCache in new ZK. It's a clean cherry pick from HDFS-15383 and HADOOP-17835.

How was this patch tested?

mvn clean install -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip build success

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

fengnanli and others added 3 commits November 5, 2022 15:37
@fengnanli
HDFS-15383. RBF: Add support for router delegation token without watch (
c5da7cc 
apache#2047)

Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
@virajjasani
HADOOP-17835. Use CuratorCache implementation instead of PathChildren…
5f31102 
…Cache / TreeCache (apache#3266)

Signed-off-by: Akira Ajisaka <[email protected]>
checkstyle fix
d6a246d
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 10m 48s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 3s Maven dependency ordering for branch
+1 💚 mvninstall 24m 9s branch-3.3 passed
+1 💚 compile 18m 1s branch-3.3 passed
+1 💚 checkstyle 3m 21s branch-3.3 passed
+1 💚 mvnsite 4m 15s branch-3.3 passed
+1 💚 javadoc 3m 48s branch-3.3 passed
+1 💚 spotbugs 5m 47s branch-3.3 passed
+1 💚 shadedclient 26m 0s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 2m 13s the patch passed
+1 💚 compile 17m 23s the patch passed
-1 ❌ javac 17m 23s /results-compile-javac-root.txt root generated 2 new + 1871 unchanged - 11 fixed = 1873 total (was 1882)
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 5 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 3m 0s /results-checkstyle-root.txt root: The patch generated 3 new + 18 unchanged - 26 fixed = 21 total (was 44)
+1 💚 mvnsite 4m 6s the patch passed
+1 💚 javadoc 3m 35s the patch passed
+1 💚 spotbugs 6m 5s the patch passed
+1 💚 shadedclient 26m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 8s hadoop-common in the patch passed.
+1 💚 unit 1m 43s hadoop-registry in the patch passed.
+1 💚 unit 18m 12s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 19s The patch does not generate ASF License warnings.
216m 44s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/1/artifact/out/Dockerfile
GITHUB PR #5112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux ed203fb1fc91 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / d6a246d
Default Java Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/1/testReport/
Max. process+thread count 2053 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 8s Maven dependency ordering for branch
+1 💚 mvninstall 24m 43s branch-3.3 passed
+1 💚 compile 18m 9s branch-3.3 passed
+1 💚 checkstyle 3m 8s branch-3.3 passed
+1 💚 mvnsite 4m 8s branch-3.3 passed
+1 💚 javadoc 3m 47s branch-3.3 passed
+1 💚 spotbugs 5m 57s branch-3.3 passed
+1 💚 shadedclient 26m 0s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 2m 11s the patch passed
+1 💚 compile 17m 33s the patch passed
-1 ❌ javac 17m 33s /results-compile-javac-root.txt root generated 2 new + 1871 unchanged - 11 fixed = 1873 total (was 1882)
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 53s /results-checkstyle-root.txt root: The patch generated 3 new + 18 unchanged - 26 fixed = 21 total (was 44)
+1 💚 mvnsite 4m 22s the patch passed
+1 💚 javadoc 3m 48s the patch passed
+1 💚 spotbugs 6m 20s the patch passed
+1 💚 shadedclient 26m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 25s hadoop-common in the patch passed.
+1 💚 unit 1m 56s hadoop-registry in the patch passed.
+1 💚 unit 20m 19s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 29s The patch does not generate ASF License warnings.
211m 35s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/2/artifact/out/Dockerfile
GITHUB PR #5112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 18e8f3c6eac5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / a989b86
Default Java Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/2/testReport/
Max. process+thread count 2060 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5112/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@virajith virajith changed the title Hadoop 18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 Nov 7, 2022
@virajith
Copy link
Contributor

virajith commented Nov 7, 2022

Thanks for the backport @melissayou . The changes look good to me - I expect the deprecated method will be addressed by HADOOP-18520. The other failures exist in trunk as well - fixing the checkstyles will not make this a clean cherry-pick.

@virajith
Copy link
Contributor

virajith commented Nov 7, 2022

I'll merge this in the next hour.

@virajith virajith merged commit 02aedd7 into apache:branch-3.3 Nov 7, 2022
NyteKnight pushed a commit to NyteKnight/hadoop that referenced this pull request Jun 25, 2024
@fengnanli @virajjasani
Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 (apa…
072c5a7 
…che#5112) (apache#26)

* HDFS-15383. RBF: Add support for router delegation token without watch (apache#2047)

Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.

* HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (apache#3266)

Signed-off-by: Akira Ajisaka <[email protected]>
Co-authored-by: lfengnan <[email protected]>
Co-authored-by: Viraj Jasani <[email protected]>
Co-authored-by: Melissa You <[email protected]>
ACLOVERRIDE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@melissayou @hadoop-yetus @virajith @fengnanli @virajjasani