HDFS-13639. SlotReleaser is not fast enough #1885

leosunli · 2020-03-08T09:11:00Z

Signed-off-by: sunlisheng [email protected]

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

hadoop-yetus · 2020-03-08T12:26:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	1m 6s	Maven dependency ordering for branch
+1 💚	mvninstall	20m 1s	trunk passed
+1 💚	compile	3m 28s	trunk passed
+1 💚	checkstyle	0m 59s	trunk passed
+1 💚	mvnsite	2m 8s	trunk passed
+1 💚	shadedclient	17m 59s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 16s	trunk passed
+0 🆗	spotbugs	2m 57s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 14s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s	the patch passed
+1 💚	compile	3m 33s	the patch passed
+1 💚	javac	3m 33s	the patch passed
-0 ⚠️	checkstyle	0m 53s	hadoop-hdfs-project: The patch generated 1 new + 125 unchanged - 0 fixed = 126 total (was 125)
+1 💚	mvnsite	1m 54s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 22s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 14s	the patch passed
+1 💚	findbugs	5m 42s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 3s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	94m 39s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 40s	The patch does not generate ASF License warnings.
		180m 44s

Reason	Tests
Failed junit tests	hadoop.hdfs.shortcircuit.TestShortCircuitCache

Subsystem	Report/Notes
Docker	Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/1/artifact/out/Dockerfile
GITHUB PR	#1885
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6a092ae9d57e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `999096d`
Default Java	1.8.0_242
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/1/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/1/testReport/
Max. process+thread count	4592 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-03-09T18:08:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 55s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	20m 5s	trunk passed
+1 💚	compile	3m 29s	trunk passed
+1 💚	checkstyle	0m 57s	trunk passed
+1 💚	mvnsite	2m 7s	trunk passed
+1 💚	shadedclient	17m 49s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 16s	trunk passed
+0 🆗	spotbugs	3m 1s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 20s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 56s	the patch passed
+1 💚	compile	3m 45s	the patch passed
+1 💚	javac	3m 45s	the patch passed
+1 💚	checkstyle	0m 54s	the patch passed
+1 💚	mvnsite	1m 54s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 22s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 6s	the patch passed
+1 💚	findbugs	5m 29s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 9s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	105m 53s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s	The patch does not generate ASF License warnings.
		191m 45s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
	hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier
	hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped

Subsystem	Report/Notes
Docker	Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/2/artifact/out/Dockerfile
GITHUB PR	#1885
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 322ef47459b4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c734d69`
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/2/testReport/
Max. process+thread count	4403 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

jojochuang · 2020-05-15T16:05:08Z

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

jojochuang · 2020-05-15T16:09:45Z

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

please add test timeout

jojochuang · 2020-05-15T16:10:01Z

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

Can we make this sleep time shorter? Waiting for 15 seconds seems too excessive. You may have to change the timeout configuration.

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

jojochuang · 2020-05-15T16:12:18Z

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

assertEquals()

jojochuang · 2020-05-15T16:12:26Z

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

.../hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java

Signed-off-by: sunlisheng <[email protected]>

leosunli · 2020-05-19T13:35:00Z

Thank @jojochuang for taking your time to review.
i have added description of performance improvement on jira.

Signed-off-by: sunlisheng <[email protected]>

hadoop-yetus · 2020-05-19T17:26:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 27s	Maven dependency ordering for branch
+1 💚	mvninstall	21m 13s	trunk passed
+1 💚	compile	3m 38s	trunk passed
+1 💚	checkstyle	0m 57s	trunk passed
+1 💚	mvnsite	2m 17s	trunk passed
+1 💚	shadedclient	17m 0s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 22s	trunk passed
+0 🆗	spotbugs	2m 56s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 10s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 56s	the patch passed
+1 💚	compile	3m 23s	the patch passed
+1 💚	javac	3m 23s	the patch passed
-0 ⚠️	checkstyle	0m 52s	hadoop-hdfs-project: The patch generated 1 new + 125 unchanged - 0 fixed = 126 total (was 125)
+1 💚	mvnsite	1m 55s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 49s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 16s	the patch passed
+1 💚	findbugs	5m 12s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 4s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	100m 53s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s	The patch does not generate ASF License warnings.
		185m 33s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
	hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.shortcircuit.TestShortCircuitCache

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/3/artifact/out/Dockerfile
GITHUB PR	#1885
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 15579af5d749 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d4e3640`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/3/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/3/testReport/
Max. process+thread count	3757 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jojochuang

Ok I think we're almost good. I am +1 except for the change required in the UT.

.../hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java

...ect/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java

Signed-off-by: sunlisheng <[email protected]>

jojochuang · 2020-05-20T17:38:55Z

The last precommit timed out. I triggered a rebuild let's see if it goes through this time.

hadoop-yetus · 2020-05-20T20:18:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 52s	Maven dependency ordering for branch
+1 💚	mvninstall	18m 45s	trunk passed
+1 💚	compile	3m 34s	trunk passed
+1 💚	checkstyle	0m 58s	trunk passed
+1 💚	mvnsite	2m 7s	trunk passed
+1 💚	shadedclient	17m 10s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 22s	trunk passed
+0 🆗	spotbugs	2m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 1s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 52s	the patch passed
+1 💚	compile	3m 22s	the patch passed
+1 💚	javac	3m 22s	the patch passed
-0 ⚠️	checkstyle	0m 52s	hadoop-hdfs-project: The patch generated 1 new + 125 unchanged - 0 fixed = 126 total (was 125)
+1 💚	mvnsite	1m 54s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 48s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 16s	the patch passed
+1 💚	findbugs	5m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 2s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	95m 9s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s	The patch does not generate ASF License warnings.
		177m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.namenode.TestNameNodeMXBean
	hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.TestBPOfferService

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/7/artifact/out/Dockerfile
GITHUB PR	#1885
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8b59fd370119 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `726b8e3`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/7/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/7/testReport/
Max. process+thread count	4626 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1885/7/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jojochuang · 2020-05-20T21:19:02Z

+1 failed tests are unrelated and are being tracked in other jiras.

(cherry picked from commit be374fa)

(cherry picked from commit be374fa) Change-Id: I2b0538ee6313f5eeb220c002e8a4052a9b70c8f5

jojochuang added the HDFS label May 1, 2020

jojochuang self-requested a review May 6, 2020 21:26

jojochuang requested changes May 15, 2020

View reviewed changes

jojochuang reviewed May 15, 2020

View reviewed changes

.../hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java Outdated Show resolved Hide resolved

jojochuang reviewed May 15, 2020

View reviewed changes

.../hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java Outdated Show resolved Hide resolved

sunlisheng added 2 commits May 19, 2020 21:31

SlotReleaser is not fast enough

7fef36f

Signed-off-by: sunlisheng <[email protected]>

Fix TestShortCircuitCache UT

7167076

Signed-off-by: sunlisheng <[email protected]>

leosunli force-pushed the HDFS-13639 branch from 49e8948 to 2f99cb1 Compare May 19, 2020 13:33

Fix TestShortCircuitCache

f2c8c58

Signed-off-by: sunlisheng <[email protected]>

leosunli force-pushed the HDFS-13639 branch from 2f99cb1 to f2c8c58 Compare May 19, 2020 13:38

jojochuang requested changes May 19, 2020

View reviewed changes

Fix UT

8652c1b

Signed-off-by: sunlisheng <[email protected]>

leosunli force-pushed the HDFS-13639 branch from cc0e0ec to 8652c1b Compare May 20, 2020 01:57

jojochuang approved these changes May 20, 2020

View reviewed changes

jojochuang merged commit be374fa into apache:trunk May 21, 2020

asfgit pushed a commit that referenced this pull request Mar 25, 2021

HDFS-13639. SlotReleaser is not fast enough (#1885)

b3e968d

(cherry picked from commit be374fa)

stiga-huang mentioned this pull request Apr 14, 2022

HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths #4158

Merged

1 task

jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023

HDFS-13639. SlotReleaser is not fast enough (apache#1885)

9afacda

(cherry picked from commit be374fa) Change-Id: I2b0538ee6313f5eeb220c002e8a4052a9b70c8f5

Uh oh!

HDFS-13639. SlotReleaser is not fast enough #1885

HDFS-13639. SlotReleaser is not fast enough #1885

Uh oh!

Conversation

leosunli commented Mar 8, 2020

NOTICE

Uh oh!

hadoop-yetus commented Mar 8, 2020

Uh oh!

hadoop-yetus commented Mar 9, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leosunli commented May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented May 19, 2020

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jojochuang commented May 20, 2020

Uh oh!

hadoop-yetus commented May 20, 2020

Uh oh!

jojochuang commented May 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leosunli commented May 19, 2020 •

edited

Loading