HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet #3114

ferhui · 2021-06-17T08:30:15Z

…#removeAndGet (#3065)

Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

…#removeAndGet (apache#3065) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

hadoop-yetus · 2021-06-17T13:20:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 18 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	28m 19s		branch-3.2 passed
+1 💚	compile	1m 6s		branch-3.2 passed
+1 💚	checkstyle	0m 55s		branch-3.2 passed
+1 💚	mvnsite	1m 14s		branch-3.2 passed
+1 💚	javadoc	0m 58s		branch-3.2 passed
+1 💚	spotbugs	3m 13s		branch-3.2 passed
+1 💚	shadedclient	16m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 9s		the patch passed
+1 💚	compile	1m 3s		the patch passed
+1 💚	cc	1m 3s		the patch passed
+1 💚	javac	1m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 51s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 1349 unchanged - 13 fixed = 1354 total (was 1362)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 48s		the patch passed
+1 💚	spotbugs	3m 21s		the patch passed
+1 💚	shadedclient	18m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	199m 37s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		289m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/1/artifact/out/Dockerfile
GITHUB PR	#3114
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname	Linux 6e537f370925 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `a900308`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/1/testReport/
Max. process+thread count	1918 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ferhui · 2021-06-18T01:12:50Z

Failed Tests passed locally

hadoop-yetus · 2021-06-18T06:02:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	buf	0m 1s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 18 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	28m 19s		branch-3.2 passed
+1 💚	compile	1m 1s		branch-3.2 passed
+1 💚	checkstyle	0m 50s		branch-3.2 passed
+1 💚	mvnsite	1m 10s		branch-3.2 passed
+1 💚	javadoc	0m 55s		branch-3.2 passed
+1 💚	spotbugs	2m 53s		branch-3.2 passed
+1 💚	shadedclient	16m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	0m 56s		the patch passed
+1 💚	cc	0m 56s		the patch passed
+1 💚	javac	0m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 46s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1349 unchanged - 13 fixed = 1350 total (was 1362)
+1 💚	mvnsite	1m 2s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 47s		the patch passed
+1 💚	spotbugs	3m 1s		the patch passed
+1 💚	shadedclient	17m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	197m 13s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		284m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestFsck
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/2/artifact/out/Dockerfile
GITHUB PR	#3114
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname	Linux b359bd9ab928 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `4c26d6a`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/2/testReport/
Max. process+thread count	2117 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

whbing · 2021-06-18T07:06:14Z

Our test cluster of 3.2 version with this PR works well. It looks good.

hust-hhb · 2021-06-18T07:06:38Z

It seems missing some change on file TestBlockInfo.java

whbing · 2021-06-18T07:08:53Z

...ct/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

      throws IOException {
+    // blockReceived reports a finalized block
+    Collection<BlockInfoToAdd> toAdd = new LinkedList<>();
+    Collection<Block> toInvalidate = new LinkedList<Block>();


Nit: can be <>

THanks, resolve

tomscut · 2021-06-18T07:16:25Z

Our test cluster of 3.2 version with this PR works well. It looks good.

Hi @whbing , I have a question. How many disks per node and how many blocks per disk in your cluster? And will the NameNode be affected when processing block reports? Thank you.

ferhui · 2021-06-18T08:03:25Z

It seems missing some change on file TestBlockInfo.java

Thanks, add the missing UT testBlockListMoveToHead

hust-hhb · 2021-06-18T08:05:38Z

LGTM

whbing · 2021-06-18T09:17:04Z

Our test cluster of 3.2 version with this PR works well. It looks good.

Hi @whbing , I have a question. How many disks per node and how many blocks per disk in your cluster? And will the NameNode be affected when processing block reports? Thank you.

env: 10 dns, 12 disks/node, 300,000 blocks/node for stress test. Full block report speed looks little different.

tomscut · 2021-06-18T09:44:03Z

Our test cluster of 3.2 version with this PR works well. It looks good.

Hi @whbing , I have a question. How many disks per node and how many blocks per disk in your cluster? And will the NameNode be affected when processing block reports? Thank you.

Our test cluster of 3.2 version with this PR works well. It looks good.

Hi @whbing , I have a question. How many disks per node and how many blocks per disk in your cluster? And will the NameNode be affected when processing block reports? Thank you.

env: 10 dns, 12 disks/node, 300,000 blocks/node for stress test. Full block report speed looks little different.

Thanks @whbing for your reply.

hadoop-yetus · 2021-06-18T13:01:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 18 new or modified test files.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	32m 17s		branch-3.2 passed
+1 💚	compile	1m 14s		branch-3.2 passed
+1 💚	checkstyle	1m 4s		branch-3.2 passed
+1 💚	mvnsite	1m 25s		branch-3.2 passed
+1 💚	javadoc	1m 14s		branch-3.2 passed
+1 💚	spotbugs	3m 24s		branch-3.2 passed
+1 💚	shadedclient	17m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 21s		the patch passed
+1 💚	compile	1m 9s		the patch passed
+1 💚	cc	1m 9s		the patch passed
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 53s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1349 unchanged - 13 fixed = 1350 total (was 1362)
+1 💚	mvnsite	1m 22s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 52s		the patch passed
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	19m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	213m 0s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		298m 6s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/3/artifact/out/Dockerfile
GITHUB PR	#3114
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname	Linux f3bad48e0df9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `b3e9099`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/3/testReport/
Max. process+thread count	1886 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3114/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ferhui · 2021-06-18T13:03:37Z

@whbing @AlphaGouGe Thanks for review, merged

…oldedTreeSet#removeAndGet (apache#3114) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java (cherry picked from commit c14f3e4) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java Change-Id: I75d19cb527b023face9b7b400628a58337ed8e15 (cherry picked from commit c6259b3955bfe79d5f6f1835669cef7d09c8e25d)

…oldedTreeSet#removeAndGet (apache#3114) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java (cherry picked from commit c14f3e4) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java Change-Id: Ie97cd44fd382192a97815a0baf1ab41524570933

remove unused imports

4c26d6a

whbing reviewed Jun 18, 2021

View reviewed changes

ferhui added 3 commits June 18, 2021 15:19

fix DirectoryScanner.java

3ba4b08

add UT testBlockListMoveToHead

69fc287

address comments

b3e9099

ferhui merged commit c14f3e4 into apache:branch-3.2 Jun 18, 2021

Uh oh!

HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet #3114

HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet #3114

Uh oh!

Conversation

ferhui commented Jun 17, 2021

NOTICE

Uh oh!

hadoop-yetus commented Jun 17, 2021

Uh oh!

ferhui commented Jun 18, 2021

Uh oh!

hadoop-yetus commented Jun 18, 2021

Uh oh!

whbing commented Jun 18, 2021

Uh oh!

hust-hhb commented Jun 18, 2021

Uh oh!

whbing Jun 18, 2021

Choose a reason for hiding this comment

Uh oh!

ferhui Jun 18, 2021

Choose a reason for hiding this comment

Uh oh!

tomscut commented Jun 18, 2021

Uh oh!

ferhui commented Jun 18, 2021

Uh oh!

hust-hhb commented Jun 18, 2021

Uh oh!

whbing commented Jun 18, 2021

Uh oh!

tomscut commented Jun 18, 2021

Uh oh!

hadoop-yetus commented Jun 18, 2021

Uh oh!

ferhui commented Jun 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants