HDFS-15906. Close FSImage and FSNamesystem after formatting is complete #2788

tomscut · 2021-03-19T11:27:08Z

We should close FSImage and FSNamesystem after formatting is complete.

tomscut · 2021-03-19T11:39:30Z

Hi @tasanuma @JunpingDu , could you please help review the code? Thank you very mush.

hadoop-yetus · 2021-03-19T16:40:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 17s		trunk passed
+1 💚	compile	1m 18s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 15s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 24s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 6s		trunk passed
+1 💚	shadedclient	16m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 57s		the patch passed
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	0m 45s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 15s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 4s		the patch passed
+1 💚	shadedclient	15m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	227m 8s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		311m 49s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/1/artifact/out/Dockerfile
GITHUB PR	#2788
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux b327ea81ee8d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4215748`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/1/testReport/
Max. process+thread count	3156 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2021-03-20T14:57:08Z

Failed junit tests:
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots

Those failed unit tests were unrelated to the change. And they work fine locally.

tasanuma · 2021-03-22T01:49:18Z

@tomscut Thanks for the PR. Some minor comments:

Maybe fsn.close() should be in the finally scope too.
It is better to verify that fsn and fsImage are not null before closing them.

… null before closing them

tomscut · 2021-03-22T02:39:17Z

@tomscut Thanks for the PR. Some minor comments:

Maybe fsn.close() should be in the finally scope too.

It is better to verify that fsn and fsImage are not null before closing them.

Thanks @tasanuma for your kind review and suggestions, I fixed them.

tasanuma · 2021-03-22T02:50:39Z

+1, pending Jenkins.

hadoop-yetus · 2021-03-22T09:39:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 18s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 27s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	18m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 17s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 21s		the patch passed
+1 💚	shadedclient	18m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	328m 45s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		421m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.datanode.TestIncrementalBrVariations
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/2/artifact/out/Dockerfile
GITHUB PR	#2788
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4a9ee8cce8ab 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `03e3205`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/2/testReport/
Max. process+thread count	2153 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2788/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2021-03-22T10:02:53Z

Failed junit tests:
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
hadoop.hdfs.server.datanode.TestIncrementalBrVariations
hadoop.hdfs.TestPersistBlocks
hadoop.hdfs.server.namenode.TestDecommissioningStatus
hadoop.hdfs.TestDFSShell
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.server.datanode.TestBlockScanner
hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
hadoop.hdfs.server.namenode.ha.TestEditLogTailer
hadoop.hdfs.server.namenode.ha.TestBootstrapStandby

Hi @tasanuma , those failed unit tests were unrelated to the change, and they all work fine locally.

tasanuma · 2021-03-22T11:28:59Z

@tomscut Thanks for your confirmation. I also confirmed the failed tests are not related. Will merge it.

…te (#2788) (cherry picked from commit 343ce8a)

tasanuma · 2021-03-22T11:48:30Z

Merged to trunk and cherry-picked to branch-3.3.

@tomscut There is a small conflict with branch-3.2. Could you create another PR for branch-3.2 if necessary?

tomscut · 2021-03-22T15:34:47Z

Thanks @tasanuma for your merge. I submitted a PR for branch-3.2.

…te (apache#2788)

HDFS-15906. Close FSImage and FSNamesystem after formatting is complete

4215748

tomscut mentioned this pull request Mar 19, 2021

HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

Merged

move close operation to finally scope, verify fsn and fsImage are not…

03e3205

… null before closing them

tasanuma merged commit 343ce8a into apache:trunk Mar 22, 2021

tasanuma pushed a commit that referenced this pull request Mar 22, 2021

HDFS-15906. Close FSImage and FSNamesystem after formatting is comple…

bc2bdee

…te (#2788) (cherry picked from commit 343ce8a)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15906. Close FSImage and FSNamesystem after formatting is comple…

7bad01d

…te (apache#2788)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15906. Close FSImage and FSNamesystem after formatting is complete #2788

HDFS-15906. Close FSImage and FSNamesystem after formatting is complete #2788

Uh oh!

tomscut commented Mar 19, 2021

Uh oh!

tomscut commented Mar 19, 2021

Uh oh!

hadoop-yetus commented Mar 19, 2021

Uh oh!

tomscut commented Mar 20, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

hadoop-yetus commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDFS-15906. Close FSImage and FSNamesystem after formatting is complete #2788

HDFS-15906. Close FSImage and FSNamesystem after formatting is complete #2788

Uh oh!

Conversation

tomscut commented Mar 19, 2021

Uh oh!

tomscut commented Mar 19, 2021

Uh oh!

hadoop-yetus commented Mar 19, 2021

Uh oh!

tomscut commented Mar 20, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

hadoop-yetus commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tasanuma commented Mar 22, 2021

Uh oh!

tomscut commented Mar 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants