HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal #2790

Nargeshdb · 2021-03-19T18:46:04Z

This PR fixes the issue mentioned here.

hadoop-yetus · 2021-03-20T00:49:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 2s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 41s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 16s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 32s		trunk passed
+1 💚	shadedclient	18m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 17s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 7s		the patch passed
+1 💚	shadedclient	16m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	273m 43s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	asflicense	0m 42s	/results-asflicense.txt	The patch generated 27 ASF License warnings.
		361m 33s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestSaveNamespace
	hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
	hadoop.hdfs.server.namenode.TestParallelImageWrite
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.namenode.TestXAttrConfigFlag
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.namenode.TestFileTruncate
	hadoop.hdfs.server.namenode.TestAddStripedBlocks
	hadoop.hdfs.server.namenode.TestFSImage
	hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
	hadoop.hdfs.server.namenode.TestStripedINodeFile
	hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.namenode.TestAuditLogs
	hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy
	hadoop.hdfs.server.namenode.ha.TestHAAppend
	hadoop.hdfs.server.namenode.TestFileContextAcl
	hadoop.hdfs.server.namenode.TestFSImageWithAcl
	hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/artifact/out/Dockerfile
GITHUB PR	#2790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 2741c9243cba 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2d1dee5`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/testReport/
Max. process+thread count	3353 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

virajjasani · 2021-03-22T09:05:49Z

...p-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java

   */
  @Override // Closeable
  public void close() throws IOException {
-    storage.close();


This is because it can throw IOException?

Yes, exactly. If it throws an exception, then committedTxnId and curSegment are never closed.

virajjasani

+1 (non-binding)

ayushtkn

LGTM
Have triggered the tests again

hadoop-yetus · 2021-03-22T17:30:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 9s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 20s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 13s		trunk passed
+1 💚	shadedclient	18m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 55s		the patch passed
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 16s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 21s		the patch passed
+1 💚	shadedclient	18m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	340m 5s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		432m 37s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.namenode.ha.TestHASafeMode
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/artifact/out/Dockerfile
GITHUB PR	#2790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux b5180c5da691 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2d1dee5`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/testReport/
Max. process+thread count	2075 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2790/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2021-03-23T11:51:02Z

Thanx @Nargeshdb for the contribution and @virajjasani for the review!!!

….server.Journal (#2790). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <[email protected]> (cherry picked from commit d77c7ab)

….server.Journal (apache#2790). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <[email protected]>

close committedTxnId and curSegment before calling close on storage

2d1dee5

virajjasani reviewed Mar 22, 2021

View reviewed changes

virajjasani approved these changes Mar 22, 2021

View reviewed changes

ayushtkn approved these changes Mar 22, 2021

View reviewed changes

ayushtkn merged commit d77c7ab into apache:trunk Mar 23, 2021

Nargeshdb deleted the possible-resource-leak branch March 23, 2021 21:22

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal…

c40135f

….server.Journal (apache#2790). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal #2790

HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal #2790

Uh oh!

Nargeshdb commented Mar 19, 2021

Uh oh!

hadoop-yetus commented Mar 20, 2021

Uh oh!

virajjasani Mar 22, 2021

Uh oh!

Nargeshdb Mar 22, 2021

Uh oh!

virajjasani left a comment

Uh oh!

ayushtkn left a comment •

edited

Loading

Uh oh!

hadoop-yetus commented Mar 22, 2021

Uh oh!

ayushtkn commented Mar 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal #2790

HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal #2790

Uh oh!

Conversation

Nargeshdb commented Mar 19, 2021

Uh oh!

hadoop-yetus commented Mar 20, 2021

Uh oh!

virajjasani Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

Nargeshdb Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Mar 22, 2021

Uh oh!

ayushtkn commented Mar 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ayushtkn left a comment •

edited

Loading