HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka #832

supratimdeka · 2019-05-18T10:54:08Z

https://issues.apache.org/jira/browse/HDDS-1535

Follow up from HDDS-1511.
Space tracking for Open Containers (committed space in the volume) relies on usedBytes in the Container state. usedBytes is not persisted for every update (chunkWrite). So on a node restart the value is stale.

The fix iterates the block DB for Open containers when building the container set (during startup).

…Contributed by Supratim Deka

hadoop-yetus · 2019-05-18T12:57:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	27	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	397	trunk passed
+1	compile	202	trunk passed
+1	checkstyle	49	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	823	branch has no errors when building and testing our client artifacts.
+1	javadoc	127	trunk passed
0	spotbugs	236	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	415	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	396	the patch passed
+1	compile	205	the patch passed
+1	javac	205	the patch passed
-0	checkstyle	30	hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	630	patch has no errors when building and testing our client artifacts.
+1	javadoc	112	the patch passed
+1	findbugs	429	the patch passed
		_ Other Tests _
-1	unit	154	hadoop-hdds in the patch failed.
-1	unit	1205	hadoop-ozone in the patch failed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		7313

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/artifact/out/Dockerfile
GITHUB PR	#832
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c94d1abf2ea8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `732133c`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/testReport/
Max. process+thread count	3953 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

jiwq

Do you mind modify the style nits?

...ainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java

…, both as per review discussion

arp7 · 2019-05-22T04:38:10Z

+1 pending CI checks.

hadoop-yetus · 2019-05-22T06:38:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	574	trunk passed
+1	compile	264	trunk passed
+1	checkstyle	70	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	863	branch has no errors when building and testing our client artifacts.
+1	javadoc	147	trunk passed
0	spotbugs	313	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	524	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	533	the patch passed
+1	compile	292	the patch passed
+1	javac	292	the patch passed
+1	checkstyle	73	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	651	patch has no errors when building and testing our client artifacts.
+1	javadoc	138	the patch passed
+1	findbugs	528	the patch passed
		_ Other Tests _
-1	unit	147	hadoop-hdds in the patch failed.
-1	unit	1650	hadoop-ozone in the patch failed.
+1	asflicense	62	The patch does not generate ASF License warnings.
		6716

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.TestMiniChaosOzoneCluster
	hadoop.ozone.TestMiniOzoneCluster
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
	hadoop.ozone.scm.TestAllocateContainer
	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-832/3/artifact/out/Dockerfile
GITHUB PR	#832
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 9512ce78074a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `9dff6ef`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/3/testReport/
Max. process+thread count	4196 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7 · 2019-05-23T05:27:11Z

/retest

hadoop-yetus · 2019-05-23T06:00:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	29	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	536	trunk passed
+1	compile	266	trunk passed
+1	checkstyle	71	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	817	branch has no errors when building and testing our client artifacts.
+1	javadoc	163	trunk passed
0	spotbugs	291	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	478	trunk passed
-0	patch	359	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
-1	mvninstall	105	hadoop-hdds in the patch failed.
-1	compile	90	hadoop-hdds in the patch failed.
-1	javac	90	hadoop-hdds in the patch failed.
-0	checkstyle	45	hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	681	patch has no errors when building and testing our client artifacts.
+1	javadoc	162	the patch passed
-1	findbugs	135	hadoop-hdds in the patch failed.
		_ Other Tests _
-1	unit	126	hadoop-hdds in the patch failed.
-1	unit	1020	hadoop-ozone in the patch failed.
+1	asflicense	57	The patch does not generate ASF License warnings.
		5868

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.impl.TestContainerPersistence
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/Dockerfile
GITHUB PR	#832
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 60e4fead4b43 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `03aa70f`
Default Java	1.8.0_212
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-mvninstall-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-compile-hadoop-hdds.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/diff-checkstyle-hadoop-hdds.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-findbugs-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/testReport/
Max. process+thread count	5059 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-05-23T17:25:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	50	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	545	trunk passed
+1	compile	287	trunk passed
+1	checkstyle	119	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1032	branch has no errors when building and testing our client artifacts.
+1	javadoc	177	trunk passed
0	spotbugs	448	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	652	trunk passed
-0	patch	500	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	507	the patch passed
+1	compile	303	the patch passed
+1	javac	303	the patch passed
+1	checkstyle	91	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	763	patch has no errors when building and testing our client artifacts.
+1	javadoc	164	the patch passed
+1	findbugs	523	the patch passed
		_ Other Tests _
-1	unit	169	hadoop-hdds in the patch failed.
-1	unit	1862	hadoop-ozone in the patch failed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		7456

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestBlockOutputStream
	hadoop.ozone.container.common.impl.TestContainerPersistence
	hadoop.ozone.scm.node.TestQueryNode
	hadoop.ozone.client.rpc.TestBCSID

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-832/6/artifact/out/Dockerfile
GITHUB PR	#832
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3c10e08de89b 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / a771e2a
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/6/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/6/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/6/testReport/
Max. process+thread count	5369 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-832/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7 · 2019-05-23T17:42:30Z

The UT failures are unrelated. I've merged this into trunk.

…anner Author: Ahmed Abdul Hamid <[email protected]> Reviewers: Aditya Toomla <[email protected]> Closes apache#832 from ahmedahamid/master

HDDS-1535. Space tracking for Open Containers : Handle Node Startup. …

869a1ab

…Contributed by Supratim Deka

jiwq reviewed May 19, 2019

View reviewed changes

Fixed checkstyle issues.

64c3985

arp7 reviewed May 21, 2019

View reviewed changes

...ainer-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java Show resolved Hide resolved

arp7 added the ozone label May 22, 2019

verifyContainerData also does fixup, renamed. Added a Javadoc comment…

456bb8a

…, both as per review discussion

Merge branch 'trunk' into HDDS-1535

9da62f3

sdeka added 2 commits May 23, 2019 11:36

fixed merge error. adapted to new signature of BlockUtils getDB

72bef0f

fixed checkstyle issue post merge

ca93760

arp7 merged commit b4b9120 into apache:trunk May 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka #832

HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka #832

Uh oh!

supratimdeka commented May 18, 2019

Uh oh!

hadoop-yetus commented May 18, 2019

Uh oh!

jiwq left a comment

Uh oh!

Uh oh!

arp7 commented May 22, 2019

Uh oh!

hadoop-yetus commented May 22, 2019

Uh oh!

arp7 commented May 23, 2019

Uh oh!

hadoop-yetus commented May 23, 2019

Uh oh!

hadoop-yetus commented May 23, 2019

Uh oh!

arp7 commented May 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka #832

HDDS-1535. Space tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka #832

Uh oh!

Conversation

supratimdeka commented May 18, 2019

Uh oh!

hadoop-yetus commented May 18, 2019

Uh oh!

jiwq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arp7 commented May 22, 2019

Uh oh!

hadoop-yetus commented May 22, 2019

Uh oh!

arp7 commented May 23, 2019

Uh oh!

hadoop-yetus commented May 23, 2019

Uh oh!

hadoop-yetus commented May 23, 2019

Uh oh!

arp7 commented May 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants