HDDS-1371. Download RocksDB checkpoint from OM Leader to Follower. #703

hanishakoneru · 2019-04-05T19:21:11Z

No description provided.

hanishakoneru · 2019-04-05T19:21:52Z

Posted initial patch. Will add unit tests soon.

arp7

We should add unit tests.

arp7 · 2019-04-09T22:53:11Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

We should probably use Paths.get to join the path here. Using / breaks Windows. Even though we don't officially support Windows we should make the path logic platform agnostic.

arp7 · 2019-04-09T22:54:47Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

What happens during upgrade if two OMs are using different compression schemes? Perhaps we should file a todo to make the compression scheme configurable later.

arp7 · 2019-04-09T22:55:31Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

Nitpick: Need a space after //.

Also in other places in the same patch.

arp7 · 2019-04-09T22:56:21Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

Let's log the dir name here.

arp7 · 2019-04-09T22:57:09Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

Is there any existing code in Apache commons/HDFS which does this? I am sure there must be. One thing we are potentially missing here is file attributes.

arp7 · 2019-04-09T23:01:17Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

I didn't understand this. Could you explain a bit what it does?

arp7 · 2019-04-09T23:01:38Z

...-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java

Could you add a Javadoc for the class please?

arp7 · 2019-04-10T20:43:47Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMNodeDetails.java

Should use getHttpPolicy here.

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

hanishakoneru · 2019-04-13T00:16:16Z

Thank you @arp7 for the review. Addressed your review comments and added a unit test.

hadoop-hdds/common/src/main/resources/ozone-default.xml

hanishakoneru · 2019-06-03T21:38:29Z

/retest

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java

hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpoint.java

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java

hadoop-hdds/common/src/main/resources/ozone-default.xml

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

bharatviswa504

Thank You @hanishakoneru for the contribution.
Overall patch LGTM. I have a few minor comments.

hanishakoneru · 2019-06-05T18:12:10Z

/retest

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

hadoop-yetus · 2019-06-05T19:41:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	63	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
+1	mvninstall	535	trunk passed
+1	compile	287	trunk passed
+1	checkstyle	81	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	950	branch has no errors when building and testing our client artifacts.
+1	javadoc	176	trunk passed
0	spotbugs	382	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	587	trunk passed
		_ Patch Compile Tests _
0	mvndep	30	Maven dependency ordering for patch
+1	mvninstall	473	the patch passed
+1	compile	291	the patch passed
+1	javac	291	the patch passed
-0	checkstyle	46	hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	733	patch has no errors when building and testing our client artifacts.
-1	javadoc	62	hadoop-hdds in the patch failed.
-1	javadoc	101	hadoop-ozone generated 7 new + 8 unchanged - 0 fixed = 15 total (was 8)
-1	findbugs	86	hadoop-ozone in the patch failed.
		_ Other Tests _
+1	unit	264	hadoop-hdds in the patch passed.
-1	unit	59	hadoop-ozone in the patch failed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		5314

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/Dockerfile
GITHUB PR	#703
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 2f113408fdf3 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `0b1e288`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/diff-checkstyle-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/testReport/
Max. process+thread count	403 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/client hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bharatviswa504

+1, pending CI. Take care of two minor comments during commit.
As discussed offline using of the new property ozone.http.policy for ozone, can we open a new Jira for that?

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java

hadoop-hdds/common/src/main/resources/ozone-default.xml

hanishakoneru · 2019-06-05T21:44:58Z

/retest

hanishakoneru · 2019-06-06T19:56:19Z

Fixed related unit test failures.
/retest

hanishakoneru · 2019-06-06T19:56:51Z

/retest

hanishakoneru · 2019-06-06T22:45:35Z

/retest

hadoop-yetus · 2019-06-07T00:58:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	45	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	524	trunk passed
+1	compile	296	trunk passed
+1	checkstyle	80	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	932	branch has no errors when building and testing our client artifacts.
+1	javadoc	179	trunk passed
0	spotbugs	385	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	618	trunk passed
		_ Patch Compile Tests _
0	mvndep	32	Maven dependency ordering for patch
+1	mvninstall	482	the patch passed
+1	compile	301	the patch passed
+1	javac	301	the patch passed
+1	checkstyle	79	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	731	patch has no errors when building and testing our client artifacts.
+1	javadoc	203	the patch passed
+1	findbugs	650	the patch passed
		_ Other Tests _
-1	unit	190	hadoop-hdds in the patch failed.
-1	unit	2369	hadoop-ozone in the patch failed.
+1	asflicense	47	The patch does not generate ASF License warnings.
		7951

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.impl.TestHddsDispatcher
	hadoop.ozone.client.rpc.TestBCSID
	hadoop.hdds.scm.pipeline.TestNode2PipelineMap
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
	hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode
	hadoop.ozone.client.rpc.TestOzoneRpcClient

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-703/11/artifact/out/Dockerfile
GITHUB PR	#703
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux c00839d6af42 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / cb9bc6e
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/11/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/11/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/11/testReport/
Max. process+thread count	5316 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-703/11/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bharatviswa504 · 2019-06-07T02:40:31Z

Thank You @hanishakoneru for the contribution.
I will commit this.

Internally at Linkedin we've observed that excluding specific `js` and `css` through rat gradle-plugin causes the ligradle build to not run the unit-tests. To fix it, this patch removes those files from rat excludes and adds the samza license to the individual files. Author: Shanthoosh Venkataraman <[email protected]> Reviewers: Prateek Maheshwari <[email protected]> Closes apache#703 from shanthoosh/master

hanishakoneru added the ozone label Apr 5, 2019

hanishakoneru requested review from arp7 and bharatviswa504 April 5, 2019 19:21

This comment has been minimized.

Sign in to view

arp7 reviewed Apr 9, 2019

View reviewed changes

arp7 reviewed Apr 10, 2019

View reviewed changes

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMNodeDetails.java Outdated

Copy link

Contributor

arp7 Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use getHttpPolicy here.

arp7 reviewed Apr 10, 2019

View reviewed changes

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java Outdated Show resolved Hide resolved

hanishakoneru force-pushed the HDDS-1371 branch from 1b0ea0c to 5d0d70f Compare April 13, 2019 00:15

This comment has been minimized.

Sign in to view

bharatviswa504 reviewed Apr 16, 2019

View reviewed changes

hadoop-hdds/common/src/main/resources/ozone-default.xml Outdated Show resolved Hide resolved

hanishakoneru force-pushed the HDDS-1371 branch 2 times, most recently from 4d8bf5d to 920852e Compare May 30, 2019 23:37

This comment has been minimized.

Sign in to view

hanishakoneru force-pushed the HDDS-1371 branch from 920852e to 31b854b Compare June 3, 2019 21:37

This comment has been minimized.

Sign in to view

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBCheckpoint.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hadoop-hdds/common/src/main/resources/ozone-default.xml Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 4, 2019

View reviewed changes

hanishakoneru force-pushed the HDDS-1371 branch from 31b854b to 338b73b Compare June 5, 2019 18:11

bharatviswa504 reviewed Jun 5, 2019

View reviewed changes

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java Outdated Show resolved Hide resolved

bharatviswa504 approved these changes Jun 5, 2019

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java Outdated Show resolved Hide resolved

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Jun 5, 2019

View reviewed changes

hadoop-hdds/common/src/main/resources/ozone-default.xml Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

OMSnapshotProvider to download DB checkpoint from leader OM.

ab7dd74

hanishakoneru force-pushed the HDDS-1371 branch from 2063433 to ab7dd74 Compare June 6, 2019 22:45

This comment has been minimized.

Sign in to view

bharatviswa504 merged commit 0b115b6 into apache:trunk Jun 7, 2019

hemantk-12 mentioned this pull request Feb 24, 2024

HDDS-10300. InstallSnapshot may fail if OM metadata dir and OM DB dir are in different local storage partitions. apache/ozone#6226

Closed

HDDS-1371. Download RocksDB checkpoint from OM Leader to Follower. #703

HDDS-1371. Download RocksDB checkpoint from OM Leader to Follower. #703

Uh oh!

Conversation

hanishakoneru commented Apr 5, 2019

Uh oh!

hanishakoneru commented Apr 5, 2019

Uh oh!

This comment has been minimized.

arp7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hanishakoneru commented Apr 13, 2019

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

hanishakoneru commented Jun 3, 2019

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bharatviswa504 left a comment

Choose a reason for hiding this comment

Uh oh!

hanishakoneru commented Jun 5, 2019

Uh oh!

Uh oh!

hadoop-yetus commented Jun 5, 2019

Uh oh!

bharatviswa504 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hanishakoneru commented Jun 5, 2019

Uh oh!

This comment has been minimized.

This comment has been minimized.

hanishakoneru commented Jun 6, 2019

Uh oh!

hanishakoneru commented Jun 6, 2019

Uh oh!

This comment has been minimized.

hanishakoneru commented Jun 6, 2019

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jun 7, 2019

Uh oh!

bharatviswa504 commented Jun 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

bharatviswa504 left a comment •

edited

Loading