HDDS-1785. OOM error in Freon due to the concurrency handling #1085

adoroszlai · 2019-07-12T11:59:14Z

What changes were proposed in this pull request?

Change concurrency in Freon RandomKeyGenerator:

create a worker for each thread
let each worker create volumes, buckets and keys, without limiting "inner" objects to specific "outer" ones (eg. create key for any bucket)

Workers coordinate the items they create using "global" counters.

https://issues.apache.org/jira/browse/HDDS-1785

How was this patch tested?

Tested with various number of volumes/buckets/threads.

$ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5 --numOfThreads 1 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 1
Number of Buckets created: 100
Number of Keys added: 500
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,100
Average Time spent in bucket creation: 00:00:00,304
Average Time spent in key creation: 00:00:01,556
Average Time spent in key write: 00:00:53,509
Total bytes written: 5120000
Total Execution time: 00:01:01,537

$ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5 --numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 1
Number of Buckets created: 100
Number of Keys added: 500
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,003
Average Time spent in bucket creation: 00:00:00,229
Average Time spent in key creation: 00:00:00,273
Average Time spent in key write: 00:00:10,375
Total bytes written: 5120000
Total Execution time: 00:00:16,872

$ ozone freon rk --numOfVolumes 10 --numOfBuckets 10 --numOfKeys 500 --numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 10
Number of Buckets created: 100
Number of Keys added: 50000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,052
Average Time spent in bucket creation: 00:00:00,240
Average Time spent in key creation: 00:00:30,742
Average Time spent in key write: 00:10:04,146
Total bytes written: 512000000
Total Execution time: 00:10:42,463

$ ozone freon rk --numOfVolumes 100 --numOfBuckets 100 --numOfKeys 2 --numOfThreads 50 --replicationType=RATIS --factor=THREE
...
Number of Volumes created: 100
Number of Buckets created: 10000
Number of Keys added: 20000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,266
Average Time spent in bucket creation: 00:00:06,388
Average Time spent in key creation: 00:00:09,324
Average Time spent in key write: 00:03:44,925
Total bytes written: 204800000
Total Execution time: 00:04:11,735

hadoop-yetus · 2019-07-12T13:53:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	39	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	456	trunk passed
+1	compile	241	trunk passed
+1	checkstyle	65	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	803	branch has no errors when building and testing our client artifacts.
+1	javadoc	173	trunk passed
0	spotbugs	303	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	595	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	427	the patch passed
+1	compile	271	the patch passed
+1	javac	271	the patch passed
+1	checkstyle	81	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	657	patch has no errors when building and testing our client artifacts.
+1	javadoc	149	the patch passed
-1	findbugs	321	hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1	unit	277	hadoop-hdds in the patch passed.
-1	unit	1730	hadoop-ozone in the patch failed.
+1	asflicense	50	The patch does not generate ASF License warnings.
		6789

Reason	Tests
FindBugs	module:hadoop-ozone
	Exception is caught when Exception is not thrown in org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(long) At RandomKeyGenerator.java:is not thrown in org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(long) At RandomKeyGenerator.java:[line 728]
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.container.ozoneimpl.TestOzoneContainer
	hadoop.ozone.client.rpc.TestWatchForCommit
	hadoop.ozone.client.rpc.TestCommitWatcher
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/1/artifact/out/Dockerfile
GITHUB PR	#1085
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 0f7d4659da4b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f9fab9f`
Default Java	1.8.0_212
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/1/artifact/out/new-findbugs-hadoop-ozone.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/1/testReport/
Max. process+thread count	5295 (vs. ulimit of 5500)
modules	C: hadoop-ozone/tools U: hadoop-ozone/tools
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

adoroszlai · 2019-07-12T14:52:39Z

@elek @iamcaoxudong please review

hadoop-yetus · 2019-07-12T16:04:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	71	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	482	trunk passed
+1	compile	248	trunk passed
+1	checkstyle	68	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	905	branch has no errors when building and testing our client artifacts.
+1	javadoc	154	trunk passed
0	spotbugs	317	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	504	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	535	the patch passed
+1	compile	310	the patch passed
+1	javac	310	the patch passed
+1	checkstyle	79	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	724	patch has no errors when building and testing our client artifacts.
+1	javadoc	153	the patch passed
+1	findbugs	608	the patch passed
		_ Other Tests _
+1	unit	353	hadoop-hdds in the patch passed.
-1	unit	198	hadoop-ozone in the patch failed.
+1	asflicense	36	The patch does not generate ASF License warnings.
		5596

Reason	Tests
Failed junit tests	hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/2/artifact/out/Dockerfile
GITHUB PR	#1085
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e9107caa267d 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `190e434`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/2/testReport/
Max. process+thread count	370 (vs. ulimit of 5500)
modules	C: hadoop-ozone/tools U: hadoop-ozone/tools
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

adoroszlai · 2019-07-16T16:15:09Z

(force-pushed to pick up recent fixes from trunk)

hadoop-yetus · 2019-07-16T18:03:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	40	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	524	trunk passed
+1	compile	264	trunk passed
+1	checkstyle	72	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	850	branch has no errors when building and testing our client artifacts.
+1	javadoc	153	trunk passed
0	spotbugs	322	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	525	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	452	the patch passed
+1	compile	257	the patch passed
+1	javac	257	the patch passed
+1	checkstyle	62	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	650	patch has no errors when building and testing our client artifacts.
+1	javadoc	156	the patch passed
+1	findbugs	514	the patch passed
		_ Other Tests _
+1	unit	286	hadoop-hdds in the patch passed.
-1	unit	1689	hadoop-ozone in the patch failed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		6705

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/3/artifact/out/Dockerfile
GITHUB PR	#1085
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 83b8f3be5986 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c5e3ab5`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/3/testReport/
Max. process+thread count	5390 (vs. ulimit of 5500)
modules	C: hadoop-ozone/tools U: hadoop-ozone/tools
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1085/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek

+1 Thank you @adoroszlai the fix for this problem.

I really like this approach as it can guarantee predictable name for the created keys/volumes/buckets (using pre defined postfix instead of a random string).

With refactoring out the validation from the random key generator we can run chaos testing easily:

generate 10M keys with freon
run validator (check the existence / content of the generated keys)
kill one datanode and restart it without the persitent data
wait for the replication
validate the keys

and we can repeat 3-5...

elek · 2019-07-17T12:48:49Z

I tested it in kubernetes with generating 25M keys, compared to the previous version I was able to generate the keys...

I will merge it soon...

Closes #1085 (cherry picked from commit 256fcc1)

* Upgrading calcite to 1.19 * Fixing the flatten testcase and adding another test * minor fixup * Removing a flaky test

Closes apache#1085

nandakumar131 assigned adoroszlai Jul 12, 2019

nandakumar131 added the ozone label Jul 12, 2019

adoroszlai force-pushed the HDDS-1785 branch from 261069b to 6961966 Compare July 12, 2019 14:30

HDDS-1785. OOM error in Freon due to the concurrency handling

d4eb682

adoroszlai force-pushed the HDDS-1785 branch from 6961966 to d4eb682 Compare July 16, 2019 16:11

elek approved these changes Jul 17, 2019

View reviewed changes

elek closed this in 256fcc1 Jul 17, 2019

elek pushed a commit that referenced this pull request Jul 17, 2019

HDDS-1785. OOM error in Freon due to the concurrency handling

669c447

Closes #1085 (cherry picked from commit 256fcc1)

adoroszlai deleted the HDDS-1785 branch July 25, 2019 16:16

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019

Samza-2253: Upgrading calcite to 1.19 (apache#1085)

1908622

* Upgrading calcite to 1.19 * Fixing the flatten testcase and adding another test * minor fixup * Removing a flaky test

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-1785. OOM error in Freon due to the concurrency handling

2fe11b3

Closes apache#1085

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1785. OOM error in Freon due to the concurrency handling #1085

HDDS-1785. OOM error in Freon due to the concurrency handling #1085

Uh oh!

adoroszlai commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

adoroszlai commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

adoroszlai commented Jul 16, 2019

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

elek left a comment

Uh oh!

elek commented Jul 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-1785. OOM error in Freon due to the concurrency handling #1085

HDDS-1785. OOM error in Freon due to the concurrency handling #1085

Uh oh!

Conversation

adoroszlai commented Jul 12, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

adoroszlai commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

adoroszlai commented Jul 16, 2019

Uh oh!

hadoop-yetus commented Jul 16, 2019

Uh oh!

elek left a comment

Choose a reason for hiding this comment

Uh oh!

elek commented Jul 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants