Skip to content

Conversation

supratimdeka
Copy link
Contributor

https://issues.apache.org/jira/browse/HDDS-1094

Added an alternate ChunkManager implementation which drops all chunk writes without writing to disk. Chunk Reads are cooked up zero-filled buffers.
The goal of this infrastructure is to enable high-throughput tests and stress the pipeline (including the Ozone metadata components) without using faster storage devices like flash drives.

Added an extension to TestDataValidate (with the RandomKeyGenerator) to test the alternate ChunkManager.

@elek elek added the ozone label Aug 21, 2019
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required because you are added a test for freon with the null ChunkManager?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is required because each individual test case in TestDataValidate sets this parameter in the RandomKeyGenerator builder.

Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this nice test hook @supratimdeka.

The patch looks pretty good to me. A few minor comments.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 109 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 63 Maven dependency ordering for branch
+1 mvninstall 631 trunk passed
+1 compile 363 trunk passed
+1 checkstyle 67 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 857 branch has no errors when building and testing our client artifacts.
+1 javadoc 151 trunk passed
0 spotbugs 414 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 610 trunk passed
_ Patch Compile Tests _
0 mvndep 28 Maven dependency ordering for patch
+1 mvninstall 567 the patch passed
+1 compile 402 the patch passed
+1 javac 402 the patch passed
-0 checkstyle 36 hadoop-hdds: The patch generated 12 new + 0 unchanged - 0 fixed = 12 total (was 0)
-0 checkstyle 36 hadoop-ozone: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
+1 mvnsite 0 the patch passed
+1 whitespace 2 The patch has no whitespace issues.
+1 shadedclient 635 patch has no errors when building and testing our client artifacts.
+1 javadoc 145 the patch passed
-1 findbugs 205 hadoop-hdds generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
-1 unit 235 hadoop-hdds in the patch failed.
-1 unit 2746 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
8522
Reason Tests
FindBugs module:hadoop-hdds
Possible doublecheck on org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerFactory.instance in org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerFactory.getChunkManager(Configuration, boolean) At ChunkManagerFactory.java:org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerFactory.getChunkManager(Configuration, boolean) At ChunkManagerFactory.java:[lines 46-48]
Failed junit tests hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.container.server.TestSecureContainerServer
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.TestOzoneConfigurationFields
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.Test2WayCommitInRatis
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.TestMiniOzoneCluster
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/Dockerfile
GITHUB PR #1323
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux a27542af2401 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ee7c261
Default Java 1.8.0_222
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/diff-checkstyle-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/new-findbugs-hadoop-hdds.html
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/testReport/
Max. process+thread count 5030 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-ozone/tools U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 69 Maven dependency ordering for branch
+1 mvninstall 658 trunk passed
+1 compile 382 trunk passed
+1 checkstyle 78 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 959 branch has no errors when building and testing our client artifacts.
+1 javadoc 180 trunk passed
0 spotbugs 443 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 660 trunk passed
-0 patch 479 Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
0 mvndep 29 Maven dependency ordering for patch
+1 mvninstall 631 the patch passed
+1 compile 396 the patch passed
+1 javac 396 the patch passed
-0 checkstyle 40 hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 754 patch has no errors when building and testing our client artifacts.
-1 javadoc 80 hadoop-hdds generated 2 new + 16 unchanged - 0 fixed = 18 total (was 16)
-1 findbugs 382 hadoop-ozone in the patch failed.
_ Other Tests _
+1 unit 343 hadoop-hdds in the patch passed.
-1 unit 415 hadoop-ozone in the patch failed.
+1 asflicense 40 The patch does not generate ASF License warnings.
6631
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/artifact/out/Dockerfile
GITHUB PR #1323
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 1bda1577d9f1 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3329257
Default Java 1.8.0_222
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/artifact/out/diff-checkstyle-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/testReport/
Max. process+thread count 864 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-ozone/tools U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1323/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@arp7 arp7 merged commit 1407414 into apache:trunk Aug 28, 2019
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants