Skip to content

Conversation

@elek
Copy link
Member

@elek elek commented Apr 4, 2019

@elek elek added the ozone label Apr 4, 2019
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make 1000_000_000 configurable?

Copy link
Contributor

@xiaoyuyao xiaoyuyao Apr 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: LOG.info("Executing {}", mountCommand);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: LOG.info("Executing {}", umountCommand);

@xiaoyuyao
Copy link
Contributor

Thanks @elek for working on this, the change LGTM overall, just few minor comments.
Please also fix the the nightly test run. +1 after that.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 44 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 72 Maven dependency ordering for branch
+1 mvninstall 1137 trunk passed
+1 compile 1359 trunk passed
+1 checkstyle 152 trunk passed
-1 mvnsite 117 hadoop-ozone in trunk failed.
-1 mvnsite 46 integration-test in trunk failed.
+1 shadedclient 781 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone hadoop-ozone/dist hadoop-ozone/integration-test
+1 findbugs 163 trunk passed
+1 javadoc 267 trunk passed
_ Patch Compile Tests _
0 mvndep 28 Maven dependency ordering for patch
-1 mvninstall 174 hadoop-ozone in the patch failed.
-1 mvninstall 28 integration-test in the patch failed.
+1 compile 1002 the patch passed
+1 cc 1002 the patch passed
+1 javac 1002 the patch passed
+1 checkstyle 152 the patch passed
-1 hadolint 0 The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
-1 mvnsite 123 hadoop-ozone in the patch failed.
-1 mvnsite 47 integration-test in the patch failed.
+1 shellcheck 26 There were no new shellcheck issues.
+1 shelldocs 33 There were no new shelldocs issues.
+1 whitespace 1 The patch has no whitespace issues.
+1 xml 7 The patch has no ill-formed XML file.
+1 shadedclient 750 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone hadoop-ozone/dist hadoop-ozone/integration-test
+1 findbugs 252 the patch passed
+1 javadoc 315 the patch passed
_ Other Tests _
+1 unit 95 common in the patch passed.
-1 unit 173 hadoop-ozone in the patch failed.
+1 unit 53 common in the patch passed.
+1 unit 45 csi in the patch passed.
+1 unit 39 dist in the patch passed.
-1 unit 47 integration-test in the patch failed.
+1 asflicense 51 The patch does not generate ASF License warnings.
8439
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/Dockerfile
GITHUB PR #693
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
uname Linux 7ba080e16418 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c35abcd
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/branch-mvnsite-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
shellcheck v0.4.6
findbugs v3.1.0-RC1
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
hadolint https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/diff-patch-hadolint.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvnsite-hadoop-ozone.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-unit-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/testReport/
Max. process+thread count 338 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-693/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in end?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Removed the modification of ozone-default.xml as I switched to use the annotation based configs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tmp directory might be emptied by admins and others without knowledge of this file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK it shouldn't be a problem. This is a socket which is created by the process to listen on and even if it's deleted the socket will work well (until the close).

But I modified it to /var/lib/csi.sock. It can be more meaningful (and maybe more safe If my previous statement is wrong ;-) )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we use the new config annotations instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. I uploaded a new version with config annotations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update this with @config annotations?

@ajayydv
Copy link
Contributor

ajayydv commented May 21, 2019

@elek thanks for the patch. LGTM, added few comments. Seems this needs a rebase as well.

@elek elek marked this pull request as ready for review May 29, 2019 12:39
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 126 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 125 Maven dependency ordering for branch
+1 mvninstall 683 trunk passed
+1 compile 295 trunk passed
+1 checkstyle 105 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 845 branch has no errors when building and testing our client artifacts.
+1 javadoc 168 trunk passed
0 spotbugs 304 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 517 trunk passed
_ Patch Compile Tests _
0 mvndep 39 Maven dependency ordering for patch
+1 mvninstall 563 the patch passed
+1 compile 322 the patch passed
+1 cc 322 the patch passed
-1 javac 214 hadoop-ozone generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
-0 checkstyle 46 hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
-1 hadolint 3 The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1 mvnsite 0 the patch passed
+1 shellcheck 25 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 10 The patch has no ill-formed XML file.
+1 shadedclient 700 patch has no errors when building and testing our client artifacts.
+1 javadoc 187 the patch passed
+1 findbugs 554 the patch passed
_ Other Tests _
-1 unit 370 hadoop-hdds in the patch failed.
-1 unit 1442 hadoop-ozone in the patch failed.
+1 asflicense 53 The patch does not generate ASF License warnings.
7551
Reason Tests
Failed junit tests hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware
hadoop.hdds.scm.pipeline.TestNodeFailure
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.hdds.scm.pipeline.TestSCMRestart
hadoop.ozone.om.TestOzoneManagerHA
hadoop.hdds.scm.pipeline.TestNode2PipelineMap
hadoop.ozone.om.TestOzoneManager
hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/Dockerfile
GITHUB PR #693
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
uname Linux 55952ec91b3e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 544876f
Default Java 1.8.0_212
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-compile-javac-hadoop-ozone.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-checkstyle-hadoop-ozone.txt
hadolint https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/diff-patch-hadolint.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/testReport/
Max. process+thread count 2112 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-693/4/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 hadolint=1.11.1-0-g0e692dd
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer
Copy link
Contributor

+1, @elek Can you please fix the checkstyle issues while committing ? Thanks

@elek
Copy link
Member Author

elek commented May 31, 2019

Thanks the review @anuengineer I am merging it right now (checkstyle issues are fixed).

And we can continue the work:

  • HDDS-1508: Example k8s files to deploy it
  • Documentation is required in the docs
  • We can make it more configurable (support other fuse drivers not just goofys?)

@elek elek closed this in 1ae062c May 31, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 71 Maven dependency ordering for branch
+1 mvninstall 528 trunk passed
+1 compile 305 trunk passed
+1 checkstyle 101 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 804 branch has no errors when building and testing our client artifacts.
+1 javadoc 179 trunk passed
0 spotbugs 309 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 503 trunk passed
_ Patch Compile Tests _
0 mvndep 37 Maven dependency ordering for patch
+1 mvninstall 540 the patch passed
+1 compile 315 the patch passed
+1 cc 315 the patch passed
-1 javac 209 hadoop-ozone generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 checkstyle 101 the patch passed
-1 hadolint 1 The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1 mvnsite 0 the patch passed
+1 shellcheck 25 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 xml 7 The patch has no ill-formed XML file.
+1 shadedclient 694 patch has no errors when building and testing our client artifacts.
+1 javadoc 183 the patch passed
+1 findbugs 618 the patch passed
_ Other Tests _
+1 unit 224 hadoop-hdds in the patch passed.
-1 unit 1467 hadoop-ozone in the patch failed.
+1 asflicense 57 The patch does not generate ASF License warnings.
7058
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/Dockerfile
GITHUB PR #693
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs cc hadolint
uname Linux 6442ceeebd8b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 52128e3
Default Java 1.8.0_212
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/diff-compile-javac-hadoop-ozone.txt
hadolint https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/diff-patch-hadolint.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/testReport/
Max. process+thread count 5294 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-ozone hadoop-ozone/common hadoop-ozone/csi hadoop-ozone/dist hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-693/5/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1 hadolint=1.11.1-0-g0e692dd
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
…vel API

Author: Prateek Maheshwari <[email protected]>

Reviewers: Yi Pan <[email protected]>, Jagadish Venkataraman <[email protected]>

Closes apache#693 from prateekm/intermediate-stream-serde
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants