Skip to content

Conversation

@kittinanasi
Copy link
Contributor

No description provided.

@elek
Copy link
Member

elek commented Apr 9, 2019

/label ozone

@elek elek added the ozone label Apr 9, 2019
@elek
Copy link
Member

elek commented Apr 9, 2019

/retest

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 28 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 21 Maven dependency ordering for branch
+1 mvninstall 1029 trunk passed
+1 compile 974 trunk passed
+1 checkstyle 189 trunk passed
-1 mvnsite 32 container-service in trunk failed.
+1 shadedclient 1100 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 30 container-service in trunk failed.
+1 javadoc 184 trunk passed
_ Patch Compile Tests _
0 mvndep 26 Maven dependency ordering for patch
+1 mvninstall 161 the patch passed
+1 compile 977 the patch passed
+1 javac 977 the patch passed
+1 checkstyle 180 the patch passed
+1 mvnsite 187 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 637 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 261 the patch passed
+1 javadoc 46 common in the patch passed.
+1 javadoc 41 hadoop-hdds_container-service generated 0 new + 7 unchanged - 2 fixed = 7 total (was 9)
+1 javadoc 31 tools in the patch passed.
+1 javadoc 26 integration-test in the patch passed.
+1 javadoc 34 ozone-manager in the patch passed.
_ Other Tests _
+1 unit 79 common in the patch passed.
-1 unit 77 container-service in the patch failed.
+1 unit 35 tools in the patch passed.
-1 unit 691 integration-test in the patch failed.
+1 unit 52 ozone-manager in the patch passed.
+1 asflicense 44 The patch does not generate ASF License warnings.
7406
Reason Tests
Failed junit tests hadoop.ozone.container.common.TestDatanodeStateMachine
hadoop.ozone.om.TestOzoneManagerHA
hadoop.ozone.ozShell.TestOzoneShell
hadoop.ozone.om.TestOzoneManager
hadoop.ozone.client.rpc.TestReadRetries
hadoop.ozone.om.TestScmChillMode
hadoop.ozone.web.client.TestKeysRatis
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestBlockOutputStream
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.om.TestContainerReportWithKeys
hadoop.ozone.om.TestMultipleContainerReadWrite
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler
hadoop.ozone.client.rpc.TestContainerStateMachine
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
hadoop.ozone.client.rpc.TestBCSID
hadoop.ozone.web.client.TestKeys
hadoop.ozone.client.rpc.TestContainerStateMachineFailures
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.om.TestOmBlockVersioning
hadoop.ozone.web.TestOzoneRestWithMiniCluster
hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/artifact/out/Dockerfile
GITHUB PR #713
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 6984318eb047 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 73f43ac
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/artifact/out/branch-mvnsite-hadoop-hdds_container-service.txt
findbugs v3.1.0-RC1
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/artifact/out/patch-unit-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/testReport/
Max. process+thread count 3775 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-713/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 27 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 84 Maven dependency ordering for branch
+1 mvninstall 1865 trunk passed
+1 compile 2200 trunk passed
+1 checkstyle 281 trunk passed
+1 mvnsite 245 trunk passed
+1 shadedclient 1420 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 744 trunk passed
+1 javadoc 565 trunk passed
_ Patch Compile Tests _
0 mvndep 35 Maven dependency ordering for patch
+1 mvninstall 469 the patch passed
+1 compile 1673 the patch passed
+1 javac 1673 the patch passed
+1 checkstyle 231 the patch passed
+1 mvnsite 349 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 1268 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 858 the patch passed
+1 javadoc 775 the patch passed
_ Other Tests _
+1 unit 101 common in the patch passed.
+1 unit 69 container-service in the patch passed.
+1 unit 30 tools in the patch passed.
-1 unit 883 integration-test in the patch failed.
+1 unit 56 ozone-manager in the patch passed.
+1 asflicense 45 The patch does not generate ASF License warnings.
14428
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.ozShell.TestOzoneDatanodeShell
hadoop.ozone.TestMiniChaosOzoneCluster
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/2/artifact/out/Dockerfile
GITHUB PR #713
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 933c32650117 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / a0468c5
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-713/2/testReport/
Max. process+thread count 3886 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-713/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 28 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 60 Maven dependency ordering for branch
+1 mvninstall 1017 trunk passed
+1 compile 954 trunk passed
+1 checkstyle 183 trunk passed
+1 mvnsite 218 trunk passed
+1 shadedclient 1095 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 219 trunk passed
+1 javadoc 170 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
+1 mvninstall 155 the patch passed
+1 compile 926 the patch passed
+1 javac 926 the patch passed
+1 checkstyle 192 the patch passed
+1 mvnsite 210 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 631 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 251 the patch passed
+1 javadoc 170 the patch passed
_ Other Tests _
+1 unit 76 common in the patch passed.
-1 unit 80 container-service in the patch failed.
+1 unit 32 tools in the patch passed.
-1 unit 1201 integration-test in the patch failed.
+1 unit 60 ozone-manager in the patch passed.
+1 asflicense 53 The patch does not generate ASF License warnings.
7867
Reason Tests
Failed junit tests hadoop.ozone.container.common.TestDatanodeStateMachine
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/3/artifact/out/Dockerfile
GITHUB PR #713
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 8ff0e5f4c43b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / bbdbc7a
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/3/artifact/out/patch-unit-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/3/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-713/3/testReport/
Max. process+thread count 4418 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-713/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@kittinanasi kittinanasi force-pushed the HDDS-1192 branch 2 times, most recently from 335811f to 38e5b23 Compare April 15, 2019 12:08
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 23 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 168 Maven dependency ordering for branch
+1 mvninstall 1167 trunk passed
+1 compile 1111 trunk passed
+1 checkstyle 144 trunk passed
+1 mvnsite 246 trunk passed
+1 shadedclient 1150 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 219 trunk passed
+1 javadoc 164 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
+1 mvninstall 178 the patch passed
+1 compile 907 the patch passed
+1 javac 907 the patch passed
+1 checkstyle 140 the patch passed
+1 mvnsite 194 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 739 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 251 the patch passed
+1 javadoc 159 the patch passed
_ Other Tests _
+1 unit 90 common in the patch passed.
+1 unit 69 container-service in the patch passed.
+1 unit 30 tools in the patch passed.
-1 unit 818 integration-test in the patch failed.
+1 unit 53 ozone-manager in the patch passed.
+1 asflicense 46 The patch does not generate ASF License warnings.
7970
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/artifact/out/Dockerfile
GITHUB PR #713
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux ad6d81684bbe 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c4c16ca
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/testReport/
Max. process+thread count 4275 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot the patch @kittinanasi This is a big step forward to start full Ozone cluster from any IDE. I tested and worked well (without adding ozone-site.xml to the classpath!)

Overall it looks good to me. I like it. I have one minor proposal to simplify the change (I think we can do exactly the same without adding the new method GenericParentCommand. Which is not a big deal but we need to add the new method to all the subcommands).

With that approach initConf also can be removed and all the HddsDatanodeService.createHddsDatanodeService can be simplified...

The failure of TestAuditParser.testHelp() seems to be related (the output of the --help is asserted but now it contains the --conf. Maybe we can simplify the assertion).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this part can be simplified (but fix me if I am wrong) with using two start method:

 /**
   * Starts HddsDatanode services.
   *
   * @param service The service instance invoking this method
   */
  @Override
  public void start(Object service) {
    if (service instanceof Configurable) {
      start(new OzoneConfiguration(((Configurable) service).getConf()));
    } else {
      start(new OzoneConfiguration());
    }
  }

  public void start(OzoneConfiguration conf) {
    this.conf = conf;
    DefaultMetricsSystem.initialize("HddsDatanode");
    OzoneConfiguration.activate();
    if (HddsUtils.isHddsEnabled(conf)) {

In this case we don't need to add the additional method to the GenericParentCommand so it can be as simple as now.

And from the new call() method we can use the createOzoneConfiguration()

  @Override
  public Void call() throws Exception {
    if (printBanner) {
      StringUtils
          .startupShutdownMessage(HddsDatanodeService.class, args, LOG);
    }
    start(createOzoneConfiguration());
    join();
    return null;
  }

initConf also can be removed with this approach (IMHO)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good idea, I simplified this part, but I introduced a setConfiguration method here as well so that I don't have to modify MiniOzoneCluster that much.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LIKE. Thanks to fix this part (and the similar items). It's better to use the included CommandLine.ParameterException...

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 @author 1 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 61 Maven dependency ordering for branch
+1 mvninstall 1129 trunk passed
+1 compile 1025 trunk passed
+1 checkstyle 145 trunk passed
+1 mvnsite 283 trunk passed
+1 shadedclient 1188 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 275 trunk passed
+1 javadoc 188 trunk passed
_ Patch Compile Tests _
0 mvndep 24 Maven dependency ordering for patch
+1 mvninstall 210 the patch passed
+1 compile 1053 the patch passed
+1 javac 1053 the patch passed
+1 checkstyle 143 the patch passed
+1 mvnsite 240 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 711 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 311 the patch passed
+1 javadoc 184 the patch passed
_ Other Tests _
+1 unit 94 common in the patch passed.
+1 unit 71 container-service in the patch passed.
+1 unit 29 tools in the patch passed.
-1 unit 461 integration-test in the patch failed.
+1 unit 54 ozone-manager in the patch passed.
+1 unit 99 tools in the patch passed.
+1 asflicense 102 The patch does not generate ASF License warnings.
7970
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/5/artifact/out/Dockerfile
GITHUB PR #713
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 2fb4a51c7469 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / b979fdd
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-713/5/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-713/5/testReport/
Max. process+thread count 2494 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/tools U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-713/5/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@kittinanasi
Copy link
Contributor Author

TestOzoneAtRestEncryption test failure does not seem related.

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thanks @kittinanasi the update.

I will commit it soon (there is one checkstyle violation, I am fixing it during the commit).

Neither the acceptance tests nor the unit tests are related. All the unit tests are related to RAFT ring.

@elek elek closed this in 3f787cd Apr 24, 2019
@kittinanasi
Copy link
Contributor Author

Thanks @elek for committing and for fixing the remaining checkstyle issue!

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
Author: Jagadish <[email protected]>

Reviewers: Jagadish<[email protected]>

Closes apache#713 from vjagadish1989/website-reorg11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants