Skip to content

Conversation

@ChenSammi
Copy link
Contributor

… Datanode details from heartbeat

#1008

  1. support datanode UUID as datanode network location name in network topology cluster
  2. send back datanode networkname/networklocation to datanode in its register command response, so that datanode heartbeat message will carry the correct networkname/networklocation information.
  3. add TestStorageContainerManager#testScmProcessDatanodeHeartbeat to verify the heartbeat change.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 165 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
0 mvndep 75 Maven dependency ordering for branch
+1 mvninstall 510 trunk passed
+1 compile 246 trunk passed
+1 checkstyle 75 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 856 branch has no errors when building and testing our client artifacts.
+1 javadoc 168 trunk passed
0 spotbugs 326 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 525 trunk passed
_ Patch Compile Tests _
0 mvndep 35 Maven dependency ordering for patch
+1 mvninstall 445 the patch passed
+1 compile 266 the patch passed
+1 cc 266 the patch passed
+1 javac 266 the patch passed
+1 checkstyle 67 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 631 patch has no errors when building and testing our client artifacts.
+1 javadoc 148 the patch passed
+1 findbugs 546 the patch passed
_ Other Tests _
-1 unit 295 hadoop-hdds in the patch failed.
-1 unit 1830 hadoop-ozone in the patch failed.
+1 asflicense 47 The patch does not generate ASF License warnings.
7208
Reason Tests
Failed junit tests hadoop.hdds.scm.server.TestSCMBlockProtocolServer
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestBlockOutputStream
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.om.TestKeyManagerImpl
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/1/artifact/out/Dockerfile
GITHUB PR #1112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux bd073b38aadc 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 256fcc1
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/1/testReport/
Max. process+thread count 5272 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@arp7 arp7 requested a review from nandakumar131 July 17, 2019 21:52
@arp7 arp7 added the ozone label Jul 17, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 39 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 71 Maven dependency ordering for branch
+1 mvninstall 554 trunk passed
+1 compile 243 trunk passed
+1 checkstyle 69 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 869 branch has no errors when building and testing our client artifacts.
+1 javadoc 165 trunk passed
0 spotbugs 314 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 510 trunk passed
_ Patch Compile Tests _
0 mvndep 35 Maven dependency ordering for patch
+1 mvninstall 451 the patch passed
+1 compile 268 the patch passed
+1 cc 268 the patch passed
+1 javac 268 the patch passed
+1 checkstyle 75 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 1 The patch has no whitespace issues.
+1 shadedclient 681 patch has no errors when building and testing our client artifacts.
+1 javadoc 160 the patch passed
+1 findbugs 524 the patch passed
_ Other Tests _
-1 unit 296 hadoop-hdds in the patch failed.
-1 unit 1942 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
7196
Reason Tests
Failed junit tests hadoop.hdds.scm.container.placement.algorithms.TestContainerPlacementFactory
hadoop.ozone.om.TestKeyManagerImpl
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.container.server.TestSecureContainerServer
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/2/artifact/out/Dockerfile
GITHUB PR #1112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux d0ff89c6ae0f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 73e6ffc
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/2/testReport/
Max. process+thread count 5185 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 74 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 24 Maven dependency ordering for branch
+1 mvninstall 472 trunk passed
+1 compile 282 trunk passed
+1 checkstyle 77 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 966 branch has no errors when building and testing our client artifacts.
+1 javadoc 178 trunk passed
0 spotbugs 403 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 627 trunk passed
_ Patch Compile Tests _
0 mvndep 36 Maven dependency ordering for patch
-1 mvninstall 97 hadoop-hdds in the patch failed.
-1 mvninstall 212 hadoop-ozone in the patch failed.
-1 compile 154 hadoop-ozone in the patch failed.
-1 cc 154 hadoop-ozone in the patch failed.
-1 javac 154 hadoop-ozone in the patch failed.
+1 checkstyle 87 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 720 patch has no errors when building and testing our client artifacts.
+1 javadoc 156 the patch passed
-1 findbugs 251 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 329 hadoop-hdds in the patch failed.
-1 unit 277 hadoop-ozone in the patch failed.
+1 asflicense 37 The patch does not generate ASF License warnings.
5510
Reason Tests
Failed junit tests hadoop.hdds.scm.container.placement.algorithms.TestContainerPlacementFactory
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/Dockerfile
GITHUB PR #1112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 0a32b44a126f 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3dc256e
Default Java 1.8.0_212
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-compile-hadoop-ozone.txt
cc https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-compile-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/testReport/
Max. process+thread count 1338 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@ChenSammi ChenSammi requested a review from xiaoyuyao July 18, 2019 11:40
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 83 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 32 Maven dependency ordering for branch
+1 mvninstall 485 trunk passed
+1 compile 251 trunk passed
+1 checkstyle 69 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 937 branch has no errors when building and testing our client artifacts.
+1 javadoc 180 trunk passed
0 spotbugs 393 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 637 trunk passed
_ Patch Compile Tests _
0 mvndep 37 Maven dependency ordering for patch
+1 mvninstall 501 the patch passed
+1 compile 266 the patch passed
+1 cc 266 the patch passed
+1 javac 266 the patch passed
+1 checkstyle 78 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 738 patch has no errors when building and testing our client artifacts.
+1 javadoc 163 the patch passed
+1 findbugs 594 the patch passed
_ Other Tests _
-1 unit 354 hadoop-hdds in the patch failed.
-1 unit 1841 hadoop-ozone in the patch failed.
+1 asflicense 44 The patch does not generate ASF License warnings.
7490
Reason Tests
Failed junit tests hadoop.hdds.scm.container.placement.algorithms.TestContainerPlacementFactory
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.om.TestKeyManagerImpl
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.container.server.TestSecureContainerServer
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/4/artifact/out/Dockerfile
GITHUB PR #1112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 853d7d09bddd 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 20cf50c
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/4/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/4/testReport/
Max. process+thread count 5288 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you are removing this assertion because it makes the test flaky (node 2 and 3 may or may not be in the same rack, the only restriction is that they are in different one than nodes 0 and 1). If so, please also remove similar assertion from testFallback:

https://github.com/apache/hadoop/blob/f9266ccc4d592681cc8227a147799d3a08623493/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementRackAware.java#L200-L201

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments. Will remove last two assertions in testFallback.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 101 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 8 new or modified test files.
_ trunk Compile Tests _
0 mvndep 23 Maven dependency ordering for branch
+1 mvninstall 515 trunk passed
+1 compile 261 trunk passed
+1 checkstyle 71 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 951 branch has no errors when building and testing our client artifacts.
+1 javadoc 172 trunk passed
0 spotbugs 334 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 543 trunk passed
_ Patch Compile Tests _
0 mvndep 31 Maven dependency ordering for patch
+1 mvninstall 450 the patch passed
+1 compile 265 the patch passed
+1 cc 265 the patch passed
+1 javac 265 the patch passed
+1 checkstyle 80 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 758 patch has no errors when building and testing our client artifacts.
+1 javadoc 163 the patch passed
+1 findbugs 553 the patch passed
_ Other Tests _
+1 unit 340 hadoop-hdds in the patch passed.
-1 unit 2201 hadoop-ozone in the patch failed.
+1 asflicense 45 The patch does not generate ASF License warnings.
7695
Reason Tests
Failed junit tests hadoop.ozone.container.server.TestSecureContainerServer
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestOzoneRpcClient
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/5/artifact/out/Dockerfile
GITHUB PR #1112
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux c1ab3b738dac 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 4e66cb9
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/5/testReport/
Max. process+thread count 4581 (vs. ulimit of 5500)
modules C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1112/5/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@nandakumar131
Copy link
Contributor

Thanks @ChenSammi for working on this. Overall the change looks good to me. Few of the test failure seems related, can you please take a look at them?

@xiaoyuyao
Copy link
Contributor

+1 for the latest change. I will merge it shortly.

@xiaoyuyao xiaoyuyao merged commit 69a46a9 into apache:trunk Jul 19, 2019
asfgit pushed a commit that referenced this pull request Jul 19, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
Add a putsAll API to the metadata store API. This allows for a delayed flush() call for underlying stores that require a flush() after write such as Kafka.

Author: Daniel Nishimura <[email protected]>

Reviewers: mynameborat <[email protected]>

Closes apache#1112 from dnishimura/samza-2276-metadata-store-putall-api and squashes the following commits:

830b5994 [Daniel Nishimura] Update javadoc for MetadataStore#putAll
e52d884c [Daniel Nishimura] SAMZA-2271: Add Metdata store putAll API
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants