Skip to content

Conversation

@iwasakims
Copy link
Member

this is backport to branch-3.3 of HADOOP-17424.

@iwasakims
Copy link
Member Author

htrace-core-3.1.0-incubating is bundled as transitive dependency of hadoop-yarn-server-timelineservice-hbase while it does not appear in the regular classpath.

$ mvn clean install -Pdist -DskipShade -DskipTests

$ find hadoop-dist/target/hadoop-3.3.2-SNAPSHOT -iname '*htrace*'
hadoop-dist/target/hadoop-3.3.2-SNAPSHOT/share/hadoop/yarn/timelineservice/lib/htrace-core-3.1.0-incubating.jar

$ hadoop-dist/target/hadoop-3.3.2-SNAPSHOT/bin/hadoop classpath --glob | sed -z 's/:/\n/g' | grep -i htrace

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 8 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 12m 9s Maven dependency ordering for branch
+1 💚 mvninstall 21m 5s branch-3.3 passed
+1 💚 compile 17m 22s branch-3.3 passed
+1 💚 checkstyle 2m 46s branch-3.3 passed
+1 💚 mvnsite 8m 35s branch-3.3 passed
+1 💚 javadoc 8m 7s branch-3.3 passed
+0 🆗 spotbugs 0m 40s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 40s branch/hadoop-client-modules/hadoop-client-api no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 38s branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 38s branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs output file (spotbugsXml.xml)
+1 💚 shadedclient 22m 55s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 13m 44s the patch passed
+1 💚 compile 16m 47s the patch passed
-1 ❌ cc 16m 47s /results-compile-cc-root.txt root generated 7 new + 152 unchanged - 7 fixed = 159 total (was 159)
+1 💚 javac 16m 47s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 42s /results-checkstyle-root.txt root: The patch generated 6 new + 1065 unchanged - 59 fixed = 1071 total (was 1124)
+1 💚 mvnsite 8m 10s the patch passed
+1 💚 xml 0m 10s The patch has no ill-formed XML file.
+1 💚 javadoc 8m 2s the patch passed
+0 🆗 spotbugs 0m 37s hadoop-project has no data from spotbugs
+0 🆗 spotbugs 0m 39s hadoop-client-modules/hadoop-client-api has no data from spotbugs
+0 🆗 spotbugs 0m 39s hadoop-client-modules/hadoop-client-runtime has no data from spotbugs
+0 🆗 spotbugs 0m 39s hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
+0 🆗 spotbugs 0m 39s hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs
+0 🆗 spotbugs 0m 39s hadoop-client-modules/hadoop-client-check-test-invariants has no data from spotbugs
+1 💚 shadedclient 22m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 36s hadoop-project in the patch passed.
+1 💚 unit 17m 9s hadoop-common in the patch passed.
+1 💚 unit 2m 35s hadoop-hdfs-client in the patch passed.
+1 💚 unit 185m 20s hadoop-hdfs in the patch passed.
+1 💚 unit 0m 50s hadoop-client-api in the patch passed.
+1 💚 unit 0m 50s hadoop-client-runtime in the patch passed.
+1 💚 unit 0m 48s hadoop-client-check-invariants in the patch passed.
+1 💚 unit 0m 50s hadoop-client-minicluster in the patch passed.
+1 💚 unit 0m 48s hadoop-client-check-test-invariants in the patch passed.
+1 💚 asflicense 1m 11s The patch does not generate ASF License warnings.
406m 35s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/1/artifact/out/Dockerfile
GITHUB PR #3520
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle cc buflint bufcompat
uname Linux a773a2441377 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 6889862f819ded53b36de328a7a6287deddbdfed
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/1/testReport/
Max. process+thread count 3289 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a few checkstyles, but I don't want to diverge from what's in trunk. The only changes I'd recommend is purging those unused imports, as they only complicate further backports (especially the .thirdparty guava)

./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsTracer.java:20:import org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;:8: Unused import - org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting. [UnusedImports]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/Tracer.java:25:  private static final Tracer globalTracer = null;:31: Name 'globalTracer' must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/Tracer.java:80:    static Tracer globalTracer;:19: Variable 'globalTracer' must be private and have accessor methods. [VisibilityModifier]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceScope.java:23:  Span span;:8: Variable 'span' must be private and have accessor methods. [VisibilityModifier]
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java:304:  public void addTraceParent(Span span) {:35: 'span' hides a field. [HiddenField]
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java:35:import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.DataTransferTraceInfoProto;:8: Unused import - org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.DataTransferTraceInfoProto. [UnusedImports]

+1 pending the import tuning

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ patch 0m 19s #3520 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR #3520
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/2/console
versions git=2.17.1
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

smengcl and others added 2 commits October 11, 2021 02:13
(cherry picked from commit 1a205cc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java
@iwasakims iwasakims force-pushed the HADOOP-17424-branch-3.3 branch from f7d078e to 8ddf763 Compare October 11, 2021 02:22
@iwasakims
Copy link
Member Author

rebased.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 8 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 12m 11s Maven dependency ordering for branch
+1 💚 mvninstall 21m 17s branch-3.3 passed
+1 💚 compile 17m 24s branch-3.3 passed
+1 💚 checkstyle 2m 50s branch-3.3 passed
+1 💚 mvnsite 8m 13s branch-3.3 passed
+1 💚 javadoc 8m 0s branch-3.3 passed
+0 🆗 spotbugs 0m 42s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 36s branch/hadoop-client-modules/hadoop-client-api no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 39s branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs output file (spotbugsXml.xml)
+1 💚 shadedclient 23m 20s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 13m 34s the patch passed
+1 💚 compile 16m 44s the patch passed
-1 ❌ cc 16m 44s /results-compile-cc-root.txt root generated 17 new + 142 unchanged - 17 fixed = 159 total (was 159)
+1 💚 javac 16m 44s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 42s /results-checkstyle-root.txt root: The patch generated 4 new + 1065 unchanged - 59 fixed = 1069 total (was 1124)
+1 💚 mvnsite 8m 15s the patch passed
+1 💚 xml 0m 11s The patch has no ill-formed XML file.
+1 💚 javadoc 8m 1s the patch passed
+0 🆗 spotbugs 0m 37s hadoop-project has no data from spotbugs
+0 🆗 spotbugs 0m 40s hadoop-client-modules/hadoop-client-api has no data from spotbugs
+0 🆗 spotbugs 0m 40s hadoop-client-modules/hadoop-client-runtime has no data from spotbugs
+0 🆗 spotbugs 0m 37s hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
+0 🆗 spotbugs 0m 38s hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs
+0 🆗 spotbugs 0m 37s hadoop-client-modules/hadoop-client-check-test-invariants has no data from spotbugs
+1 💚 shadedclient 22m 57s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 37s hadoop-project in the patch passed.
+1 💚 unit 17m 4s hadoop-common in the patch passed.
+1 💚 unit 2m 32s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 186m 47s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 unit 0m 45s hadoop-client-api in the patch passed.
+1 💚 unit 0m 43s hadoop-client-runtime in the patch passed.
+1 💚 unit 0m 41s hadoop-client-check-invariants in the patch passed.
+1 💚 unit 0m 40s hadoop-client-minicluster in the patch passed.
+1 💚 unit 0m 37s hadoop-client-check-test-invariants in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
407m 21s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.datanode.TestBPOfferService
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/3/artifact/out/Dockerfile
GITHUB PR #3520
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle cc buflint bufcompat
uname Linux 2b23589d0b8a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 8ddf763
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/3/testReport/
Max. process+thread count 3429 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3520/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@iwasakims
Copy link
Member Author

Test failures are not related to the patch and not reproducible on my local.

$ mvn test -Dtest=TestUnderReplicatedBlocks,TestBalancerWithHANameNodes,TestBPOfferService
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.479 s - in org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
[INFO] Running org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.326 s - in org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
[INFO] Running org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 111.393 s - in org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0

@iwasakims
Copy link
Member Author

I'm merging this based on the +1 of @steveloughran.

@iwasakims iwasakims merged commit 9e2936f into apache:branch-3.3 Oct 11, 2021
deepakdamri pushed a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
(cherry picked from commit 1a205cc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <[email protected]>
deepakdamri added a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants