Skip to content

HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) #4246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 16, 2022
Merged

HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) #4246

merged 1 commit into from
May 16, 2022

Conversation

@saintstack
Copy link
Contributor

@saintstack saintstack commented Apr 28, 2022

Description of PR

Cherry-pick of 9ed8d60

How was this patch tested?

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

hadoop-yetus commented Apr 29, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 14m 29s Maven dependency ordering for branch
+1 💚 mvninstall 23m 53s branch-3.3 passed
+1 💚 compile 18m 9s branch-3.3 passed
+1 💚 checkstyle 3m 23s branch-3.3 passed
+1 💚 mvnsite 20m 24s branch-3.3 passed
+1 💚 javadoc 8m 2s branch-3.3 passed
+1 💚 spotbugs 35m 29s branch-3.3 passed
+1 💚 shadedclient 59m 16s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for patch
+1 💚 mvninstall 22m 45s the patch passed
+1 💚 compile 17m 35s the patch passed
+1 💚 javac 17m 35s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 3m 6s the patch passed
+1 💚 mvnsite 19m 49s the patch passed
+1 💚 javadoc 7m 57s the patch passed
+1 💚 spotbugs 35m 39s the patch passed
+1 💚 shadedclient 55m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 579m 22s /patch-unit-root.txt root in the patch failed.
+0 🆗 asflicense 1m 41s ASF License check generated no output?
874m 55s
Reason Tests
Failed junit tests hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
hadoop.yarn.server.resourcemanager.TestRMHA
hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.TestClientRMService
hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
hadoop.yarn.client.TestGetGroups
hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
hadoop.crypto.key.kms.server.TestKMS
hadoop.crypto.key.kms.server.TestKMSWithZK
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.namenode.TestFileTruncate
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.mapreduce.v2.TestMRJobs
hadoop.mapreduce.v2.TestMRJobsWithProfiler
hadoop.mapreduce.v2.TestRMNMInfo
hadoop.mapreduce.v2.TestMRAppWithCombiner
hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp
hadoop.mapreduce.v2.TestUberAM
hadoop.mapred.TestLocalDistributedCacheManager
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/1/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense codespell compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname Linux 1aea2f037823 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 43230dd82870bbe797cba510b53fbda41f16ad39
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/1/testReport/
Max. process+thread count 3196 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Apr 30, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 14m 31s Maven dependency ordering for branch
+1 💚 mvninstall 23m 49s branch-3.3 passed
+1 💚 compile 18m 8s branch-3.3 passed
+1 💚 checkstyle 3m 20s branch-3.3 passed
+1 💚 mvnsite 20m 21s branch-3.3 passed
+1 💚 javadoc 8m 8s branch-3.3 passed
+1 💚 spotbugs 35m 15s branch-3.3 passed
+1 💚 shadedclient 55m 37s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 36s Maven dependency ordering for patch
+1 💚 mvninstall 22m 39s the patch passed
+1 💚 compile 17m 33s the patch passed
+1 💚 javac 17m 33s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 3m 12s the patch passed
+1 💚 mvnsite 20m 0s the patch passed
+1 💚 javadoc 8m 23s the patch passed
+1 💚 spotbugs 35m 44s the patch passed
+1 💚 shadedclient 55m 48s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 660m 38s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 2m 42s The patch does not generate ASF License warnings.
947m 46s
Reason Tests
Failed junit tests hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
hadoop.yarn.server.resourcemanager.TestRMHA
hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.TestClientRMService
hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
hadoop.yarn.client.TestGetGroups
hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
hadoop.crypto.key.kms.server.TestKMS
hadoop.crypto.key.kms.server.TestKMSWithZK
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.mapred.TestLocalDistributedCacheManager
hadoop.yarn.sls.appmaster.TestAMSimulator
hadoop.yarn.sls.TestSLSRunner
hadoop.yarn.sls.TestSLSDagAMSimulator
hadoop.yarn.sls.TestReservationSystemInvariants
hadoop.yarn.sls.TestSLSStreamAMSynth
hadoop.yarn.sls.TestSLSGenericSynth
hadoop.yarn.sls.nodemanager.TestNMSimulator
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/2/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense codespell compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname Linux 81ea8277c7bc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / d474aa1ef07bf02f7e7405a3dd7676762890bfd8
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/2/testReport/
Max. process+thread count 3091 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented May 4, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 28s Maven dependency ordering for branch
+1 💚 mvninstall 23m 59s branch-3.3 passed
+1 💚 compile 17m 59s branch-3.3 passed
+1 💚 checkstyle 3m 19s branch-3.3 passed
+1 💚 mvnsite 20m 20s branch-3.3 passed
+1 💚 javadoc 8m 0s branch-3.3 passed
+1 💚 spotbugs 35m 23s branch-3.3 passed
+1 💚 shadedclient 55m 50s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 22m 36s the patch passed
+1 💚 compile 17m 28s the patch passed
+1 💚 javac 17m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 3m 11s the patch passed
+1 💚 mvnsite 20m 9s the patch passed
+1 💚 javadoc 7m 57s the patch passed
+1 💚 spotbugs 35m 51s the patch passed
+1 💚 shadedclient 55m 52s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 662m 48s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 2m 40s The patch does not generate ASF License warnings.
956m 39s
Reason Tests
Failed junit tests hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
hadoop.yarn.server.resourcemanager.TestRMHA
hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
hadoop.yarn.server.resourcemanager.TestClientRMService
hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
hadoop.yarn.client.TestGetGroups
hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl
hadoop.crypto.key.kms.server.TestKMS
hadoop.crypto.key.kms.server.TestKMSWithZK
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.mapred.TestLocalDistributedCacheManager
hadoop.yarn.sls.appmaster.TestAMSimulator
hadoop.yarn.sls.TestSLSRunner
hadoop.yarn.sls.TestSLSDagAMSimulator
hadoop.yarn.sls.TestReservationSystemInvariants
hadoop.yarn.sls.TestSLSStreamAMSynth
hadoop.yarn.sls.TestSLSGenericSynth
hadoop.yarn.sls.nodemanager.TestNMSimulator
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/4/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense codespell compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname Linux 68a021bfcf89 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / d474aa1ef07bf02f7e7405a3dd7676762890bfd8
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/4/testReport/
Max. process+thread count 3265 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/4/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented May 5, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 15m 30s Maven dependency ordering for branch
+1 💚 mvninstall 23m 51s branch-3.3 passed
+1 💚 compile 18m 4s branch-3.3 passed
+1 💚 checkstyle 3m 21s branch-3.3 passed
+1 💚 mvnsite 20m 22s branch-3.3 passed
+1 💚 javadoc 8m 34s branch-3.3 passed
+1 💚 spotbugs 35m 24s branch-3.3 passed
+1 💚 shadedclient 55m 48s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 22m 54s the patch passed
+1 💚 compile 17m 42s the patch passed
+1 💚 javac 17m 42s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 3m 8s the patch passed
+1 💚 mvnsite 19m 55s the patch passed
+1 💚 javadoc 7m 50s the patch passed
+1 💚 spotbugs 35m 55s the patch passed
+1 💚 shadedclient 56m 14s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 308m 42s /patch-unit-root.txt root in the patch failed.
+0 🆗 asflicense 1m 11s ASF License check generated no output?
596m 16s
Reason Tests
Failed junit tests hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
hadoop.yarn.server.resourcemanager.TestResourceTrackerService
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1
hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer
hadoop.crypto.key.kms.server.TestKMS
hadoop.crypto.key.kms.server.TestKMSWithZK
hadoop.hdfs.server.federation.router.TestRouterRpc
hadoop.hdfs.server.federation.security.TestRouterSecurityManager
hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/5/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense codespell compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname Linux 69ddfeecc933 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 0f8ad00d4c5a1d15bc4bbe1be7b8a93163053362
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/5/testReport/
Max. process+thread count 3315 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/5/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 6, 2022

Looking at these test failures:

  • on a full run -- three of the test runs -- I see ~100-120 failures.
  • looking at the hdfs server, hdfs router, and mapreduce, failures, the tests fail for me locally whether the patch is in place or not except for one test, see the last bullet
  • I ignore the yarn failures ruling them as unrelated to this minor change in an aspect of NameNode node tracking.
  • The hdfs test TestBPOfferService failed in two of the three runs above. It passes locally. Flakey?

Let me try pushing a patch w/ no functional change to see what list of failures I get.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from 0f8ad00 to e9049bd Compare May 10, 2022 04:40
@hadoop-yetus
Copy link

hadoop-yetus commented May 10, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 34s branch-3.3 passed
+1 💚 compile 1m 32s branch-3.3 passed
+1 💚 checkstyle 1m 12s branch-3.3 passed
+1 💚 mvnsite 1m 40s branch-3.3 passed
+1 💚 javadoc 1m 58s branch-3.3 passed
+1 💚 spotbugs 3m 32s branch-3.3 passed
+1 💚 shadedclient 26m 34s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 19s the patch passed
+1 💚 compile 1m 13s the patch passed
+1 💚 javac 1m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 49s the patch passed
+1 💚 mvnsite 1m 21s the patch passed
+1 💚 javadoc 1m 34s the patch passed
+1 💚 spotbugs 3m 17s the patch passed
+1 💚 shadedclient 26m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 189m 1s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 9s The patch does not generate ASF License warnings.
303m 48s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
hadoop.hdfs.server.datanode.TestRefreshNamenodes
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.datanode.TestDataNodeReconfiguration
hadoop.hdfs.server.datanode.TestTransferRbw
hadoop.hdfs.server.datanode.TestBatchIbr
hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
hadoop.hdfs.qjournal.server.TestJournalNodeSync
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/6/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux fabca6d9bc43 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / e9049bdf18516093b053cd83bd2374262cb2519c
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/6/testReport/
Max. process+thread count 3227 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/6/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 10, 2022

Node seems to be having issues...

java.lang.OutOfMemoryError: unable to create new native thread

Let me try a new push.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from e9049bd to 306eb36 Compare May 10, 2022 15:18
@hadoop-yetus
Copy link

hadoop-yetus commented May 10, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 30s branch-3.3 passed
+1 💚 compile 1m 32s branch-3.3 passed
+1 💚 checkstyle 1m 13s branch-3.3 passed
+1 💚 mvnsite 1m 40s branch-3.3 passed
+1 💚 javadoc 1m 51s branch-3.3 passed
+1 💚 spotbugs 3m 36s branch-3.3 passed
+1 💚 shadedclient 27m 14s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 22s the patch passed
+1 💚 compile 1m 15s the patch passed
+1 💚 javac 1m 15s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 48s the patch passed
+1 💚 mvnsite 1m 21s the patch passed
+1 💚 javadoc 1m 31s the patch passed
+1 💚 spotbugs 3m 16s the patch passed
+1 💚 shadedclient 25m 57s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 190m 14s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 12s The patch does not generate ASF License warnings.
299m 0s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/7/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 3b44c5c05aa1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 306eb361f6831f1d7722049133e89a2184b47cd1
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/7/testReport/
Max. process+thread count 3175 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/7/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from 306eb36 to 2966e53 Compare May 10, 2022 20:47
@hadoop-yetus
Copy link

hadoop-yetus commented May 11, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 23s branch-3.3 passed
+1 💚 compile 1m 32s branch-3.3 passed
+1 💚 checkstyle 1m 12s branch-3.3 passed
+1 💚 mvnsite 1m 37s branch-3.3 passed
+1 💚 javadoc 1m 56s branch-3.3 passed
+1 💚 spotbugs 3m 32s branch-3.3 passed
+1 💚 shadedclient 27m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 22s the patch passed
+1 💚 compile 1m 12s the patch passed
+1 💚 javac 1m 12s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 49s the patch passed
+1 💚 mvnsite 1m 22s the patch passed
+1 💚 javadoc 1m 32s the patch passed
+1 💚 spotbugs 3m 15s the patch passed
+1 💚 shadedclient 26m 15s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 190m 33s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 13s The patch does not generate ASF License warnings.
299m 15s
Reason Tests
Failed junit tests hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.TestDFSUpgradeFromImage
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/8/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 90a26f48a12b 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 2966e5350569cbcc88f66382d19537201deff0e7
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/8/testReport/
Max. process+thread count 3708 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/8/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 11, 2022

A noop patch has these failures:

    org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromRel1BBWImage
    org.apache.hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor.testDecommissionWithMissingBlock
    org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade.testWithLayoutChangeAndFinalize

A push with the backport has these failures:

    org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks
    org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade.testWithLayoutChangeAndRollback

One overlap: TestDataNodeRollingUpgrade. Other seems unrelated. Let me try a repush.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from 2966e53 to 53fdbf6 Compare May 11, 2022 16:50
@hadoop-yetus
Copy link

hadoop-yetus commented May 11, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 27s branch-3.3 passed
+1 💚 compile 1m 33s branch-3.3 passed
+1 💚 checkstyle 1m 13s branch-3.3 passed
+1 💚 mvnsite 1m 45s branch-3.3 passed
+1 💚 javadoc 1m 52s branch-3.3 passed
+1 💚 spotbugs 3m 41s branch-3.3 passed
+1 💚 shadedclient 27m 12s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 22s the patch passed
+1 💚 compile 1m 17s the patch passed
+1 💚 javac 1m 17s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 0m 45s the patch passed
+1 💚 mvnsite 1m 17s the patch passed
+1 💚 javadoc 1m 29s the patch passed
+1 💚 spotbugs 3m 59s the patch passed
+1 💚 shadedclient 33m 9s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 34m 55s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+0 🆗 asflicense 0m 41s ASF License check generated no output?
150m 13s
Reason Tests
Failed junit tests hadoop.hdfs.tools.TestECAdmin
hadoop.hdfs.tools.TestViewFileSystemOverloadSchemeWithDFSAdmin
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
hadoop.cli.TestHDFSCLI
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
hadoop.hdfs.TestBlockStoragePolicy
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/9/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 80c88472574d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 53fdbf60a26f18341150743f860b0713ca2d632a
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/9/testReport/
Max. process+thread count 1567 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/9/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented May 12, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 45s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
-1 ❌ mvninstall 6m 15s /branch-mvninstall-root.txt root in branch-3.3 failed.
-1 ❌ compile 0m 30s /branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in branch-3.3 failed.
-0 ⚠️ checkstyle 0m 27s /buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt The patch fails to run checkstyle in hadoop-hdfs
+1 💚 mvnsite 3m 14s branch-3.3 passed
+1 💚 javadoc 1m 55s branch-3.3 passed
+1 💚 spotbugs 4m 0s branch-3.3 passed
+1 💚 shadedclient 29m 31s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 21s the patch passed
+1 💚 compile 1m 14s the patch passed
-1 ❌ javac 1m 14s /results-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project_hadoop-hdfs generated 567 new + 0 unchanged - 0 fixed = 567 total (was 0)
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 55s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 52 new + 0 unchanged - 0 fixed = 52 total (was 0)
+1 💚 mvnsite 1m 22s the patch passed
+1 💚 javadoc 1m 27s the patch passed
+1 💚 spotbugs 3m 15s the patch passed
+1 💚 shadedclient 26m 0s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 189m 51s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
270m 37s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/10/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 691ecf7100cf 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / eb904f3adaa55d44aa6494ad116344317e9ec882
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/10/testReport/
Max. process+thread count 3443 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/10/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 12, 2022

Removing a space had us run more tests and 4 tests failed instead of 44 on previous run.

Below is the change in last run.

From eb904f3adaa55d44aa6494ad116344317e9ec882 Mon Sep 17 00:00:00 2001
From: stack <[email protected]>
Date: Wed, 11 May 2022 15:42:18 -0700
Subject: [PATCH] Remove a space at end of line inside a comment -- non-change

---
 .../hadoop/hdfs/server/blockmanagement/DatanodeManager.java     | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
index 4f74fe0c7364..1937e5d7acc3 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
@@ -1195,7 +1195,7 @@ public void registerDatanode(DatanodeRegistration nodeReg)
         } else {
           // nodeS is found
           /*
-            The registering datanode is a replacement node for the existing
+            The registering datanode is a replacement node for the existing
             data storage, which from now on will be served by a new node.
             If this message repeats, both nodes might have same storageID
             by (insanely rare) random chance. User needs to restart one of the

The tests that failed on this run:

org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.testRead
org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas
org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend.testMultipleAppendsDuringCatchupTailing

do not overlap at all with the failures in the previous run.

I see the branch-mvninstall-root failed with "unable to create new native thread"

Let me try one more run shifting a space and see what I get.

I don't think this patch has any relation to the failures I'm seeing. I'll merge unless I see an obvious relation in the next run.

@hadoop-yetus
Copy link

hadoop-yetus commented May 12, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 36m 42s branch-3.3 passed
+1 💚 compile 1m 33s branch-3.3 passed
+1 💚 checkstyle 1m 13s branch-3.3 passed
+1 💚 mvnsite 1m 40s branch-3.3 passed
+1 💚 javadoc 1m 58s branch-3.3 passed
+1 💚 spotbugs 3m 39s branch-3.3 passed
+1 💚 shadedclient 27m 1s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 1m 16s the patch passed
+1 💚 javac 1m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 50s the patch passed
+1 💚 mvnsite 1m 23s the patch passed
+1 💚 javadoc 1m 26s the patch passed
+1 💚 spotbugs 3m 19s the patch passed
+1 💚 shadedclient 26m 13s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 193m 17s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 15s The patch does not generate ASF License warnings.
302m 10s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/11/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 14ee2742708c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 53773ea019ca5ed793d36035c7adbfe589f5926c
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/11/testReport/
Max. process+thread count 2983 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/11/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 13, 2022

Two failures:

TestBPOfferService.testMissBlocksWhenReregister
TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks

They come up often enough. Let me try again. Meantime running locally.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from 53773ea to 007c9e8 Compare May 13, 2022 18:13
@hadoop-yetus
Copy link

hadoop-yetus commented May 13, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 35m 26s branch-3.3 passed
+1 💚 compile 1m 31s branch-3.3 passed
+1 💚 checkstyle 1m 13s branch-3.3 passed
+1 💚 mvnsite 1m 41s branch-3.3 passed
+1 💚 javadoc 1m 53s branch-3.3 passed
+1 💚 spotbugs 3m 36s branch-3.3 passed
+1 💚 shadedclient 27m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 21s the patch passed
+1 💚 compile 1m 16s the patch passed
+1 💚 javac 1m 16s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 0m 46s the patch passed
+1 💚 mvnsite 1m 21s the patch passed
+1 💚 javadoc 1m 28s the patch passed
+1 💚 spotbugs 3m 18s the patch passed
+1 💚 shadedclient 26m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 190m 47s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 15s The patch does not generate ASF License warnings.
299m 1s
Reason Tests
Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.namenode.ha.TestHAAppend
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/12/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux bdf2fce93bdd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 007c9e844ffed2e12691b6774fd738688e7c1c06
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/12/testReport/
Max. process+thread count 3333 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/12/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented May 14, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 37m 26s branch-3.3 passed
+1 💚 compile 1m 31s branch-3.3 passed
+1 💚 checkstyle 1m 13s branch-3.3 passed
+1 💚 mvnsite 1m 40s branch-3.3 passed
+1 💚 javadoc 1m 57s branch-3.3 passed
+1 💚 spotbugs 3m 38s branch-3.3 passed
+1 💚 shadedclient 26m 54s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 23s the patch passed
+1 💚 compile 1m 17s the patch passed
+1 💚 javac 1m 17s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 49s the patch passed
+1 💚 mvnsite 1m 24s the patch passed
+1 💚 javadoc 1m 30s the patch passed
+1 💚 spotbugs 3m 22s the patch passed
+1 💚 shadedclient 26m 23s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 190m 22s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
300m 22s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/13/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 60cd37572baf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 21686a25ac0fc811c894b03354592b811e84b1eb
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/13/testReport/
Max. process+thread count 3589 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/13/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack saintstack force-pushed the branch-3.3-HDFS-16540.backport branch from 21686a2 to a509522 Compare May 15, 2022 21:31
@hadoop-yetus
Copy link

hadoop-yetus commented May 16, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 35m 58s branch-3.3 passed
+1 💚 compile 1m 34s branch-3.3 passed
+1 💚 checkstyle 1m 14s branch-3.3 passed
+1 💚 mvnsite 1m 39s branch-3.3 passed
+1 💚 javadoc 1m 52s branch-3.3 passed
+1 💚 spotbugs 3m 35s branch-3.3 passed
+1 💚 shadedclient 27m 7s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 1m 14s the patch passed
+1 💚 javac 1m 14s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 0m 48s the patch passed
+1 💚 mvnsite 1m 20s the patch passed
+1 💚 javadoc 1m 33s the patch passed
+1 💚 spotbugs 3m 17s the patch passed
+1 💚 shadedclient 26m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 190m 25s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 14s The patch does not generate ASF License warnings.
298m 53s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/14/artifact/out/Dockerfile
GITHUB PR #4246
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 8f80b78e9fdd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / a509522
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/14/testReport/
Max. process+thread count 3224 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4246/14/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 16, 2022

I ran the two test below in loops locally.

TestBPOfferService.testMissBlocksWhenReregister
TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks

The first failed once out of ten cycles both when the patch was in place and when not (jibes w/ what we see here in test runs where sometimes it fails but not always).

TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks shows up consistently but when I run it locally in multiple cycles, it passes whether the patch is applied or not. I see that in the last full branch-3.3 run, back on May 5th (https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/54/), it failed for same reason (the May 12th run was incomplete). This test is about block replication where the PR here is about a minor adjustment in NN node accounting. Unrelated I'd say. Pushing the backport.

Will push in the morning.

@saintstack saintstack merged commit 1d22687 into apache:branch-3.3 May 16, 2022
@huaxiangsun
Copy link
Contributor

huaxiangsun commented May 17, 2022

Thanks a lot, @saintstack!

@Hexiaoqiao
Copy link
Contributor

Hexiaoqiao commented May 18, 2022

I don't get what changes here. Maybe I missed something? Thanks.

@d-c-manning
Copy link

d-c-manning commented Sep 20, 2023

I think the wrong change was accidentally committed - also made the comment on the JIRA just now: https://issues.apache.org/jira/browse/HDFS-16540?focusedCommentId=17767163

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@saintstack @hadoop-yetus @huaxiangsun @Hexiaoqiao @d-c-manning