HDFS-15249 ThrottledAsyncChecker is not thread-safe. #1922

brfrn169 · 2020-03-30T07:37:12Z

No description provided.

hadoop-yetus · 2020-03-30T11:42:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	30m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	26m 36s	trunk passed
+1 💚	compile	1m 50s	trunk passed
+1 💚	checkstyle	1m 14s	trunk passed
+1 💚	mvnsite	2m 4s	trunk passed
+1 💚	shadedclient	25m 53s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 2s	trunk passed
+0 🆗	spotbugs	5m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	4m 57s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 51s	the patch passed
+1 💚	compile	1m 45s	the patch passed
+1 💚	javac	1m 45s	the patch passed
-0 ⚠️	checkstyle	1m 10s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	1m 53s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	21m 35s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 53s	the patch passed
+1 💚	findbugs	4m 55s	the patch passed
		_ Other Tests _
-1 ❌	unit	114m 18s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		244m 31s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/1/artifact/out/Dockerfile
GITHUB PR	#1922
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 047e77c030e4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 960c9eb
Default Java	1.8.0_242
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/1/testReport/
Max. process+thread count	3275 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-03-30T15:57:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 22s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	24m 8s	trunk passed
+1 💚	compile	1m 9s	trunk passed
+1 💚	checkstyle	0m 47s	trunk passed
+1 💚	mvnsite	1m 18s	trunk passed
+1 💚	shadedclient	17m 33s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 43s	trunk passed
+0 🆗	spotbugs	3m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 59s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s	the patch passed
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
-0 ⚠️	checkstyle	0m 41s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	1m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 19s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 38s	the patch passed
+1 💚	findbugs	3m 5s	the patch passed
		_ Other Tests _
+1 💚	unit	109m 4s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s	The patch does not generate ASF License warnings.
		183m 55s

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/2/artifact/out/Dockerfile
GITHUB PR	#1922
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4dba9332a7bd 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 960c9eb
Default Java	1.8.0_242
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/2/testReport/
Max. process+thread count	3021 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-03-31T00:59:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 21s	trunk passed
+1 💚	compile	1m 7s	trunk passed
+1 💚	checkstyle	0m 42s	trunk passed
+1 💚	mvnsite	1m 14s	trunk passed
+1 💚	shadedclient	17m 28s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 39s	trunk passed
+0 🆗	spotbugs	3m 0s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 59s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 9s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	checkstyle	0m 38s	the patch passed
+1 💚	mvnsite	1m 6s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 26s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 38s	the patch passed
+1 💚	findbugs	3m 8s	the patch passed
		_ Other Tests _
-1 ❌	unit	113m 8s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		184m 57s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks

Subsystem	Report/Notes
Docker	Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/3/artifact/out/Dockerfile
GITHUB PR	#1922
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 72a1f47c6ca9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `80b877a`
Default Java	1.8.0_242
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/3/testReport/
Max. process+thread count	3260 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1922/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

brfrn169 · 2020-03-31T01:01:48Z

I don't think the test failure in the last QA is related to this patch.

brfrn169 · 2020-03-31T02:42:03Z

Could you please take a look at this when you get a chance? @arp7 @jojochuang

jojochuang · 2020-03-31T04:13:46Z

I am not familiar with this area of code.

Each DataNode object has one DatasetVolumeChecker, one ThrottledAsyncChecker. If it synchronizes on schedule(), the method asynchronously submits a Checkable to run in a thread. So looks correct and it still has parallelism.

aajisaka · 2020-04-07T05:37:47Z

LGTM, +1

(cherry picked from commit c12ddbd)

(cherry picked from commit c12ddbd) (cherry picked from commit e9b06d9) Change-Id: I7ad99401da2edf4b18fa57405f3e754284336239 (cherry picked from commit 84d0a81)

brfrn169 force-pushed the HDFS-15249 branch from 847705b to 78c8a48 Compare March 30, 2020 12:52

HDFS-15249 ThrottledAsyncChecker is not thread-safe.

b8b6fc0

brfrn169 force-pushed the HDFS-15249 branch from 78c8a48 to b8b6fc0 Compare March 30, 2020 21:53

jojochuang approved these changes Mar 31, 2020

View reviewed changes

aajisaka closed this Apr 7, 2020

aajisaka reopened this Apr 7, 2020

aajisaka merged commit c12ddbd into apache:trunk Apr 7, 2020

asfgit pushed a commit that referenced this pull request Mar 24, 2021

HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922)

e9b06d9

(cherry picked from commit c12ddbd)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HDFS-15249 ThrottledAsyncChecker is not thread-safe. #1922

HDFS-15249 ThrottledAsyncChecker is not thread-safe. #1922

Uh oh!

brfrn169 commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 31, 2020

Uh oh!

brfrn169 commented Mar 31, 2020

Uh oh!

brfrn169 commented Mar 31, 2020

Uh oh!

jojochuang commented Mar 31, 2020

Uh oh!

aajisaka commented Apr 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

HDFS-15249 ThrottledAsyncChecker is not thread-safe. #1922

HDFS-15249 ThrottledAsyncChecker is not thread-safe. #1922

Uh oh!

Conversation

brfrn169 commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 30, 2020

Uh oh!

hadoop-yetus commented Mar 31, 2020

Uh oh!

brfrn169 commented Mar 31, 2020

Uh oh!

brfrn169 commented Mar 31, 2020

Uh oh!

jojochuang commented Mar 31, 2020

Uh oh!

aajisaka commented Apr 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants