Hadoop 17132. ABFS: Fix Rename and Delete Idempotency check trigger #2150

snvijaya · 2020-07-17T13:38:20Z

Trigger to call the idempotent code introduced in https://issues.apache.org/jira/browse/HADOOP-17015 is incomplete.

When a request fails at the AbfsRestOperation, an exception is thrown from the REST response handling code . HADOOP-17015 was failing to catch it before checking the AbfsRestOperation instance status code.

The request retry count was also getting set to 1 even when request wasnt re-tired. That has been fixed too.

Tests are added by setting final fields in AbfsClient by reflection and mimicking the idempotency check return values to ensure that the trigger handles all scenarios.

snvijaya · 2020-07-17T13:41:32Z

Test results: From East US2 accounts. With and Without HNS results below:

non-HNS

[INFO] Tests run: 85, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 450, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140

HNS

[INFO] Tests run: 85, Failures: 0, Errors: 0, Skipped: 0

[ERROR] Failures: 
[ERROR]   ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics:207->AbstractAbfsIntegrationTest.assertAbfsStatistics:445->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Mismatch in get_responses expected:<8> but was:<7>
[ERROR]   ITestAbfsNetworkStatistics.testAbfsHttpSendStatistics:91->AbstractAbfsIntegrationTest.assertAbfsStatistics:445->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Mismatch in connections_made expected:<6> but was:<5>
[INFO] 
[ERROR] Tests run: 450, Failures: 2, Errors: 0, Skipped: 75
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 140

The 2 tests from NetworkStatistics is expecting a certain number of GetFileStatus calls before the test issues any request. This is needs to be fixed. Created JIRA to track it : https://issues.apache.org/jira/browse/HADOOP-17137

hadoop-yetus · 2020-07-17T15:20:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	30m 3s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 51s	trunk passed
+1 💚	compile	0m 33s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 29s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 23s	trunk passed
+1 💚	mvnsite	0m 32s	trunk passed
+1 💚	shadedclient	16m 25s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 25s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 22s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	0m 56s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 53s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 28s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 28s	the patch passed
+1 💚	compile	0m 23s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 28s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 24s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 25s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	1m 7s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 28s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		95m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/1/artifact/out/Dockerfile
GITHUB PR	#2150
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 16da032499fd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `2ba44a7`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/1/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

snvijaya · 2020-07-17T15:43:09Z

Javadoc failure because of JDK11 support. Being addressed in https://issues.apache.org/jira/browse/HADOOP-16862

bilaharith

LGTM

steveloughran

This has pretty serious testing...the production code isn't so complex but I'm still going to have to trust you and bilihari's review

LGTM

A couple of minor nits and then it's ready to go in.

steveloughran · 2020-07-20T20:19:32Z

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java

   * HTTP operations.
   */
-  void execute() throws AzureBlobFileSystemException {
+   public void execute() throws AzureBlobFileSystemException {


tag this as @VisibleForTesting too

steveloughran · 2020-07-20T20:20:39Z

...hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java

+        false,
+        null));
+
+    // mock idempotency check to mimick retried case


check spelling of mimic

snvijaya · 2020-07-20T20:50:48Z

Thanks @steveloughran . Have addressed the comments.

hadoop-yetus · 2020-07-20T22:21:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 11s	trunk passed
+1 💚	compile	0m 33s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	0m 29s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 22s	trunk passed
+1 💚	mvnsite	0m 32s	trunk passed
+1 💚	shadedclient	16m 24s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 24s	hadoop-azure in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 22s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	0m 54s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 51s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 27s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	0m 27s	the patch passed
+1 💚	compile	0m 23s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	mvnsite	0m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 31s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 22s	hadoop-azure in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 20s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 13s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		66m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/2/artifact/out/Dockerfile
GITHUB PR	#2150
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 978ee2d7e818 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3833c61`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/2/testReport/
Max. process+thread count	294 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2150/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

snvijaya · 2020-07-21T03:29:49Z

Javadoc failure because of JDK11 support. Being addressed in https://issues.apache.org/jira/browse/HADOOP-16862

DadanielZ

LTGM, +1

snvijaya added 2 commits July 16, 2020 11:56

Fix idempotency pre-check

f1dad3d

Fix for idempotency fix

0d4a593

bilaharith approved these changes Jul 20, 2020

View reviewed changes

steveloughran requested changes Jul 20, 2020

View reviewed changes

Review comments

7bf8fcd

DadanielZ approved these changes Jul 21, 2020

View reviewed changes

DadanielZ merged commit d23cc9d into apache:trunk Jul 21, 2020

Hadoop 17132. ABFS: Fix Rename and Delete Idempotency check trigger #2150

Hadoop 17132. ABFS: Fix Rename and Delete Idempotency check trigger #2150

Uh oh!

Conversation

snvijaya commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snvijaya commented Jul 17, 2020

Uh oh!

hadoop-yetus commented Jul 17, 2020

Uh oh!

snvijaya commented Jul 17, 2020

Uh oh!

bilaharith left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

snvijaya Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

steveloughran Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

snvijaya Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

snvijaya commented Jul 20, 2020

Uh oh!

hadoop-yetus commented Jul 20, 2020

Uh oh!

snvijaya commented Jul 21, 2020

Uh oh!

DadanielZ left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

snvijaya commented Jul 17, 2020 •

edited

Loading