Skip to content

HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO. #1918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 31, 2020
Merged

HADOOP-16910: ABFS Streams to update FileSystem.Statistics counters on IO. #1918

merged 1 commit into from
Mar 31, 2020

Conversation

@mehakmeet
Copy link
Contributor

Contributed by Mehakmeet Singh

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@mehakmeet
Copy link
Contributor Author

test run : mvn -T 1C -Dparallel-tests=abfs clean verify
Region : East US, West US

related to previous PR #1881

@mehakmeet
Copy link
Contributor Author

Full test results(error is container based) :

[INFO] Results:
[INFO]
[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestGetNameSpaceEnabled.testFailedRequestWhenCredentialsNotCorrect:91->AbstractAbfsIntegrationTest.getFileSystem:229 » InvalidConfigurationValue
[INFO]
[ERROR] Tests run: 416, Failures: 0, Errors: 1, Skipped: 62
[INFO] Results:
[INFO]
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29

@mukund-thakur
Copy link
Contributor

Amend the commit and put jira Id.

@bgaborg
Copy link

bgaborg commented Mar 31, 2020

+1 from @steveloughran on #1881, and the compile error is fixed now.
Merging.

@bgaborg bgaborg merged commit c734d24 into apache:trunk Mar 31, 2020
@steveloughran
Copy link
Contributor

thanks

asfgit pushed a commit that referenced this pull request Apr 22, 2020
@mehakmeet @steveloughran
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters …
f74a571 
…on IO. (#1918). Contributed by Mehakmeet Singh.
@amarnathkarthik
Copy link
Contributor

@mehakmeet Integration test fails when running against ABS East US.

Command: mvn -T 1C -Dparallel-tests=abfs clean verify

[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0

[ERROR] Failures: 
[ERROR]   ITestAbfsStreamStatistics.testAbfsStreamOps:92->assertReadWriteOps:154->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Mismatch in read operations expected:<2> but was:<3>
**[ERROR] Tests run: 424, Failures: 1, Errors: 0, Skipped: 267**

[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 29

CC: @steveloughran

@mehakmeet
Copy link
Contributor Author

@amarnathkarthik it fails when you run it alone too ?

mvn -T 1C -Dparallel-tests=abfs clean verify -Dtest=none -Dit.test=ITestAbfsStreamStatistics

Never failed for me, Either way I'll look into it. Thank you.

@apache apache deleted a comment from hadoop-yetus Apr 28, 2020
@apache apache deleted a comment from hadoop-yetus Apr 28, 2020
@apache apache deleted a comment from hadoop-yetus Apr 28, 2020
@steveloughran
Copy link
Contributor

steveloughran commented Apr 28, 2020 •
edited
Loading

this sounds intermittent or configuration related.

Does it show up in standalone runs or only in parallel ones?

@mehakmeet
Copy link
Contributor Author

stack trace for the error:

2020-04-28 15:49:04,106 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - GetAcl
2020-04-28 15:49:04,151 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:08,430 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 404,,cid=1a653b2f-8796-4c5d-ab72-f04bb1296d08,rid=8026b894-a01f-005a-6846-1d2ad8000000,sent=0,recv=0,HEAD,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12//?upn=false&action=getAccessControl&timeout=90
2020-04-28 15:49:08,446 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - CreateFileSystem
2020-04-28 15:49:08,446 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:08,752 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 201,,cid=494b1081-ddfc-4f7f-8b58-ff93ef1e1813,rid=8026b896-a01f-005a-6a46-1d2ad8000000,sent=0,recv=0,PUT,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12?resource=filesystem
2020-04-28 15:49:08,774 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - GetAcl
2020-04-28 15:49:08,775 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:09,019 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 200,,cid=8b51c94e-53d6-4bf9-95ae-a220452926d3,rid=8026b8a9-a01f-005a-7d46-1d2ad8000000,sent=0,recv=0,HEAD,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12//?upn=false&action=getAccessControl&timeout=90
2020-04-28 15:49:09,020 INFO  [JUnit-testAbfsStreamOps]: azurebfs.AbstractAbfsTestWithTimeout (AbstractAbfsTestWithTimeout.java:describe(88)) - 

testAbfsStreamOps: Test to see correct population of read and write operations in Abfs

2020-04-28 15:49:09,035 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - CreatePath
2020-04-28 15:49:09,035 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:09,300 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 201,,cid=eb705c96-722d-4775-9100-161c64f11145,rid=8026b8be-a01f-005a-1246-1d2ad8000000,sent=0,recv=0,PUT,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?resource=file&timeout=90
2020-04-28 15:49:09,319 DEBUG [pool-2-thread-1]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - Append
2020-04-28 15:49:09,319 DEBUG [pool-2-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:09,565 DEBUG [pool-2-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 202,,cid=b7680b95-e3d0-42dd-9a2f-2fe33652d4ba,rid=8026b8c8-a01f-005a-1c46-1d2ad8000000,sent=9,recv=0,PUT,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?action=append&position=0&timeout=90
2020-04-28 15:49:09,571 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - Flush
2020-04-28 15:49:09,571 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:09,825 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 200,,cid=e8dad7d3-4f4a-4b87-90a4-81ced45fa71b,rid=8026b8d9-a01f-005a-2c46-1d2ad8000000,sent=0,recv=0,PUT,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?action=flush&retainUncommittedData=false&position=9&close=false&timeout=90
2020-04-28 15:49:09,826 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - GetPathStatus
2020-04-28 15:49:09,827 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:10,573 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 200,,cid=f0b7fda3-ae2a-4a74-811c-4d7647f773a6,rid=8026b8e9-a01f-005a-3c46-1d2ad8000000,sent=0,recv=0,HEAD,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?upn=false&timeout=90
2020-04-28 15:49:10,644 DEBUG [ABFS-prefetch-4]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - ReadFile
2020-04-28 15:49:10,646 DEBUG [ABFS-prefetch-4]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:11,389 DEBUG [ABFS-prefetch-4]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 206,,cid=fe0d1b0b-3f46-41fd-8611-5f8fa7c75564,rid=8026b8f0-a01f-005a-4346-1d2ad8000000,sent=0,recv=9,GET,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?timeout=90
2020-04-28 15:49:11,389 INFO  [JUnit-testAbfsStreamOps]: azurebfs.ITestAbfsStreamStatistics (ITestAbfsStreamStatistics.java:testAbfsStreamOps(85)) - Result of Read operation : 9
2020-04-28 15:49:11,389 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - Flush
2020-04-28 15:49:11,390 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:12,132 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 200,,cid=cb07f1b1-6067-4547-8fce-e5b18d8c8ef7,rid=8026b8fc-a01f-005a-4f46-1d2ad8000000,sent=0,recv=0,PUT,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12/user/mehakmeet.singh/testOneReadWriteOps?action=flush&retainUncommittedData=false&position=9&close=true&timeout=90
2020-04-28 15:49:12,134 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - DeleteFileSystem
2020-04-28 15:49:12,134 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:13,179 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 202,,cid=f8fb143a-f007-4d1e-9fe1-b74ed981bbb8,rid=8026b948-a01f-005a-1b46-1d2ad8000000,sent=0,recv=0,DELETE,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12?resource=filesystem&timeout=90
2020-04-28 15:49:13,196 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:execute(133)) - First execution of REST operation - GetFileSystemProperties
2020-04-28 15:49:13,197 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(175)) - Signing request with shared key
2020-04-28 15:49:13,439 DEBUG [JUnit-testAbfsStreamOps]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(223)) - HttpRequest: 404,,cid=0b9689d7-970f-4817-b153-54d906ee5d64,rid=8026b95c-a01f-005a-2f46-1d2ad8000000,sent=0,recv=0,HEAD,https://mehakmeetdata.dfs.core.windows.net/abfs-testcontainer-ce1153dd-6c16-402b-91c8-099051333a12?resource=filesystem&timeout=90

java.lang.AssertionError: Mismatch in read operations 
Expected :2
Actual   :3
 <Click to see difference>


	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics.assertReadWriteOps(ITestAbfsStreamStatistics.java:154)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics.testAbfsStreamOps(ITestAbfsStreamStatistics.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

http 404 error seems interesting here.

@amarnathkarthik
Copy link
Contributor

Standalone and parallel fails with same error. Also i'd try to run at least multiple (10) times and failed every single time.

mvn -T 1C -Dparallel-tests=abfs clean verify -Dtest=none -Dit.test=ITestAbfsStreamStatistics

Stack trace:

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.261 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[ERROR] testAbfsStreamOps(org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics)  Time elapsed: 2.003 s  <<< FAILURE!
java.lang.AssertionError: Mismatch in read operations expected:<2> but was:<3>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics.assertReadWriteOps(ITestAbfsStreamStatistics.java:154)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics.testAbfsStreamOps(ITestAbfsStreamStatistics.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:745)

@mehakmeet
Copy link
Contributor Author

Ok, that is not the case with my configurations maybe. I am getting random fails and the test failed like 1/20 times for me.
It doesn't break consecutively this much for me.
Test result for a test I ran just now:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 469.869 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]

I have opened up this Jira regarding this issue.
https://issues.apache.org/jira/browse/HADOOP-17018

I'll try to get the PR up for this issue and get it merged as soon as possible. Thank you.

@mehakmeet
Copy link
Contributor Author

@amarnathkarthik Can you tell me if you've changed any specific configurations and then run the tests or maybe share what you are using in your auth-keys.xml to run the tests?
Just want to pin-point why all your tests are failing because they are not failing for me.

@mehakmeet
Copy link
Contributor Author

PR #1990 has been opened regarding this and contains the reasoning of intermittent test failures.

@amarnathkarthik
Copy link
Contributor

@amarnathkarthik Can you tell me if you've changed any specific configurations and then run the tests or maybe share what you are using in your auth-keys.xml to run the tests?
Just want to pin-point why all your tests are failing because they are not failing for me.

@mehakmeet There is no custom configuration change on storage side, i'd run with both with hierarchical namespace enabled and disabled. Let me know if you need further details.

snvijaya pushed a commit to snvijaya/hadoop that referenced this pull request May 23, 2021
@mehakmeet @snvijaya
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters …
f8306a8 
…on IO. (apache#1918). Contributed by Mehakmeet Singh.
arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 11, 2022
@mehakmeet @arjun4084346
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters …
ce76aa5 
…on IO. (apache#1918). Contributed by Mehakmeet Singh.

(cherry picked from commit c734d24)
zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022
@mehakmeet @zhangxiping1
HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters …
2f363a2 
…on IO. (apache#1918). Contributed by Mehakmeet Singh.
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
@steveloughran
CDPD-8485. HADOOP-16910 : ABFS Streams to update FileSystem.Statistic…
9c457ee 
…s counters on IO. (apache#1918).

Contributed by Mehakmeet Singh.

Change-Id: If7d82e40427a3ccc1418b935cd4574dee262b37b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mehakmeet @mukund-thakur @bgaborg @steveloughran @amarnathkarthik