HADOOP-1320. Dir Marker getFileStatus() changes backport

[steveloughran]

This contains the getFileStatus changes of #2149

This does not change dir marker deletion, but it

• Changes s3GetFileStatus to do HEAD + LIST only
• add in StatusProbeEnum from the 404 work
• Fix relevant tests to work -tweaking the FileOperationsCost rather than
  the big move to the new cost test framework

These changes can actually go in ahead of the dir marker patch as they should marginally speed up directory listings everywhere.

A variant of this patch will be needed for 3.0+ branches; this will be the basis for those.

( I should probably have a separate JIRA ID...)

[steveloughran]

failure on unguarded run

[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 36.712 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[ERROR] testCostOfGetFileStatusOnNonEmptyDir(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost)  Time elapsed: 5.415 s  <<< FAILURE!
java.lang.AssertionError: Count of object_metadata_requests starting=5 current=6 diff=1: object_metadata_requests expected:<2> but was:<1>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:691)
	at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:700)
	at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir(ITestS3AFileOperationCost.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at 

[steveloughran]

[ERROR] testCreateSubdirWithDifferentKey(org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC)  Time elapsed: 1.54 s  <<< FAILURE!
java.lang.AssertionError: Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : S3AFileStatus{path=s3a://stevel-ireland/test/testCreateSubdirWithDifferentKey/nestedDir; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:492)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:377)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:446)
	at org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC.testCreateSubdirWithDifferentKey(ITestS3AEncryptionSSEC.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 4.045 s - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestS3AEncryptionSSEC.testCreateSubdirWithDifferentKey:124 Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : S3AFileStatus{path=s3a://stevel-ireland/test/testCreateSubdirWithDifferentKey/nestedDir; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE
[ERROR]   ITestS3AEncryptionSSEC.testListEncryptedDir:208 Expecting java.nio.file.AccessDeniedException with text Service: Amazon S3; Status Code: 403; but got : void
[ERROR]   ITestS3AEncryptionSSEC.testListStatusEncryptedDir:253 Expecting java.nio.file.AccessDeniedException with text Service: Amazon S3; Status Code: 403; but got : void
[INFO] 
[ERROR] Tests run: 80, Failures: 3, Errors: 0, Skipped: 9
[INFO] 

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	20m 6s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ branch-3.2 Compile Tests _
+0 🆗	mvndep	0m 51s	Maven dependency ordering for branch
+1 💚	mvninstall	21m 54s	branch-3.2 passed
+1 💚	compile	18m 32s	branch-3.2 passed
+1 💚	checkstyle	2m 49s	branch-3.2 passed
+1 💚	mvnsite	1m 53s	branch-3.2 passed
+1 💚	shadedclient	19m 53s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 42s	branch-3.2 passed
+0 🆗	spotbugs	1m 2s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 16s	branch-3.2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 31s	the patch passed
+1 💚	compile	16m 16s	the patch passed
+1 💚	javac	16m 16s	the patch passed
-0 ⚠️	checkstyle	2m 46s	root: The patch generated 19 new + 5 unchanged - 0 fixed = 24 total (was 5)
+1 💚	mvnsite	2m 3s	the patch passed
-1 ❌	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	shadedclient	14m 6s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 34s	the patch passed
+1 💚	findbugs	3m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 55s	hadoop-common in the patch passed.
+1 💚	unit	4m 26s	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s	The patch does not generate ASF License warnings.
		147m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2170/2/artifact/out/Dockerfile
GITHUB PR	#2170
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c7ef8f03d436 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	branch-3.2 / 863689f
Default Java	Private Build-1.8.0_252-8u252-b09-1~16.04-b09
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2170/2/artifact/out/diff-checkstyle-root.txt
whitespace	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2170/2/artifact/out/whitespace-eol.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2170/2/testReport/
Max. process+thread count	1378 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2170/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.