HADOOP-18544: S3A: add option to disable probe for dir marker recreation on delete/rename.

[HarshitGupta11]

Description of PR

In applications which do many single-file deletions on the same dir, a lot of time is wasted in maybeCreateFakeParentDirectory().

Proposed: add an option to disable the probe, for use by applications which are happy for parent dirs to sometimes disappear after a cleanup.

file by file delete is still woefully inefficient because of the HEAD request on every file, but there's no need to amplify the damage.

How was this patch tested?

The patch was tested against s3 bucket in US-West 2

For code changes:

Caveats:

Parent directories might disappear on delete or on renames.

What breaks:

The rename tests are failing for the FileContext renames as both S3AFileSystem and the FileContext have different probes and different rules.

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 59s		Maven dependency ordering for branch
+1 💚	mvninstall	30m 53s		trunk passed
+1 💚	compile	23m 1s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 23s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	3m 46s		trunk passed
+1 💚	mvnsite	2m 44s		trunk passed
-1 ❌	javadoc	1m 15s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	3m 58s		trunk passed
+1 💚	shadedclient	23m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 49s		the patch passed
+1 💚	compile	22m 26s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 26s		the patch passed
+1 💚	compile	20m 29s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 37s	/results-checkstyle-root.txt	root: The patch generated 40 new + 49 unchanged - 0 fixed = 89 total (was 49)
+1 💚	mvnsite	2m 38s		the patch passed
-1 ❌	javadoc	1m 6s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 12s		the patch passed
+1 💚	shadedclient	24m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	18m 21s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	unit	2m 50s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		237m 26s

Reason	Tests
Failed junit tests	hadoop.fs.TestFilterFs

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/Dockerfile
GITHUB PR	#5354
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8f26950b449b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0f96e64
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/testReport/
Max. process+thread count	1302 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

harshit, now that #6789 is in, we support an option fs.s3a.performance.flags for a CSV list of flags. You could make this one of them and people can enable it for applications which are ok with the changed semantics