HADOOP-17409. Remove s3guard from S3A module (#3534)

[steveloughran]

Completely removes S3Guard support from the S3A codebase.

If the connector is configured to use any metastore other than
the null and local stores (i.e. DynamoDB is selected) the s3a client
will raise an exception and refuse to initialize.

This is to ensure that there is no mix of S3Guard enabled and disabled
deployments with the same configuration but different hadoop releases
-it must be turned off completely.

The "hadoop s3guard" command has been retained -but the supported
subcommands have been reduced to those which are not purely S3Guard
related: "bucket-info" and "uploads".

This is major change in terms of the number of files
changed; before cherry picking subsequent s3a patches into
older releases, this patch will probably need backporting
first.

Goodbye S3Guard, your work is done. Time to die.

Contributed by Steve Loughran.

[steveloughran]

this is #3534 for branch 3.3
doesn't compile yet. I want to cherrypick the lists change of #3072 first, even though now I've dealt with a lot of merge conflict here on deleted files, it'll make rebasing this needlessly hard

[steveloughran]

tested s3 london, -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=delete -Dscale

all happy

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 4s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 110 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	3m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 16s		branch-3.3 passed
+1 💚	compile	20m 21s		branch-3.3 passed
+1 💚	checkstyle	2m 59s		branch-3.3 passed
+1 💚	mvnsite	3m 49s		branch-3.3 passed
+1 💚	javadoc	3m 18s		branch-3.3 passed
+0 🆗	spotbugs	0m 31s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	30m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 46s		the patch passed
+1 💚	compile	23m 14s		the patch passed
+1 💚	javac	23m 14s		root generated 0 new + 1921 unchanged - 50 fixed = 1921 total (was 1971)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 26s	/results-checkstyle-root.txt	root: The patch generated 1 new + 72 unchanged - 83 fixed = 73 total (was 155)
+1 💚	mvnsite	4m 23s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 7s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 29s		hadoop-project in the patch passed.
+1 💚	javadoc	1m 51s		hadoop-common in the patch passed.
+1 💚	javadoc	0m 43s		hadoop-tools_hadoop-aws generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63)
+1 💚	javadoc	0m 38s		hadoop-azure in the patch passed.
+0 🆗	spotbugs	0m 31s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	30m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 29s		hadoop-project in the patch passed.
+1 💚	unit	20m 22s		hadoop-common in the patch passed.
+1 💚	unit	2m 20s		hadoop-aws in the patch passed.
+1 💚	unit	2m 32s		hadoop-azure in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		218m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/2/artifact/out/Dockerfile
GITHUB PR	#3897
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs
uname	Linux e5d846a5058e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 075edbe9bcf36cb36f34b222b4fcf6a301e5c892
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/2/testReport/
Max. process+thread count	1252 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	10m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 5s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 110 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	3m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 30s		branch-3.3 passed
+1 💚	compile	17m 33s		branch-3.3 passed
+1 💚	checkstyle	2m 45s		branch-3.3 passed
+1 💚	mvnsite	4m 6s		branch-3.3 passed
+1 💚	javadoc	3m 58s		branch-3.3 passed
+0 🆗	spotbugs	0m 41s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	24m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 17s		the patch passed
+1 💚	compile	16m 54s		the patch passed
+1 💚	javac	16m 54s		root generated 0 new + 1921 unchanged - 50 fixed = 1921 total (was 1971)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 36s		root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155)
+1 💚	mvnsite	4m 3s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 5s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 37s		hadoop-project in the patch passed.
+1 💚	javadoc	1m 40s		hadoop-common in the patch passed.
+1 💚	javadoc	0m 46s		hadoop-tools_hadoop-aws generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63)
+1 💚	javadoc	0m 44s		hadoop-azure in the patch passed.
+0 🆗	spotbugs	0m 37s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	24m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 37s		hadoop-project in the patch passed.
+1 💚	unit	19m 35s		hadoop-common in the patch passed.
+1 💚	unit	2m 24s		hadoop-aws in the patch passed.
+1 💚	unit	2m 24s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		190m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/3/artifact/out/Dockerfile
GITHUB PR	#3897
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs
uname	Linux f84d65c15cb0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / c4a3883500e48fe594f48dc6ae1d32b7943a3c17
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/3/testReport/
Max. process+thread count	1265 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/3/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

ok, yetus happy. will merge

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 4s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 110 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	3m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 19s		branch-3.3 passed
+1 💚	compile	18m 17s		branch-3.3 passed
+1 💚	checkstyle	2m 53s		branch-3.3 passed
+1 💚	mvnsite	3m 34s		branch-3.3 passed
+1 💚	javadoc	3m 16s		branch-3.3 passed
+0 🆗	spotbugs	0m 30s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	25m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 13s		the patch passed
+1 💚	compile	17m 31s		the patch passed
-1 ❌	javac	17m 31s	/results-compile-javac-root.txt	root generated 1 new + 1924 unchanged - 50 fixed = 1925 total (was 1974)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 46s		root: The patch generated 0 new + 72 unchanged - 83 fixed = 72 total (was 155)
+1 💚	mvnsite	3m 28s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 5s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 28s		hadoop-project in the patch passed.
+1 💚	javadoc	1m 34s		hadoop-common in the patch passed.
+1 💚	javadoc	0m 36s		hadoop-tools_hadoop-aws generated 0 new + 40 unchanged - 23 fixed = 40 total (was 63)
+1 💚	javadoc	0m 34s		hadoop-azure in the patch passed.
+0 🆗	spotbugs	0m 28s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	25m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 26s		hadoop-project in the patch passed.
+1 💚	unit	18m 42s		hadoop-common in the patch passed.
+1 💚	unit	2m 5s		hadoop-aws in the patch passed.
+1 💚	unit	2m 10s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		182m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/4/artifact/out/Dockerfile
GITHUB PR	#3897
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint spotbugs checkstyle shellcheck shelldocs
uname	Linux 2fbbb28e1b23 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / cdb6aac
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/4/testReport/
Max. process+thread count	1975 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3897/4/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.