HDFS-14478: Add libhdfs APIs for openFile

[steveloughran]

Description of PR

pr #955 rebased to trunk

How was this patch tested?

there's a new test

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

this is the original PR; if it compiles and the tests pass, i will merge as it.

i do think it needs some more tests (what if a file isn't there, setting invalid options etc)

be interesting to think of actually adding tests for this against s3/abfs too

[steveloughran]

test run of the branch in the docker image on an MBP m1 with tests in /hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client

     [exec] The following tests FAILED:
     [exec]      14 - memcheck_rpc_engine (Failed)
     [exec]      34 - memcheck_hdfs_config_connect_bugs (Failed)
     [exec]      38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)

will need to see if they happen on trunk as is

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	44m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 54s		trunk passed
+1 💚	compile	3m 46s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	70m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 13s		the patch passed
+1 💚	compile	3m 38s		the patch passed
+1 💚	cc	3m 38s		the patch passed
+1 💚	golang	3m 38s		the patch passed
+1 💚	javac	3m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	22m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	93m 15s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		236m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/artifact/out/Dockerfile
GITHUB PR	#4166
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux e288a158ab82 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0d383e5
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/testReport/
Max. process+thread count	522 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 50s		trunk passed
+1 💚	compile	3m 55s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	shadedclient	53m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	3m 45s		the patch passed
+1 💚	cc	3m 45s		the patch passed
+1 💚	golang	3m 45s		the patch passed
+1 💚	javac	3m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	shadedclient	22m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	104m 33s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		213m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/artifact/out/Dockerfile
GITHUB PR	#4166
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux f79fb0ed5fd5 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0d383e5
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/testReport/
Max. process+thread count	598 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	15m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 28s		trunk passed
+1 💚	compile	3m 32s		trunk passed
+1 💚	mvnsite	0m 21s		trunk passed
+1 💚	shadedclient	67m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 27s		the patch passed
+1 💚	cc	3m 27s		the patch passed
+1 💚	golang	3m 27s		the patch passed
+1 💚	javac	3m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	31m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	92m 27s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		213m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/artifact/out/Dockerfile
GITHUB PR	#4166
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 10f36a3ff98f 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0d383e5
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/testReport/
Max. process+thread count	536 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 26s		trunk passed
+1 💚	compile	4m 11s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	compile	4m 9s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 22s		trunk passed
+1 💚	shadedclient	56m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 57s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	cc	3m 57s		the patch passed
+1 💚	golang	3m 57s		the patch passed
+1 💚	javac	3m 57s		the patch passed
+1 💚	compile	4m 1s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	4m 1s		the patch passed
+1 💚	golang	4m 1s		the patch passed
+1 💚	javac	4m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	22m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	117m 41s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		209m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/artifact/out/Dockerfile
GITHUB PR	#4166
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux e0ace121df62 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0d383e5
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/testReport/
Max. process+thread count	612 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

The same tests fail on my arm64 docker vm too; so not related.

+1 for sahil's patch.

I'd like to followup with some tests of failure conditions, especially once #2584 is in, but not here

[steveloughran]

failures on branch-3.3 are the same,
[exec] The following tests FAILED:
[exec] 14 - memcheck_rpc_engine (Failed)
[exec] 34 - memcheck_hdfs_config_connect_bugs (Failed)
[exec] 38 - memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)