HADOOP-14566. Add seek support for SFTP FileSystem. #1999

mpryahin · 2020-05-06T19:44:50Z

This PR implements seek method for SFTP File System which supports both forward and backward lazy seeks.

Normally subsequent reads proceed reading from the position where the previous read finished, meaning we can avoid making seek operations in this case. We only need to seek when the current FsDataInputstream#getPos() doesn't equal a requested position to read from.

Whenever seek is called it's deferred until the next read operation. If a consecutive read happens at the same position where the previous read finished then it merely proceeds reading. If a consecutive read happens at the position greater than the position where the previous read finished we seek forward to the requested position. And finally, if a consecutive read happens at the position smaller than the current inputstream position then we close the current inputstream, reopen a new one and seek to the requested position.

hadoop-yetus · 2020-05-06T22:04:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 21s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 24s	trunk passed
+1 💚	compile	17m 3s	trunk passed
+1 💚	checkstyle	0m 53s	trunk passed
+1 💚	mvnsite	1m 27s	trunk passed
+1 💚	shadedclient	16m 29s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 7s	trunk passed
+0 🆗	spotbugs	2m 17s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 15s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 53s	the patch passed
+1 💚	compile	16m 56s	the patch passed
+1 💚	javac	16m 56s	the patch passed
-0 ⚠️	checkstyle	0m 49s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 30s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	2m 47s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 3s	the patch passed
-1 ❌	findbugs	2m 15s	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
-1 ❌	unit	9m 37s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 53s	The patch does not generate ASF License warnings.
		118m 58s

Reason	Tests
FindBugs	module:hadoop-common-project/hadoop-common
	Inconsistent synchronization of org.apache.hadoop.fs.sftp.SFTPInputStream.nextPos; locked 72% of time Unsynchronized access at SFTPInputStream.java:72% of time Unsynchronized access at SFTPInputStream.java:[line 72]
Failed junit tests	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.fs.contract.localfs.TestLocalFSContractMultipartUploader
	hadoop.io.compress.TestCompressorDecompressor

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux c5d60f4143bd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3501012`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/testReport/
Max. process+thread count	2202 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-05-07T08:53:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 30s	trunk passed
+1 💚	compile	17m 5s	trunk passed
+1 💚	checkstyle	0m 50s	trunk passed
+1 💚	mvnsite	1m 24s	trunk passed
+1 💚	shadedclient	17m 20s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 3s	trunk passed
+0 🆗	spotbugs	2m 9s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 7s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s	the patch passed
+1 💚	compile	16m 15s	the patch passed
+1 💚	javac	16m 15s	the patch passed
-0 ⚠️	checkstyle	0m 51s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	13m 53s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 2s	the patch passed
+1 💚	findbugs	2m 16s	the patch passed
		_ Other Tests _
-1 ❌	unit	9m 31s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		108m 12s

Reason	Tests
Failed junit tests	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor
	hadoop.fs.contract.localfs.TestLocalFSContractMultipartUploader
	hadoop.io.compress.TestCompressorDecompressor
	hadoop.fs.ftp.TestFTPFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/2/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 17f3607b64bf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `99840aa`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/2/testReport/
Max. process+thread count	2851 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-05-07T13:29:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 43s	trunk passed
+1 💚	compile	17m 50s	trunk passed
+1 💚	checkstyle	0m 45s	trunk passed
+1 💚	mvnsite	1m 21s	trunk passed
+1 💚	shadedclient	17m 30s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 55s	trunk passed
+0 🆗	spotbugs	2m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 6s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s	the patch passed
+1 💚	compile	17m 6s	the patch passed
+1 💚	javac	17m 6s	the patch passed
-0 ⚠️	checkstyle	0m 44s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 23s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 25s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 55s	the patch passed
+1 💚	findbugs	2m 13s	the patch passed
		_ Other Tests _
-1 ❌	unit	9m 49s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		113m 36s

Reason	Tests
Failed junit tests	hadoop.io.compress.TestCompressorDecompressor
	hadoop.io.compress.snappy.TestSnappyCompressorDecompressor

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/3/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 15a013c2df32 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `192cad9`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/3/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/3/testReport/
Max. process+thread count	1346 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

mpryahin · 2020-05-07T14:08:30Z

The PR is ready for review.

…14566

hadoop-yetus · 2020-05-11T10:41:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 50s	trunk passed
+1 💚	compile	17m 17s	trunk passed
+1 💚	checkstyle	0m 45s	trunk passed
+1 💚	mvnsite	1m 23s	trunk passed
+1 💚	shadedclient	15m 42s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	trunk passed
+0 🆗	spotbugs	2m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 7s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 53s	the patch passed
+1 💚	compile	17m 15s	the patch passed
+1 💚	javac	17m 15s	the patch passed
-0 ⚠️	checkstyle	0m 47s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	18m 21s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 8s	the patch passed
+1 💚	findbugs	2m 57s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 48s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		135m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/4/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 535da9d2cfdb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `328eae9`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/4/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/4/testReport/
Max. process+thread count	2161 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…14566

hadoop-yetus · 2020-05-15T13:54:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
-1 ❌	mvninstall	20m 24s	root in trunk failed.
+1 💚	compile	17m 17s	trunk passed
+1 💚	checkstyle	0m 52s	trunk passed
+1 💚	mvnsite	1m 28s	trunk passed
-1 ❌	shadedclient	17m 21s	branch has errors when building and testing our client artifacts.
+1 💚	javadoc	1m 6s	trunk passed
+0 🆗	spotbugs	2m 11s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 9s	trunk passed
-0 ⚠️	patch	2m 30s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s	the patch passed
+1 💚	compile	16m 34s	the patch passed
+1 💚	javac	16m 34s	the patch passed
-0 ⚠️	checkstyle	0m 52s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
-1 ❌	shadedclient	14m 21s	patch has errors when building and testing our client artifacts.
+1 💚	javadoc	1m 6s	the patch passed
+1 💚	findbugs	2m 17s	the patch passed
		_ Other Tests _
-1 ❌	unit	9m 27s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		132m 14s

Reason	Tests
Failed junit tests	hadoop.metrics2.source.TestJvmMetrics
	hadoop.security.token.delegation.TestZKDelegationTokenSecretManager

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux f1722534f24a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ac4a2e1`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/artifact/out/branch-mvninstall-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/testReport/
Max. process+thread count	1595 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/5/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…14566

hadoop-yetus · 2020-05-18T13:18:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
-1 ❌	mvninstall	10m 20s	root in trunk failed.
-1 ❌	compile	0m 37s	root in trunk failed.
-0 ⚠️	checkstyle	0m 25s	The patch fails to run checkstyle in hadoop-common
-1 ❌	mvnsite	0m 26s	hadoop-common in trunk failed.
-1 ❌	shadedclient	2m 15s	branch has errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 28s	hadoop-common in trunk failed.
+0 🆗	spotbugs	3m 14s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 29s	hadoop-common in trunk failed.
-0 ⚠️	patch	3m 41s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
-1 ❌	mvninstall	0m 23s	hadoop-common in the patch failed.
+1 💚	compile	21m 52s	the patch passed
-1 ❌	javac	21m 52s	root generated 1859 new + 1 unchanged - 0 fixed = 1860 total (was 1)
-0 ⚠️	checkstyle	0m 52s	hadoop-common-project/hadoop-common: The patch generated 20 new + 0 unchanged - 0 fixed = 20 total (was 0)
+1 💚	mvnsite	1m 33s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 40s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 58s	hadoop-common-project_hadoop-common generated 101 new + 0 unchanged - 0 fixed = 101 total (was 0)
+1 💚	findbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 27s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		72m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux ee7e96502fa0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b65815d`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/branch-compile-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/buildtool-branch-checkstyle-hadoop-common-project_hadoop-common.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/diff-compile-javac-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/testReport/
Max. process+thread count	2884 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/6/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…14566

hadoop-yetus · 2020-05-18T20:22:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 40s	trunk passed
+1 💚	compile	16m 54s	trunk passed
+1 💚	checkstyle	0m 51s	trunk passed
+1 💚	mvnsite	1m 27s	trunk passed
+1 💚	shadedclient	16m 33s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 3s	trunk passed
+0 🆗	spotbugs	2m 9s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 7s	trunk passed
-0 ⚠️	patch	2m 28s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s	the patch passed
+1 💚	compile	16m 17s	the patch passed
+1 💚	javac	16m 17s	the patch passed
-0 ⚠️	checkstyle	0m 51s	hadoop-common-project/hadoop-common: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1 💚	mvnsite	1m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 9s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	the patch passed
+1 💚	findbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 34s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		106m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/7/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 63b7866f1387 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c84e6be`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/7/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/7/testReport/
Max. process+thread count	1714 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/7/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java

steveloughran

You done some great work here!

A few minor details and it will be ready to go in.

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java

steveloughran · 2020-05-19T16:30:16Z

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

+    sshd.start();
+    int port = sshd.getPort();
+
+    conf.setClass("fs.sftp.impl", SFTPFileSystem.class, FileSystem.class);


shouldn't this be set in core-default.xml already?
if not, sftp:// urls would break. (yes, i know every stack overflow spark example does this, but that is just superstition)

it's not set in core-default.xml, and if not specified here sftp urls won't be resolved by sftp schema. Could you please clarify a bit what exactly you mean here? Thank you!

double checked, it's quite strange. Is it a candidate for an improvement issue?

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

steveloughran · 2020-05-19T16:33:46Z

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

+  }
+
+  @Override
+  public Path getTestPath() {


shame we didn't declare this as raising an ioe; probably too late now

it's not too late I suppose, would you like me to try to declare it as throwing IOE? I can issue a new Jira and fix it there.

hadoop-common-project/hadoop-common/src/test/resources/contract/sftp.xml

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java

…14566

mpryahin

thanks a lot for your review

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

hadoop-common-project/hadoop-common/src/test/resources/contract/sftp.xml

mpryahin · 2020-05-19T17:29:38Z

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

+    sshd.start();
+    int port = sshd.getPort();
+
+    conf.setClass("fs.sftp.impl", SFTPFileSystem.class, FileSystem.class);


it's not set in core-default.xml, and if not specified here sftp urls won't be resolved by sftp schema. Could you please clarify a bit what exactly you mean here? Thank you!

mpryahin · 2020-05-19T17:39:30Z

...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/sftp/SFTPContract.java

+  }
+
+  @Override
+  public Path getTestPath() {


it's not too late I suppose, would you like me to try to declare it as throwing IOE? I can issue a new Jira and fix it there.

hadoop-yetus · 2020-05-19T23:07:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 42s	trunk passed
+1 💚	compile	16m 59s	trunk passed
+1 💚	checkstyle	0m 52s	trunk passed
+1 💚	mvnsite	1m 25s	trunk passed
+1 💚	shadedclient	16m 36s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	trunk passed
+0 🆗	spotbugs	2m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 4s	trunk passed
-0 ⚠️	patch	2m 25s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s	the patch passed
+1 💚	compile	16m 17s	the patch passed
+1 💚	javac	16m 17s	the patch passed
+1 💚	checkstyle	0m 51s	the patch passed
+1 💚	mvnsite	1m 28s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	13m 56s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 2s	the patch passed
+1 💚	findbugs	2m 14s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 27s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		106m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/8/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 0aa5cb019133 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `8f78aeb`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/8/testReport/
Max. process+thread count	1903 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/8/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

mpryahin · 2020-05-23T11:26:21Z

@steveloughran , thanks a lot for review. All the issues you pointed out have been fixed. Is there anything else to be done from my side?

...on-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractFSContract.java

hadoop-yetus · 2020-06-02T15:59:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 22s	trunk passed
+1 💚	compile	17m 15s	trunk passed
+1 💚	checkstyle	0m 53s	trunk passed
+1 💚	mvnsite	1m 28s	trunk passed
+1 💚	shadedclient	16m 48s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 5s	trunk passed
+0 🆗	spotbugs	2m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 6s	trunk passed
-0 ⚠️	patch	2m 28s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s	the patch passed
+1 💚	compile	16m 21s	the patch passed
+1 💚	javac	16m 21s	the patch passed
+1 💚	checkstyle	0m 51s	hadoop-common-project/hadoop-common: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19)
+1 💚	mvnsite	1m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	13m 50s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	the patch passed
+1 💚	findbugs	2m 15s	the patch passed
		_ Other Tests _
+1 💚	unit	9m 31s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		107m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/9/artifact/out/Dockerfile
GITHUB PR	#1999
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 71756c43fcb1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `aa6d134`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/9/testReport/
Max. process+thread count	1393 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1999/9/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Contributed by Mikhail Pryakhin

steveloughran · 2020-06-03T10:39:18Z

+1, merged to trunk and branch-3.3

thanks!

mpryahin added 3 commits May 6, 2020 20:39

seek method implementation

7f95e24

fixed code style issues

e0fa916

close ssh server only in case it is initialised

9fa1307

fixed broken thread safety

9896efd

Fixed broken tests

46a2d51

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

fb098c7

…14566

mpryahin added 3 commits May 14, 2020 20:52

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

6bad392

…14566

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

a9bc169

…14566

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

773284d

…14566

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

5bf201c

…14566

mpryahin added 3 commits May 18, 2020 17:31

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

bd5f4df

…14566

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

5760937

…14566

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

86bdff3

…14566

steveloughran reviewed May 19, 2020

View reviewed changes

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java Show resolved Hide resolved

steveloughran requested changes May 19, 2020

View reviewed changes

mpryahin added 3 commits May 19, 2020 20:50

code review improvements

4e823fd

Merge branch 'trunk' of https://github.com/apache/hadoop into HADOOP-…

3bc3e90

…14566

codestyle improvements

fb49a21

mpryahin commented May 19, 2020

View reviewed changes

make sure an underlying channel gets closed

b874cdf

mpryahin requested a review from steveloughran May 21, 2020 07:01

steveloughran requested changes Jun 2, 2020

View reviewed changes

...on-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractFSContract.java Outdated Show resolved Hide resolved

fixed missing persiod in javadoc

86bf896

mpryahin requested a review from steveloughran June 2, 2020 13:52

steveloughran merged commit 97c98ce into apache:trunk Jun 3, 2020

asfgit pushed a commit that referenced this pull request Jun 3, 2020

HADOOP-14566. Add seek support for SFTP FileSystem. (#1999)

cf84bec

Contributed by Mikhail Pryakhin

HADOOP-14566. Add seek support for SFTP FileSystem. #1999

HADOOP-14566. Add seek support for SFTP FileSystem. #1999

Uh oh!

Conversation

mpryahin commented May 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented May 6, 2020

Uh oh!

hadoop-yetus commented May 7, 2020

Uh oh!

hadoop-yetus commented May 7, 2020

Uh oh!

mpryahin commented May 7, 2020

Uh oh!

hadoop-yetus commented May 11, 2020

Uh oh!

hadoop-yetus commented May 15, 2020

Uh oh!

hadoop-yetus commented May 18, 2020

Uh oh!

hadoop-yetus commented May 18, 2020

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

steveloughran May 19, 2020

Choose a reason for hiding this comment

Uh oh!

mpryahin May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpryahin May 19, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran May 19, 2020

Choose a reason for hiding this comment

Uh oh!

mpryahin May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mpryahin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mpryahin May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpryahin May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented May 19, 2020

Uh oh!

mpryahin commented May 23, 2020

Uh oh!

Uh oh!

hadoop-yetus commented Jun 2, 2020

Uh oh!

steveloughran commented Jun 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

mpryahin commented May 6, 2020 •

edited

Loading

mpryahin May 19, 2020 •

edited

Loading

mpryahin May 19, 2020 •

edited

Loading

mpryahin May 19, 2020 •

edited

Loading

mpryahin May 19, 2020 •

edited

Loading