HADOOP-17338. Intermittent S3AInputStream failures: Premature end of …

[yzhangal]

…Content-Length delimited message body etc

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 23s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 33s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 25s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	shadedclient	18m 1s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 29s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	1m 6s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 3s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 28s		the patch passed
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 3s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	1m 8s		the patch passed
			_ Other Tests _
+1 💚	unit	1m 21s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		83m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2497/1/artifact/out/Dockerfile
GITHUB PR	#2497
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3c8eb8c7c0be 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 68442b4
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2497/1/testReport/
Max. process+thread count	595 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2497/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[yzhangal]

Test result of hadoop-tools/hadoop-aws tests with command "mvn verify -Dscale -DtestsThreadCount=8 ":

[ERROR] Errors:
[ERROR] ITestS3AContractRootDir.testListEmptyRootDirectory:82->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:196 » TestTimedOut
[ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:265 » TestTimedOut
[ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:101 » TestTimedOut
[ERROR] ITestS3ATemporaryCredentials.testSTS:133 » AccessDenied : request session cred...
[INFO]
[ERROR] Tests run: 260, Failures: 0, Errors: 4, Skipped: 45

Hi @steveloughran thanks a lot for the review so far. Would you please help taking a look?

The STS one is understandable because we don't have STS support in my test env. The others doesn't seem related. wonder if it runs for you?

Thanks.

[steveloughran]

if your test setup doesn' t have session credentials, don't worry about it.
What are you testing against? AWS or something internal?

[steveloughran]

thanks, I'll try and look @ this week. First I'll have to look at the previous one to remember where we had got to on that...I don't want to suddenly be inconsistent with what I'd written there.

[yzhangal]

thanks, I'll try and look @ this week. First I'll have to look at the previous one to remember where we had got to on that...I don't want to suddenly be inconsistent with what I'd written there.

Thanks a lot @steveloughran ! I tested against AWS.

[steveloughran]

ok. merged to trunk & just doing the 3.3 branch now

[yzhangal]

ok. merged to trunk & just doing the 3.3 branch now

Great, thanks so much @steveloughran !

[steveloughran]

Great, thanks so much @steveloughran !

thank you for finding/fixing an obscure bug.

Incidentally, if you are hanging on to streams for a long time. the unbuffer() method will release the stream and push out the current statistics to the FileSystem stats; this is how Impala manages long-lived streams

[yzhangal]

Happy new year and many thanks again @steveloughran !

Good info about unbuffer() too!

[steveloughran]

yes, use unbuffer if you can. It frees up the HTTPS connection. And while AWS S3 won't have problems, its probably good for other S3 stores as it will reduce the #of open connections the server has to maintain