-
Couldn't load subscription status.
- Fork 9.1k
HADOOP-15870 S3AInputStream.remainingInFile should use nextReadPos #433
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Fix the nextPos
Tests. In
For some of the other tests, I think you insert a couple of checks to say " available > 0" after a seek + read() Call. This also clarifies that "what should be available after a seek but before a read()?" As for cloudstores with lazy seek, available == 0, though if that breaks gzip then maybe they should return 1, so the assert should be available >1 with an option in the contract to say "actually returns 0". It's a tricky one to test on because really, the sole asserts should be
Probably assert #3 is the one to check for: available() is valid when it is >=0 and <= (filelength -getPos()), with care taken about that second clause to make sure there's no off-by-one errors in the assert/code |
|
@steveloughran Thanks very much , I will update. |
1, Fix remainingInCurrentRequest 2, append assert in the Junit Test
|
@steveloughran I have updated the code , please help reveiw and comment, thanks . |
|
As the usual test police, which S3 endpoint did you rerun all the hadoop-aws tests against? This only takes < 10 minutes, even remotely I'll help with testing the other stores (azure wasb/abfs/adl), before the commit, but I'd appreciate the diligence here as it shows you've done the homework. thx |
|
@steveloughran Due to the upgrade to the latest version of the Mac OS, I need take some time to resolve the environment before I can run the test successful . |
|
no worries |
fix junit
update the available logic fix junit
Fix the nextPos
|
this is merged in, closing |
1. Added a meaningful name for the container thread pool threads. 2. Made the thread names for framework threads consistent. 3. Made a couple of monitoring/metrics threads daemon. 4. Fixed a few checkstyle warning about missing param/throws documentation. Author: Prateek Maheshwari <[email protected]> Reviewers: Jagadish <[email protected]>, Jacob M <[email protected]> Closes apache#433 from prateekm/container-thread-pool-name
Co-authored-by: Joseph DellAringa <[email protected]>
Author: Nandor Kollar <[email protected]> Closes apache#433 from nandorKollar/PARQUET-1115 and squashes the following commits: 5504a39 [Nandor Kollar] PARQUET-1115: Warn users when misusing parquet-tools merge f2ece26 [Nandor Kollar] PARQUET-1115: Warn users when misusing parquet-tools merge 4f3ec99 [Nandor Kollar] PARQUET-1115: Warn users when misusing parquet-tools merge f97e620 [Nandor Kollar] PARQUET-1115: Prevent users from misusing parquet-tools merge
Fix the nextPos