HADOOP-18853. Upgrades SDK version to 2.20.28 and restores multipart copy #5960
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@steveloughran this updates the SDK version, so the Java async client has MPU. Since Multipart operations are in the java async client now, and not the TM, we could consider removing the TM in a follow up PR. We'd lose out on the transfer listener..but we don't really use any of the other additional functionality the TM provides us. |
steveloughran
approved these changes
Aug 18, 2023
There was a problem hiding this comment.
Looks good though I want my #5872 patch in first -which is depending on approval from someone with commit rights.
please make sure jira and pr title states what library is being updated and what is it changing too, e.g
Upgrade AWS SDK to 2.19.12 and restore multipart copy
Once this PR is merged in the v2 line is feature complete w.r.t the v1 SDK code, isn't it? which means it'll be time to rebase the branch then merge to trunk, plus same for the 3.3 backport!
| Configuration conf = getConf(); | ||
| String bucket = uri.getHost(); | ||
|
|
||
| NettyNioAsyncHttpClient.Builder httpClientBuilder = AWSClientConfig |
There was a problem hiding this comment.
is this going to build properly with unshaded artifacts? we've caused problems in the path (#2599) because of refs to .shaded classes.. the netty and client stuff are public/stable unshaded classes, correct?
|
💔 -1 overall
This message was automatically generated. |
|
@ahmarsuhail can you rebase and retest this, then i'll merge. once in we can rebase the whole feature branch, retest -and then merge to trunk! |
ahmarsuhail
changed the title
ahmarsuhail
changed the title
ahmarsuhail
force-pushed
the
feature-HADOOP-18853-upgrade-v2-version
branch
from
6a328d3 to
54dcf48
Compare
|
@steveloughran - Rebased and tested, mostly all good. Also seeing failures in Fails on my EC2, but works ok on my mac. It's probably a config issue, but haven't seen this ever before so need to investigate. Again, don't think it has anything to do with this change. so feel like we should be good to merge. Once we're in trunk, I'll work on adding CSE, that's the only thing missing now. |
|
💔 -1 overall
This message was automatically generated. |
|
your ec2 vm is using IAM session credentials, isn't it? so it can't issue full credential DTs. probably worth having the test skip if that's the case |
steveloughran
requested changes
Aug 22, 2023
hadoop-project/pom.xml
Outdated
| <hsqldb.version>2.7.1</hsqldb.version> | ||
| <aws-java-sdk-v2.version>2.19.12</aws-java-sdk-v2.version> | ||
| <aws-java-sdk-v2.version>2.20.128</aws-java-sdk-v2.version> | ||
| <aws.evenstream.version>1.0.1</aws.evenstream.version> |
There was a problem hiding this comment.
while you are here, how about renaming evenstream to eventstream; only just noticed this
|
💔 -1 overall
This message was automatically generated. |
steveloughran
pushed a commit
to steveloughran/hadoop
that referenced
this pull request
Aug 23, 2023
Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail
steveloughran
pushed a commit
to steveloughran/hadoop
that referenced
this pull request
Aug 28, 2023
Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail
steveloughran
pushed a commit
to steveloughran/hadoop
that referenced
this pull request
Aug 31, 2023
Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail
steveloughran
pushed a commit
to steveloughran/hadoop
that referenced
this pull request
Sep 1, 2023
This is an aggregate patch of the changes from feature-HADOOP-18073-s3a-sdk-upgrade and moves the S3A connector to to using the V2 AWS SDK This is a major change: See aws_sdk_v2_changelog.md for details. A new shaded v2 SDK JAR "bundle.jar" needs to be distributed with the connector to interact with S3 stores All code which was using the V1 SDK classes with the S3AFileSystem will need upgrading. Contributed by Ahmar Suhail HADOOP-18820. Cut AWS v1 support (apache#5872) This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran HADOOP-18853. Upgrade AWS SDK version to 2.20.28 (apache#5960) Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail HADOOP-18818. Merge aws v2 upgrade feature branch into trunk Contains HADOOP-18863. AWS SDK V2 - AuditFailureExceptions aren't being translated properly Change-Id: I96b26cc1ee535c519248ca6541fb157017dcc7e4
steveloughran
pushed a commit
to steveloughran/hadoop
that referenced
this pull request
Sep 1, 2023
This is an aggregate patch of the changes from feature-HADOOP-18073-s3a-sdk-upgrade and moves the S3A connector to to using the V2 AWS SDK This is a major change: See aws_sdk_v2_changelog.md for details. A new shaded v2 SDK JAR "bundle.jar" needs to be distributed with the connector to interact with S3 stores All code which was using the V1 SDK classes with the S3AFileSystem will need upgrading. Contributed by Ahmar Suhail HADOOP-18820. Cut AWS v1 support (apache#5872) This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran HADOOP-18853. Upgrade AWS SDK version to 2.20.28 (apache#5960) Upgrades the AWS sdk v2 version to 2.20.28 This * adds multipart COPY/rename in the java async client * removes the aws-crt JAR dependency Contributed by Ahmar Suhail HADOOP-18818. Merge aws v2 upgrade feature branch into trunk Contains HADOOP-18863. AWS SDK V2 - AuditFailureExceptions aren't being translated properly Change-Id: I96b26cc1ee535c519248ca6541fb157017dcc7e4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
How was this patch tested?
Tested in eu-west-1 with
mvn -Dparallel-tests -DtestsThreadCount=16 clean verify.Also ran checked output of ITestS3HugeFileArrayBlocks.test_100_renameHugeFile(), time taken to rename a 256MB file is at par with V1, around ~1s from my m4.2xlarge EC2.