Skip to content

Commit d43e7ce

Browse files
steveloughransteveloughran
committed
HADOOP-18820. Cut AWS v1 support (apache#5872)
This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran This patch (unlike the feature branch) does not have an enforcer rule to reject use of com.amazonaws classes; yetus was rejecting it. Change-Id: I854fc5d8707016a87a1321e46c1f0ad726ceff3a
1 parent d2eade0 commit d43e7ce

File tree

70 files changed

+2296
-1211
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+2296
-1211
lines changed

‎LICENSE-binary‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -364,6 +364,8 @@ org.objenesis:objenesis:2.6
364364
org.xerial.snappy:snappy-java:1.1.10.1
365365
org.yaml:snakeyaml:2.0
366366
org.wildfly.openssl:wildfly-openssl:1.1.3.Final
367+
software.amazon.awssdk:bundle:jar:2.19.12
368+
software.amazon.awssdk.crt:aws-crt:0.21.0
367369

368370

369371
--------------------------------------------------------------------------------

‎hadoop-common-project/hadoop-common/src/main/resources/core-default.xml‎

Lines changed: 11 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1387,61 +1387,31 @@
13871387
<description>AWS secret key used by S3A file system. Omit for IAM role-based or provider-based authentication.</description>
13881388
</property>
13891389

1390+
<property>
1391+
<name>fs.s3a.session.token</name>
1392+
<description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
1393+
as one of the providers.
1394+
</description>
1395+
</property>
1396+
13901397
<property>
13911398
<name>fs.s3a.aws.credentials.provider</name>
13921399
<value>
13931400
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
13941401
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
1395-
com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
1402+
software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
13961403
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
13971404
</value>
13981405
<description>
13991406
Comma-separated class names of credential provider classes which implement
1400-
com.amazonaws.auth.AWSCredentialsProvider.
1407+
software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.
14011408

14021409
When S3A delegation tokens are not enabled, this list will be used
14031410
to directly authenticate with S3 and other AWS services.
14041411
When S3A Delegation tokens are enabled, depending upon the delegation
14051412
token binding it may be used
14061413
to communicate wih the STS endpoint to request session/role
14071414
credentials.
1408-
1409-
These are loaded and queried in sequence for a valid set of credentials.
1410-
Each listed class must implement one of the following means of
1411-
construction, which are attempted in order:
1412-
* a public constructor accepting java.net.URI and
1413-
org.apache.hadoop.conf.Configuration,
1414-
* a public constructor accepting org.apache.hadoop.conf.Configuration,
1415-
* a public static method named getInstance that accepts no
1416-
arguments and returns an instance of
1417-
com.amazonaws.auth.AWSCredentialsProvider, or
1418-
* a public default constructor.
1419-
1420-
Specifying org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider allows
1421-
anonymous access to a publicly accessible S3 bucket without any credentials.
1422-
Please note that allowing anonymous access to an S3 bucket compromises
1423-
security and therefore is unsuitable for most use cases. It can be useful
1424-
for accessing public data sets without requiring AWS credentials.
1425-
1426-
If unspecified, then the default list of credential provider classes,
1427-
queried in sequence, is:
1428-
* org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider: looks
1429-
for session login secrets in the Hadoop configuration.
1430-
* org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider:
1431-
Uses the values of fs.s3a.access.key and fs.s3a.secret.key.
1432-
* com.amazonaws.auth.EnvironmentVariableCredentialsProvider: supports
1433-
configuration of AWS access key ID and secret access key in
1434-
environment variables named AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
1435-
and AWS_SESSION_TOKEN as documented in the AWS SDK.
1436-
* org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider: picks up
1437-
IAM credentials of any EC2 VM or AWS container in which the process is running.
1438-
</description>
1439-
</property>
1440-
1441-
<property>
1442-
<name>fs.s3a.session.token</name>
1443-
<description>Session token, when using org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
1444-
as one of the providers.
14451415
</description>
14461416
</property>
14471417

@@ -1539,10 +1509,10 @@
15391509
Note: for job submission to actually collect these tokens,
15401510
Kerberos must be enabled.
15411511

1542-
Options are:
1512+
Bindings available in hadoop-aws are:
15431513
org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding
15441514
org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding
1545-
and org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding
1515+
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding
15461516
</description>
15471517
</property>
15481518

‎hadoop-project/pom.xml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,7 @@
185185
<aws-java-sdk.version>1.12.499</aws-java-sdk.version>
186186
<hsqldb.version>2.7.1</hsqldb.version>
187187
<aws-java-sdk-v2.version>2.19.12</aws-java-sdk-v2.version>
188+
<aws.evenstream.version>1.0.1</aws.evenstream.version>
188189
<awscrt.version>0.21.0</awscrt.version>
189190
<frontend-maven-plugin.version>1.11.2</frontend-maven-plugin.version>
190191
<jasmine-maven-plugin.version>2.1</jasmine-maven-plugin.version>
@@ -1132,18 +1133,29 @@
11321133
<groupId>com.amazonaws</groupId>
11331134
<artifactId>aws-java-sdk-core</artifactId>
11341135
<version>${aws-java-sdk.version}</version>
1136+
<exclusions>
1137+
<exclusion>
1138+
<groupId>*</groupId>
1139+
<artifactId>*</artifactId>
1140+
</exclusion>
1141+
</exclusions>
11351142
</dependency>
11361143
<dependency>
11371144
<groupId>software.amazon.awssdk</groupId>
11381145
<artifactId>bundle</artifactId>
11391146
<version>${aws-java-sdk-v2.version}</version>
11401147
<exclusions>
11411148
<exclusion>
1142-
<groupId>io.netty</groupId>
1149+
<groupId>*</groupId>
11431150
<artifactId>*</artifactId>
11441151
</exclusion>
11451152
</exclusions>
11461153
</dependency>
1154+
<dependency>
1155+
<groupId>software.amazon.eventstream</groupId>
1156+
<artifactId>eventstream</artifactId>
1157+
<version>${aws.evenstream.version}</version>
1158+
</dependency>
11471159
<dependency>
11481160
<groupId>software.amazon.awssdk.crt</groupId>
11491161
<artifactId>aws-crt</artifactId>

‎hadoop-tools/hadoop-aws/dev-support/findbugs-exclude.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,11 @@
6464
<Field name="futurePool"/>
6565
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
6666
</Match>
67+
<Match>
68+
<Class name="org.apache.hadoop.fs.s3a.S3AFileSystem"/>
69+
<Field name="s3AsyncClient"/>
70+
<Bug pattern="IS2_INCONSISTENT_SYNC"/>
71+
</Match>
6772
<Match>
6873
<Class name="org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo"/>
6974
<Method name="run"/>

‎hadoop-tools/hadoop-aws/pom.xml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -494,10 +494,14 @@
494494
<scope>test</scope>
495495
<type>test-jar</type>
496496
</dependency>
497+
498+
<!-- The v1 SDK is used at compilation time for adapter classes in
499+
org.apache.hadoop.fs.s3a.adapter. It is not needed at runtime
500+
unless a non-standard v1 credential provider is declared. -->
497501
<dependency>
498502
<groupId>com.amazonaws</groupId>
499503
<artifactId>aws-java-sdk-core</artifactId>
500-
<scope>compile</scope>
504+
<scope>provided</scope>
501505
</dependency>
502506
<dependency>
503507
<groupId>software.amazon.awssdk</groupId>
@@ -507,7 +511,11 @@
507511
<dependency>
508512
<groupId>software.amazon.awssdk.crt</groupId>
509513
<artifactId>aws-crt</artifactId>
510-
<scope>compile</scope>
514+
</dependency>
515+
<dependency>
516+
<groupId>software.amazon.eventstream</groupId>
517+
<artifactId>eventstream</artifactId>
518+
<scope>test</scope>
511519
</dependency>
512520
<dependency>
513521
<groupId>org.assertj</groupId>

‎hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java‎

Lines changed: 14 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -27,28 +27,21 @@
2727
import java.util.concurrent.atomic.AtomicInteger;
2828
import java.util.stream.Collectors;
2929

30-
import com.amazonaws.auth.AWSCredentials;
31-
import com.amazonaws.auth.AWSCredentialsProvider;
32-
import com.amazonaws.auth.BasicAWSCredentials;
33-
import com.amazonaws.auth.BasicSessionCredentials;
34-
import org.apache.hadoop.classification.VisibleForTesting;
35-
import org.apache.hadoop.fs.s3a.adapter.V1V2AwsCredentialProviderAdapter;
36-
import org.apache.hadoop.util.Preconditions;
37-
3830
import org.slf4j.Logger;
3931
import org.slf4j.LoggerFactory;
4032

4133
import org.apache.commons.lang3.StringUtils;
34+
import org.apache.hadoop.classification.VisibleForTesting;
4235
import org.apache.hadoop.classification.InterfaceAudience;
4336
import org.apache.hadoop.classification.InterfaceStability;
4437
import org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException;
4538
import org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException;
4639
import org.apache.hadoop.io.IOUtils;
40+
import org.apache.hadoop.util.Preconditions;
4741

4842
import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
4943
import software.amazon.awssdk.auth.credentials.AwsCredentials;
5044
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
51-
import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
5245
import software.amazon.awssdk.core.exception.SdkException;
5346

5447
/**
@@ -106,23 +99,8 @@ public AWSCredentialProviderList() {
10699
* @param providers provider list.
107100
*/
108101
public AWSCredentialProviderList(
109-
Collection<AWSCredentialsProvider> providers) {
110-
for (AWSCredentialsProvider provider: providers) {
111-
this.providers.add(V1V2AwsCredentialProviderAdapter.adapt(provider));
112-
}
113-
}
114-
115-
/**
116-
* Create with an initial list of providers.
117-
* @param name name for error messages, may be ""
118-
* @param providerArgs provider list.
119-
*/
120-
public AWSCredentialProviderList(final String name,
121-
final AWSCredentialsProvider... providerArgs) {
122-
setName(name);
123-
for (AWSCredentialsProvider provider: providerArgs) {
124-
this.providers.add(V1V2AwsCredentialProviderAdapter.adapt(provider));
125-
}
102+
Collection<AwsCredentialsProvider> providers) {
103+
this.providers.addAll(providers);
126104
}
127105

128106
/**
@@ -148,14 +126,6 @@ public void setName(final String name) {
148126
}
149127
}
150128

151-
/**
152-
* Add a new provider.
153-
* @param provider provider
154-
*/
155-
public void add(AWSCredentialsProvider provider) {
156-
providers.add(V1V2AwsCredentialProviderAdapter.adapt(provider));
157-
}
158-
159129
/**
160130
* Add a new SDK V2 provider.
161131
* @param provider provider
@@ -164,7 +134,6 @@ public void add(AwsCredentialsProvider provider) {
164134
providers.add(provider);
165135
}
166136

167-
168137
/**
169138
* Add all providers from another list to this one.
170139
* @param other the other list.
@@ -174,19 +143,11 @@ public void addAll(AWSCredentialProviderList other) {
174143
}
175144

176145
/**
177-
* This method will get credentials using SDK V2's resolveCredentials and then convert it into
178-
* V1 credentials. This required by delegation token binding classes.
179-
* @return SDK V1 credentials
146+
* Was an implementation of the v1 refresh; now just
147+
* a no-op.
180148
*/
181-
public AWSCredentials getCredentials() {
182-
AwsCredentials credentials = resolveCredentials();
183-
if (credentials instanceof AwsSessionCredentials) {
184-
return new BasicSessionCredentials(credentials.accessKeyId(),
185-
credentials.secretAccessKey(),
186-
((AwsSessionCredentials) credentials).sessionToken());
187-
} else {
188-
return new BasicAWSCredentials(credentials.accessKeyId(), credentials.secretAccessKey());
189-
}
149+
@Deprecated
150+
public void refresh() {
190151
}
191152

192153
/**
@@ -257,8 +218,7 @@ public AwsCredentials resolveCredentials() {
257218
*
258219
* @return providers
259220
*/
260-
@VisibleForTesting
261-
List<AwsCredentialsProvider> getProviders() {
221+
public List<AwsCredentialsProvider> getProviders() {
262222
return providers;
263223
}
264224

@@ -290,9 +250,11 @@ public String listProviderNames() {
290250
*/
291251
@Override
292252
public String toString() {
293-
return "AWSCredentialProviderList[" +
294-
name +
295-
"refcount= " + refCount.get() + ": [" +
253+
return "AWSCredentialProviderList"
254+
+ " name=" + name
255+
+ "; refcount= " + refCount.get()
256+
+ "; size="+ providers.size()
257+
+ ": [" +
296258
StringUtils.join(providers, ", ") + ']'
297259
+ (lastProvider != null ? (" last provider: " + lastProvider) : "");
298260
}

‎hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -738,11 +738,19 @@ private Constants() {
738738
public static final String STREAM_READ_GAUGE_INPUT_POLICY =
739739
"stream_read_gauge_input_policy";
740740

741+
/**
742+
* S3 Client Factory implementation class: {@value}.
743+
* Unstable and incompatible between v1 and v2 SDK versions.
744+
*/
741745
@InterfaceAudience.Private
742746
@InterfaceStability.Unstable
743747
public static final String S3_CLIENT_FACTORY_IMPL =
744748
"fs.s3a.s3.client.factory.impl";
745749

750+
/**
751+
* Default factory:
752+
* {@code org.apache.hadoop.fs.s3a.DefaultS3ClientFactory}.
753+
*/
746754
@InterfaceAudience.Private
747755
@InterfaceStability.Unstable
748756
public static final Class<? extends S3ClientFactory>

0 commit comments

Comments
 (0)