Skip to content

HADOOP-17649. Update wildfly openssl to 2.2.1.Final #4539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from
Open

HADOOP-17649. Update wildfly openssl to 2.2.1.Final #4539

wants to merge 2 commits into from
+8 −2

Conversation

@hotcodemacha
Copy link
Contributor

@hotcodemacha hotcodemacha commented Jul 9, 2022 •
edited
Loading

Description of PR

Update wildfly openssl to 2.2.1.Final.

https://nvd.nist.gov/vuln/detail/CVE-2020-25644

A memory leak flaw was found in WildFly OpenSSL in versions prior to 1.1.3.Final, where it removes an HTTP session. It may allow the attacker to cause OOM leading to a denial of service. The highest threat from this vulnerability is to system availability.

JIRA - HADOOP-17649

How was this patch tested?

  1. CI/Build.

  2. After setting this property

    <property>
      <name>fs.s3a.ssl.channel.mode</name>
      <value>openssl</value>
    </property>

Added test results for hadoop-aws integration tests

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hotcodemacha
Copy link
Contributor Author

hotcodemacha commented Jul 9, 2022 •
edited
Loading 

[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestPaths
[INFO] Running org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.426 s - in org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 s - in org.apache.hadoop.fs.s3a.commit.staging.TestPaths
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestHeaderProcessing
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestCreateFileBuilder
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.38 s - in org.apache.hadoop.fs.s3a.impl.TestHeaderProcessing
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.46 s - in org.apache.hadoop.fs.s3a.impl.TestCreateFileBuilder
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestS3AMultipartUploaderSupport
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.367 s - in org.apache.hadoop.fs.s3a.impl.TestS3AMultipartUploaderSupport
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestRequestFactory
[INFO] Running org.apache.hadoop.fs.s3a.impl.TestDirectoryMarkerPolicy
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 s - in org.apache.hadoop.fs.s3a.impl.TestRequestFactory
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.602 s - in org.apache.hadoop.fs.s3a.impl.TestDirectoryMarkerPolicy
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.377 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter
88665a380593:hadoop-aws ashugpt$ 
88665a380593:hadoop-aws ashugpt$ 
88665a380593:hadoop-aws ashugpt$ tail -f integration_test_response_hadoop_trunk.out
[INFO] Running org.apache.hadoop.fs.s3a.audit.TestLoggingAuditor
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.039 s - in org.apache.hadoop.fs.s3a.audit.TestHttpReferrerAuditHeader
[INFO] Running org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AUnbuffer
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.259 s - in org.apache.hadoop.fs.s3a.audit.TestLoggingAuditor
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.062 s - in org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestMetastoreChecking
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.494 s - in org.apache.hadoop.fs.s3a.s3guard.TestMetastoreChecking
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 s - in org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.172 s - in org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.083 s - in org.apache.hadoop.fs.s3a.TestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.TestDataBlocks
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.822 s - in org.apache.hadoop.fs.s3a.TestDataBlocks
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.95 s - in org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream
[INFO] Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.433 s - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
[INFO] Running org.apache.hadoop.fs.s3a.TestArnResource
[INFO] Running org.apache.hadoop.fs.s3native.TestS3xLoginHelper
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.114 s - in org.apache.hadoop.fs.s3a.TestArnResource
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.764 s - in org.apache.hadoop.fs.s3native.TestS3xLoginHelper
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.404 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.454 s - in org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.804 s - in org.apache.hadoop.fs.s3a.audit.impl.TestActiveAuditManagerThreadLeakage
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.458 s - in org.apache.hadoop.fs.s3a.TestS3AInputStreamRetry
[INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.71 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.766 s - in org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 406, Failures: 0, Errors: 0, Skipped: 4
[INFO] 
[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ hadoop-aws ---
[INFO] Building jar: /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/hadoop-aws-3.4.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:copy-dependencies (default) @ hadoop-aws ---
...
...
...
...
[INFO] 
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ hadoop-aws ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default-integration-test) @ hadoop-aws ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.391 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 17, Time elapsed: 55.794 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[WARNING] Tests run: 16, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 106.574 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.799 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 116.207 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.144 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Running org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 23.567 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.191 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.746 s - in org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.099 s - in org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.225 s - in org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.903 s - in org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.009 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.479 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Tests run: 40, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 248.811 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Running org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 335.192 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.47 s - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[ERROR] Tests run: 14, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 213.419 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[ERROR] testSTS(org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials)  Time elapsed: 11.626 s  <<< ERROR!
com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
        at org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials.testSTS(ITestS3ATemporaryCredentials.java:130)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[ERROR] Tests run: 12, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 124.126 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[ERROR] test_200_execute[directory](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)  Time elapsed: 20.406 s  <<< FAILURE!
java.lang.AssertionError: 
Job job_1657332805778_0001 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-directory: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3445)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2336)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2355)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3443)
        at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:74)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 14 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 35 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[ERROR] test_200_execute[partitioned](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)  Time elapsed: 16.402 s  <<< FAILURE!
java.lang.AssertionError: 
Job job_1657332805778_0002 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-partitioned/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
        at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
        at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
        at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.setupJob(StagingCommitter.java:430)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 31 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)  Time elapsed: 18.506 s  <<< FAILURE!
java.lang.AssertionError: 
Job job_1657332805778_0003 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-magic/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
        at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
        at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
        at org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.setupJob(MagicS3GuardCommitter.java:104)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 31 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:295)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.702 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.838 s - in org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.266 s - in org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[WARNING] Tests run: 26, Failures: 0, Errors: 0, Skipped: 26, Time elapsed: 57.353 s - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.312 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.384 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.199 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[WARNING] Tests run: 22, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 286.83 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 306.805 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.55 s - in org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 322.362 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[WARNING] Tests run: 72, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 525.202 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.726 s - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.461 s - in org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.763 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.936 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.811 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.822 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.534 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 22.98 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 8.885 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.191 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.869 s - in org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.08 s - in org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 28.149 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.733 s - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.137 s - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.639 s - in org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.782 s - in org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.694 s - in org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.347 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 23.265 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 36.966 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 223.452 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.769 s - in org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[ERROR] Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 56.217 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[ERROR] testDelegationBindingMismatch2(org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem)  Time elapsed: 3.096 s  <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
        at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationIT.newS3AInstance(AbstractDelegationIT.java:103)
        at org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem.testDelegationBindingMismatch2(ITestSessionDelegationInFileystem.java:511)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
        ... 19 more

[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 2, Time elapsed: 28.99 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[ERROR] testCommonCrawlLookup[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)  Time elapsed: 0.107 s  <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3563)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3668)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3615)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:556)
        at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
        at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:189)
        at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:100)
        at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.setup(ITestDelegatedMRJob.java:182)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
        ... 26 more

[ERROR] testJobSubmissionCollectsTokens[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)  Time elapsed: 0.099 s  <<< ERROR!
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:857)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:522)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3563)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3668)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3615)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:556)
        at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
        at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:189)
        at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:100)
        at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.setup(ITestDelegatedMRJob.java:182)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.delegation.DelegationTokenIOException: no credentials in configuration or environment variables:  No AWS credentials
        at org.apache.hadoop.fs.s3a.auth.MarshalledCredentials.validate(MarshalledCredentials.java:336)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.loadAWSCredentials(FullCredentialsTokenBinding.java:107)
        at org.apache.hadoop.fs.s3a.auth.delegation.FullCredentialsTokenBinding.deployUnbonded(FullCredentialsTokenBinding.java:120)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.deployUnbonded(S3ADelegationTokens.java:251)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.bindToAnyDelegationToken(S3ADelegationTokens.java:284)
        at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.serviceStart(S3ADelegationTokens.java:205)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
        ... 26 more

[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[WARNING] Tests run: 25, Failures: 0, Errors: 0, Skipped: 25, Time elapsed: 76.937 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.447 s - in org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 s - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.658 s - in org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.433 s - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 57.47 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.659 s - in org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.127 s - in org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.379 s - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.832 s - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.934 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.826 s - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.43 s - in org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 199.136 s - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.782 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.541 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.733 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.126 s - in org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.025 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 37.833 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.247 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.148 s - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.533 s - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.617 s - in org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.757 s - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 594.266 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.064 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.442 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 43.857 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.704 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.862 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.54 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.96 s - in org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.41 s - in org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 213.801 s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.286 s - in org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.433 s - in org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.175 s - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.026 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.142 s - in org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 8.87 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 283.275 s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.511 s - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 703.526 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 386.416 s - in org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 314.899 s - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 475.876 s - in org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0001 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-directory: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3445)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2336)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2355)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3443)
        at org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter.setupJob(DirectoryStagingCommitter.java:74)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 14 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 35 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR]   ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0003 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-magic/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
        at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
        at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
        at org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter.setupJob(MagicS3GuardCommitter.java:104)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 31 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR]   ITestS3ACommitterMRJob.test_200_execute:295->Assert.fail:89 Job job_1657332805778_0002 failed in state FAILED with cause Job setup failed : java.nio.file.AccessDeniedException: s3a://ashugpt-emr-dev-blr-itest/fork-0008/test/ITestS3ACommitterMRJob-execute-partitioned/_SUCCESS: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)
        at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3565)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3468)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:3070)
        at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.deleteSuccessMarker(CommitOperations.java:455)
        at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.setupJob(AbstractS3ACommitter.java:600)
        at org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.setupJob(StagingCommitter.java:430)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:255)
        at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:235)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6428)
        at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6401)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5438)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2506)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2494)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2474)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3540)
        ... 10 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
        at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
        ... 31 more

.
Consult logs under /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/test-dir/8/yarn-2022-07-09-03.13.21.65/yarn-13195147
[ERROR] Errors: 
[ERROR]   ITestS3ATemporaryCredentials.testSTS:130 » SdkClient Unable to find a region v...
[ERROR]   ITestDelegatedMRJob.setup:182->AbstractS3ATestBase.setup:100->AbstractFSContractTestBase.setup:189 » ServiceState
[ERROR]   ITestDelegatedMRJob.setup:182->AbstractS3ATestBase.setup:100->AbstractFSContractTestBase.setup:189 » ServiceState
[ERROR]   ITestSessionDelegationInFileystem.testDelegationBindingMismatch2:511->AbstractDelegationIT.newS3AInstance:103 » ServiceState
[INFO] 
[ERROR] Tests run: 1128, Failures: 3, Errors: 4, Skipped: 190
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (sequential-integration-tests) @ hadoop-aws ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.939 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.553 s - in org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.881 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed: 27.711 s - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.282 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.563 s - in org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.987 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.918 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.195 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.788 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 16.748 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 17.04 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 197.267 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 124, Failures: 0, Errors: 0, Skipped: 84
[INFO] 
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @ hadoop-aws ---
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (default-integration-test) @ hadoop-aws ---
[ERROR] There are test failures.

Please refer to /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M1:verify (sequential-integration-tests) @ hadoop-aws ---
[ERROR] There are test failures.

Please refer to /Users/ashugpt/Documents/OSS/hadoop/hadoop-tools/hadoop-aws/target/failsafe-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29:09 min
[INFO] Finished at: 2022-07-09T03:36:38+01:00
[INFO] ------------------------------------------------------------------------

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 18s Maven dependency ordering for branch
+1 💚 mvninstall 28m 16s trunk passed
+1 💚 compile 24m 53s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 21m 40s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 mvnsite 19m 44s trunk passed
+1 💚 javadoc 8m 31s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 7m 32s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 38m 41s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 25m 46s the patch passed
+1 💚 compile 24m 26s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 24m 26s the patch passed
+1 💚 compile 21m 40s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 21m 40s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 19m 15s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 8m 29s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 7m 26s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 40m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 1042m 48s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 2m 14s The patch does not generate ASF License warnings.
1328m 32s
Reason Tests
Failed junit tests hadoop.security.ssl.TestDelegatingSSLSocketFactory
hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/artifact/out/Dockerfile
GITHUB PR #4539
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux 1866eb1dbc20 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / cacf145
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/testReport/
Max. process+thread count 2731 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/1/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hotcodemacha
Copy link
Contributor Author

@steveloughran - I saw your comments on this PR - #3029 about running the integration tests for

  1. hadoop-aws (Attached results in the PR)
  2. hadoop-azure - I dont have account with Azure, Can you suggest someone in community who can helop in running hadoop-azure tests for this upgrade?

Let me know if you have any comments for this upgrade.

@steveloughran steveloughran changed the title HADOOP-17649.Update wildfly openssl to 2.2.1.Final HADOOP-17649. Update wildfly openssl to 2.2.1.Final Jul 13, 2022
@hotcodemacha
Copy link
Contributor Author

hotcodemacha commented Aug 1, 2022 •
edited
Loading

@steveloughran - Can you please help with review this PR? Thanks.

@steveloughran
Copy link
Contributor

sort out some azure creds. good to play with the opposition anyway

@steveloughran
Copy link
Contributor

also, be good to know what linux version you are testing on. i will have to test this on a vm before merging too, because the openssl native integration has been brittle in the past

@hotcodemacha
Copy link
Contributor Author

I am testing this on:

System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0

@steveloughran
Copy link
Contributor

ok. i will do an (ec2) linux run later. that or fix my pi400 up for testing, which might be fun anyway

@steveloughran
Copy link
Contributor

catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?

@hotcodemacha
Copy link
Contributor Author

catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?

Hi @steveloughran , sorry got busy with other few important stuff earlier. I will rebase it.

Thanks for reminder.

@hotcodemacha
Copy link
Contributor Author

catching up on this. its' too late for 3.3.5, but we can get it into branch-3.3. for the successor. can you rebase?

@steveloughran , I have rebased. Please help in review.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 1s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 3s Maven dependency ordering for branch
+1 💚 mvninstall 33m 55s trunk passed
+1 💚 compile 25m 23s trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 compile 21m 54s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 mvnsite 20m 10s trunk passed
-1 ❌ javadoc 1m 22s /branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 7m 20s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 shadedclient 34m 2s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 34m 19s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 31m 8s the patch passed
+1 💚 compile 24m 40s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 javac 24m 40s the patch passed
+1 💚 compile 21m 35s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 javac 21m 35s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 19m 49s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
-1 ❌ javadoc 1m 11s /patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt root in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 7m 25s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 shadedclient 34m 58s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 785m 11s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 6s The patch does not generate ASF License warnings.
1073m 7s
Reason Tests
Failed junit tests hadoop.security.ssl.TestDelegatingSSLSocketFactory
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.TestDFSInotifyEventInputStream
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/artifact/out/Dockerfile
GITHUB PR #4539
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux b35af670a7ef 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 217ff94
Default Java Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/testReport/
Max. process+thread count 2095 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4539/2/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

revisting this. we need

  • a rebase
  • full test of hadoop-azure abfs and aws s3a tests. these are the places which will break

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hotcodemacha @hadoop-yetus @steveloughran