S3 input plugin does not work with IAM role WebIdentity

[mmujat]

Hi,
I'm using IAM role attached to the Service Account with WebIdentity in amazon EKS, but I'm not able to get logstash s3 input plugin working using the service account IAM role and not the node instance role.

As an input configuration we have the following:
s3 {
bucket => "${LOGBUCKET}"
prefix => "elb/xyz-app/"
region => "${AWS_REGION}"
add_field => { "context" => "xyz-app" }
tags => ["elblogs"]
additional_settings => {
"force_path_style" => true
"follow_redirects" => false
}

After removing the node role IAM polices for the S3 bucket we get the following error:

[2022-02-22T10:59:07,836][ERROR][logstash.inputs.s3 ][main][bc7c59a365d681e2cbd0d4cdac2804f2cd6088892917e7a15bccaade4e5cd31f] Unable to list objects in bucket {:exception=>Aws::S3::Errors::AccessDenied, :message=>"Access Denied", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/response_paging.rb:26:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:in send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/request.rb:24:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/operations.rb:139:in all_batches'", "org/jruby/RubyEnumerator.java:396:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/collection.rb:18:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:143:in list_new_files'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:185:in process_files'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:133:in block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in interval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:132:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:409:in inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:400:in block in start_input'"], :prefix=>"elb/xyz-app/"}

The Trust Policy on the Service Account role looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB": "sts.amazonaws.com",
"oidc.eks.eu-central-1.amazonaws.com/id/BABCDEFAAABBB": "system:serviceaccount:logging:logstash"
}
}
}
]
}

And the policies attached to the role:
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::logs-dev",
"arn:aws:s3:::logs-dev/*"
],
"Effect": "Allow",
"Sid": "AllowLogstashToGetS3Logs"
}

We also tried to add the role_arn in the S3 input configuration:
s3 {
bucket => "${LOGBUCKET}"
prefix => "elb/xyz-app/"
region => "${AWS_REGION}"
role_arn => "${SERVICE_ACCOUNT_ROLE_ARN}"
add_field => { "context" => "xyz-app" }
tags => ["elblogs"]
additional_settings => {
"force_path_style" => true
"follow_redirects" => false
}

But we get the the following weird error:
[2022-02-23T12:33:17,299][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Aws::STS::Errors::AccessDenied: User: arn:aws:sts::123456789:assumed-role/eks-dev-NodeInstanceRole-AAAI/i-xyz is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789:role/es-dev-ElasticSearch-LogstashRole-BBB>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:in send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/assume_role_credentials.rb:49:in refresh'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/refreshing_credentials.rb:20:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/assume_role_credentials.rb:40:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-aws-5.0.0/lib/logstash/plugin_mixins/aws_config/v2.rb:67:in assume_role'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-aws-5.0.0/lib/logstash/plugin_mixins/aws_config/v2.rb:17:in aws_options_hash'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:439:in get_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.8.1/lib/logstash/inputs/s3.rb:106:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:390:in start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:315:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/beats.conf", "/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x35938dae run>"}
[2022-02-23T12:33:17,311][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create

, action_result: false", :backtrace=>nil}

Logstash (7.16.3)
logstash-input-s3 plugin (3.8.1)

Has anyone used before the sts:AssumeRoleWithWebIdentity and made it work?
Thank you

[kares]

What and how are the AWS credentials set? I do not see any hints for that in the plugin configuration ..

[mmujat]

What and how are the AWS credentials set? I do not see any hints for that in the plugin configuration ..

I am using a service account token secret. I haven't declared any AWS credentials in the plugin configuration because I assumed the service account token will be used. But it seams it is not.
This works for logstash, I am able to push logs to AWS OpenSearch, but when it comes to getting logs from S3 I get an access denied.

[ottramst]

Experiencing the exact same issues as described above.

Seems like this plugin uses a pretty outdated version of AWS SDK which doesn't support AWS IRSA
According to AWS documentation the SDK version must be >= 3.58.0, while this plugin uses 2.11.632

Source: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html