Feature Request: Access AWS S3 via IRSA

[putnam120]

From the documentation it appears that in order to access AWS S3 from a Kubernetes cluster you need to provide access and secret keys. However, when working with an EKS cluster, or any cluster that has enabled AWS IAM Roles as Service Accounts (IRSA) it would be better to use the service account instead of having to provide credentials.

[nitisht]

Hey @putnam120 thanks for reporting, we recently added support for this. You can now set an env var with the AWS profile name so Parseable can authenticate with S3 using that profile: https://www.parseable.io/docs/env-vars#applicable-to-aws

We'll improve docs further so this is more clear. But just to make sure, this is what you are looking for, right?

[putnam120]

I'm not fully aware of how profiles work in this situation. From my experience a profile is usually specified in the ~/.aws/credentials file and has the form

[default]
aws_access_key_id = ...
aws_secret_access_key = ...

[anotherProfileName]
aws_access_key_id = ...
aws_secret_Access_key = ...

This still requires the user to specify the access and secret keys. For IRSA I was thinking more along the lines of the ability to link a Kubernetes Service account to an IAM Role through the addition of an annotation.

apiVersion: v1
kind: ServiceAccount
metadata:
    name: my-sa
    namespace: my-namespace
    annotations:
        eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role

Then you specify for your Pod/Deployment/StatefulSet to use this ServiceAccount and it will have permissions to interact with AWS as specified by the IAM Role.

I saw that there was an older issue which added support for profiles and it looked to be using the IAM role that was attached to an EC2 instance.

The reason I'm asking for this feature is so if the Parseable configurations are stored in a git repo they won't include your AWS credentials. And only an IAM role name which is much less sensitive.

[nitisht]

Thanks @putnam120, I believe this should be possible, but we need to try and document this in detail. Let me get back on this.

[nitisht]

@putnam120 I have made changes for this, I could use some help in testing this on a like EKS environment. Would you be able to run a Docker image + Helm to try if everything works as expected?

[putnam120]

@nitisht Yeah I can definitely give it try and let you know how things go. I should have some time over the weekend to take a look.

[nitisht]

Thanks, i’ll get back

[putnam120]

Hi @nitisht,

I attempted to deploy to an EKS cluster and use the changes you made for IRSA but hit an issue.
The pod for the deployment hit a CrashLoopBackOff, and its logs said

error: the following required arguments were not provided:
  --access-key-id <access-key>
  --secret-key <secret-key>

Here are the steps that I took for the deployment:

• checkout the main branch of the git repo and build the image.
• Push the image to my container repository.
• Created the parseable-env-secret using the file contents described below.
• Created a values.yaml file as described below.
• Obtained all the k8s yaml configurations by running helm template parseable parseable/parseable -n parseable -f values.yaml > resources.yaml.
• Updated the image used by the deployment to the one that I previously pushed to my container repository.
• Used kubectl to apply the secret and then the resources.yaml file.

parseable-env-secret file

s3.url=https://s3.amazonaws.com
s3.region=us-east-1
s3.bucket=my-bucket
addr=0.0.0.0:8000
staging.dir=./staging
fs.dir=./data
username=admin
password=admin

values.yaml file

parseable:
  metrics:
    serviceMonitor:
      enabled: false

  s3ModeSecret:
    - type: env
      name: parseable-env-secret
      prefix: P_
      keys:
          - addr
          - username
          - password
          - staging.dir
          - fs.dir
          - s3.url
          - s3.bucket
          - s3.region

IN this file I removed the keys for the access and secret keys as they should not be needed once using IRSA. Let me know if I'm mistaken on that point. But if they are used then from my understanding of the AWS SDK will first use the environment variables and later the web identity token (https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) so putting dummy values here isn't possible.

[nitisht]

Thanks for the detailed overview @putnam120 , I agree server should not expect S3 access and secret key always. Let me make these changes and publish a test image. We can then try out with that image.

[nitisht]

@putnam120 can you please try again with the deployment image set to trueleo8/parseable:test-aws. This change allows server to start even if S3 creds are not provided.