Make S3 role_arn optional

[bacek]

Is your feature request related to a problem? Please describe.

We are deploying Polaris into k8s environment, where we provide a ServiceAccount with correct permissions to access bucket/db/etc via IRSI (for AWS) or WebIdentity (for GCP). Having an extra required role_arn for catalog is redundant and forces to generate extra role, with similar permissions just to be able to configure it. Also, role_arn is not needed for minio.

Describe the solution you'd like

Make role_arn optional. Use AWS DefaultCredentialsProvider for getting credentials.

Describe alternatives you've considered

No response

Additional context

No response

[snazy]

Sounds like a legit ask. A role-ARN shouldn't be required. Other S3 implementations (Minio, Ceph, many of the appliances) also do not all require a role-ARN.

[dennishuo]

It seems like there are three distinct "new" use cases:

1. Using DefaultCredentialsProvider directly without subscoping to access storage when running on AWS and using AWS S3
2. Using DefaultCredentialsProvider directly without subscoping to access non-AWS s3-compat storage
3. Using DefaultCredentialsProvider directly with subscoping to access non-AWS s3-compat storage

These are all different from the "normal" flow:

4. Using DefaultCredentialsProvider as the super-root to assumeRole on a provided role with subscoping to access storage on S3

For (1) and (2), setting SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true is explicitly intended for that use case, though looking at the code it seems we still need to remove "validate" checks for roleARN, otherwise parsing-validation fails at createCatalog time.

@bacek any chance you could verify whether the following in your application.properties combined with providing any "dummy" syntactically-valid roleArn such as arn:aws:iam::123456789012:role/my-role already works for you before PR #2325 ?

polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"=true

Looking at MinIO that's certainly very interesting that AssumeRoleWithWebIdentity makes roleArn optional -- it's not 100% clear whether the provide Policy is still applied to the returned token. I'm also not 100% clear on how we map the stsClient to point at WebIdentity vs CustomToken flows for MinIO - for example AssumeRoleWithCustomToken still requires roleArn: https://docs.min.io/enterprise/aistor-object-store/developers/security-token-service/assumerolewithcustomtoken/

But assuming the subscoping does work, then (3) is a substantially new flow where the assumeRole indirection is applied, but yet the identity is the service-wide default credentials provider where SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=false is used despite being no roleArn provided. This new use case would need a separate FeatureConfiguration to avoid multi-tenant deployments from "accidentally" exposing the service identity through vended credentials.