Performing any transactions in MinIO server like creating tables throws "Unable to process: Failed to get subscoped credentials"

[mthirani2021]

Describe the bug

Trying to use/ configure MinIO server for storing all my tables/ metadata file for Polaris Catalogs.
I spinned up Polaris at my local setting my default base location to MinIO bucket and used a dummy ARN (not sure if it has to be dummy btw). Any transaction related API like creating tables throws the below error:

Unable to process: Failed to get subscoped credentials: User: arn:aws:iam::<aws_account>:user/mayank.thirani is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<aws_account>:role/dummy_arn (Service: Sts, Status Code: 403, Request ID: 6c3d0a67-3d94-4c55-b4ae-9372b7046669)

NOTE: aws_account is the real AWS account but dummy_arn doesn't exists. Trying to by pass it for MinIO like storage providers.

To Reproduce

1. All my access keys at local env are set to MinIO based access/ secret keys. This is how I spinned up Polaris:
   ./polaris --client-id ${CLIENT_ID} --client-secret ${CLIENT_SECRET} catalogs create --storage-type s3 --role-arn arn:aws:iam::<aws_account>:role/dummy_arn --default-base-location s3://minio-bucket default
2. Creating a table using the API mentioned per https://polaris.io/#tag/Catalog-API/operation/createTable, which throws the error mentioned in description

Actual Behavior

System throws the error as mentioned in description

Expected Behavior

System should allow to create tables in MinIO (S3 compatible like) storage providers.

Additional context

No response

System information

OS: MacOS
Latest OSS Polaris pulled from repo
Stortage: MinIO

[mthirani2021]

@dimas-b FYI

[dimas-b]

It looks like the deeper issue here is that there's no way to configure non-default S3 endpoints for a catalog.

[omarsmak]

Looks this will be fixed via #389

[dvaldivia]

If you can use credentials for polaris to access MinIO, I think that would fix your issue, but if you need STS, then polaris needs to support setting MinIO as the custom STS endpoint as well

[mthirani2021]

@dvaldivia Do you mean right IAM role (instead of dummy ones) in polaris to access MinIO ?

[george-zubrienko]

We current run the pre-0.9 build on MinIO, you need to set two additional env vars:

AWS_ENDPOINT_URL_S3 : https://myminio-s3.mydomain.com
AWS_ENDPOINT_URL_STS : https://myminio-s3.mydomain.com

in additional to setting global aws access key/secret this works.

[mthirani2021]

@george-zubrienko I tried running the Polaris with right ARN role(not a dummy one)/ external id and setting the base location to my minio-bucket location as stated in the description and had my global aws creds set in the env along with those two additional endpoint URL set. Receiving below error:

(org.apache.iceberg.exceptions.RESTException) Unable to process: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Cod│
e: 301,..)

[george-zubrienko]

@george-zubrienko I tried running the Polaris with right ARN role(not a dummy one)/ external id and setting the base location to my minio-bucket location as stated in the description and had my global aws creds set in the env along with those two additional endpoint URL set. Receiving below error:

(org.apache.iceberg.exceptions.RESTException) Unable to process: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Cod│
e: 301,..)

MinIO doesn't have a concept of IAM role/external id. As I wrote, you just override STS endpoint globally and put a dummy role in catalog creation. I just updated to 0.9 and it works as well.
I can show how we do it right now (still waiting for S3Compat PR to be merged) - can connect via Slack some time this week. Alternatively, I can share build and HCL helm_release scripts

[george-zubrienko]

Btw, this is how I add catalogs:

PRINCIPAL_TOKEN="..."
curl -i -X POST -H "Authorization: Bearer $PRINCIPAL_TOKEN" -H 'Accept: application/json' -H 'Content-Type: application/json'   http://localhost:8181/api/management/v1/catalogs   -d '{
        "catalog": {
          "name": "staging",
          "type": "INTERNAL",
          "readOnly": false,
          "properties": {
            "default-base-location": "s3://tmp/production/staging/"
          },
          "storageConfigInfo": {
            "storageType": "S3",
            "allowedLocations": [
              "s3://tmp/production/staging/"
            ],
            "roleArn": "arn:aws:iam::000000000000:role/polaris-access-role"
          }
        }
      }'

roleArn is a valid, but non-existing role.

[mthirani2021]

@george-zubrienko Hmm, even having the dummy role and setting those env vars, I got the same error. FYI, this is my Polaris CLI to create a default catalog, which works fine:
./polaris --client-id ${CLIENT_ID} --client-secret ${CLIENT_SECRET} catalogs create --storage-type s3 --role-arn arn:aws:iam::000000000000:role/polaris-access-role --default-base-location s3://minio-bucket default

But even the simple create table throws the same error as stated in description. Happy to connect in slack.
Did you try to create a simple table, after your catalog gets created, which is the source of the error?

[george-zubrienko]

@george-zubrienko Hmm, even having the dummy role and setting those env vars, I got the same error. FYI, this is my Polaris CLI to create a default catalog, which works fine: ./polaris --client-id ${CLIENT_ID} --client-secret ${CLIENT_SECRET} catalogs create --storage-type s3 --role-arn arn:aws:iam::000000000000:role/polaris-access-role --default-base-location s3://minio-bucket default

But even the simple create table throws the same error as stated in description. Happy to connect in slack. Did you try to create a simple table, after your catalog gets created, which is the source of the error?

We have over 300 tables there, so yeah, I tried all of it, and it works. However, I do remember struggling with STS a bit. I collected all the necessary info below:

First, make sure your polaris user (we use local user for this) has similar permissions (note explicit STS permission):

Second, we have these set on polaris server container (note username/secret are used, not access key):

AWS_ACCESS_KEY_ID : <minio-user-name>
AWS_ENDPOINT_URL_S3 : https://our-polaris-url
AWS_ENDPOINT_URL_STS : https://our-polaris-url
AWS_REGION : us-east-1
AWS_SECRET_ACCESS_KEY : <minio-user-secret>

You can also check this docker-compose setup for a bit older build of Polaris, but the idea is the same. They don't use dropwizard so no need for -Ddw... stuff now :)

You can find me in Slack by using my corp mail gzu@ecco.com - throw me a DM and we can agree on some time to chat, if you still have issues.