LoadTable returns Access Denied instead of 404 for a missing metadata file in S3

[fivetran-ashokborra]

Describe the bug

Bug:
LoadTable returns Access Denied error instead of 404 for a missing metadata file

Error log:

"logger":"org.apache.polaris.service.catalog.BasePolarisCatalog",
"message":"Access Denied or Forbidden error: User: arn:aws:sts::{account}:assumed-role/{role}/PolarisAwsCredentialsStorageIntegration is not authorized to perform: s3:ListBucket on resource: \"arn:aws:s3:::ashok-test-local\" because no session policy allows the s3:ListBucket action (Service: S3, Status Code: 403, Request ID: W6Q2D563ETEKR6XZ, Extended Request ID: Izq3QS7eZmGjhjfyoxJWMHeCgrFvUlpZjj73JYMO8i/qnKw6CjOaPVgOWVLFr/JsToTeTxO0YaM=)"

To Reproduce

1. Create a catalog, namespace and table
2. Insert some data in the table
3. Go to the table's path in S3 and delete the latest metadata file referenced by the table

Catalog details:

{
    "type": "INTERNAL",
    "type": "INTERNAL",
    "name": "test_catalog",
    "properties": {
        "default-base-location": "s3://ashok-test-local"
    },
    "createTimestamp": 1744800339841,
    "lastUpdateTimestamp": 1744800339841,
    "entityVersion": 1,
    "storageConfigInfo": {
        "storageType": "S3",
        "roleArn": "arn:aws:iam::{account}:role/{role}",
        "externalId": "test_id",
        "userArn": null,
        "storageType": "S3",
        "allowedLocations": [
            "s3://ashok-test-local"
        ]
    }
}

Note: The role had ListBucket permission on the entire bucket, attached IAM policy below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAll",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::ashok-test-local/*",
                "arn:aws:s3:::ashok-test-local"
            ]
        }
    ]
}

Actual Behavior

Throws 403 error instead of 404 for the missing metadata file

Expected Behavior

Should return 404 error with the missing file location

Additional context

Same test case in the context of Azure returns 404

"logger":"org.apache.polaris.service.exception.IcebergExceptionMapper","message":"Unhandled exception returning INTERNAL_SERVER_ERROR","exception":"com.azure.storage.blob.models.BlobStorageException: Status code 404, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.\nRequestId:0e14b080-f01e-0066-1bb2-ae9d8f000000\nTime:2025-04-16T09:36:06.9902103Z</Message></Error>\"\n\tat java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)\n\tat com.azure.core.implementation.MethodHandleReflectiveInvoker.invokeStatic(MethodHandleReflectiveInvoker.java:26)\n\tat com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:53)\n\tat com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:407)\n\tat com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:135)\n\tat

System information

No response