[FEATURE REQUEST] add support for random prefixes (MURMUR3 S3 hash)

[TomerHeber]

Is your feature request related to a problem? Please describe.

AWS S3 has hard limits for each prefix index.
The best way to scale with S3 is to create a random prefix for each object/file.

This is described here:
https://aws.amazon.com/blogs/big-data/improve-operational-efficiencies-of-apache-iceberg-tables-built-on-amazon-s3-data-lakes/ (write.object-storage.enabled'=true).

Setting this option in Polaris, returns a "temporarily unsupported" error.

Describe the solution you'd like

Below is an example:

bucket//catalog/namespace/table/file1
bucket//catalog/namespace/table/file2
etc...

This strategy removes any hard limits concerns with S3 buckets.

Describe alternatives you've considered

For each catalog generate set random location prefix:
bucket/<random_location_prefix>/namespace/table

Additional context

The current limitation may be due to RBAC concerns.
However, I could be wrong, but the desired solution may still be able support RBAC as required.

[sfc-gh-ygu]

It's pretty hard to support pattern bucket/<random_location_prefix>/namespace/table as s3 policy used by RBAC depends on the path prefix(I might be wrong). Although, I think it is possible to support the following randomization prefix pattern.
bucket/namespace/table/data/<random_location_prefix>

[TomerHeber]

@sfc-gh-ygu - thank you for the response.

I would argue (and of course depends on the exact implementation details) that this should be possible (in theory).

Let's assume the catalog generates MURMUR3 hash (8 hex characters).
Wouldn't the catalog be able to ignore the first 8 characters and '/'.

bucket/<random_location_prefix>/namespace/table is not going to work - agreed.

but how about:
bucket/<random_prefix>/catalog/namespace/table

I see no reason why RBAC cannot be enforced with this approach.

Please let me know what I'm missing here.

Thanks!

[sfc-gh-ygu]

@TomerHeber , it has two implications:

1. The credential vending needs a fixed prefix. The random prefix (no matter if the length is fixed or not) will post a security risk for credential vending.
2. It is a challenge for the table maintenance job like RemoveOrphanfile.

Instead, we can put the the hash string after the table location, like the following example shows. This may be not as effective as the one you proposed in terms of path randomness, but it's still effective per our experience.

bucket/namespace/table/<hash_string>