Introduce an option to add object storage prefix to table locations

[eric-maynard]

Problem

Currently, Polaris enforces that the physical layout of entities maps to the logical layout:

catalog
└── ns1
    ├── ns2
    │   └── table_b
    └── table_a

In the above example, the base locations of table_a and ns2 are expected to be children of ns1, and the location of table_b is expected to be a child of ns2.

This behavior is controlled by ALLOW_UNSTRUCTURED_TABLE_LOCATION and is the basis for the sibling overlap check when OPTIMIZED_SIBLING_CHECK is disabled or persistence cannot support the optimized check.

However, some users have reported that this physical organization of data can lead to undesirable performance characteristics when hotspotting occurs across namespaces. If the underlying storage is range partitioned by key, this organization will tend to physically collocate logically-similar entities.

Solution

To solve this problem, this PR introduces a new option DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED which alters the behavior of the catalog when creating a table without a user-specified location. With the feature disabled, a table such as ns1.table_a will have a path like this:

s3://catalog/base/ns1/table_a/

With the feature enabled, a prefix is added before the namespace:

s3://catalog/base/0010/0101/0110/10010100/ns1/table_a/

This serves to eliminate the physical collocation of tables in the same namespace (or with similarly-named namespaces or table names).

This functionality is similar to Iceberg's write.object-storage.enabled, but it applies across tables and namespaces. The two features can and should be combined to achieve the best distribution of data files throughout the key space.

Configuration & Sibling Overlap Check

If an admin doesn't care about the risk of vending credentials with the sibling overlap check disabled, they can enable the feature with these configs:

polaris.features.DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED=true
polaris.features.ALLOW_UNSTRUCTURED_TABLE_LOCATION=true
polaris.features.ALLOW_TABLE_LOCATION_OVERLAP=true
polaris.behavior-changes.VALIDATE_VIEW_LOCATION_OVERLAP=false

In order to use this feature and to preserve the sibling overlap check, you can configure the service with:

polaris.features.DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED=true
polaris.features.ALLOW_UNSTRUCTURED_TABLE_LOCATION=true
polaris.features.OPTIMIZED_SIBLING_CHECK=true

However, note that the OPTIMIZED_SIBLING_CHECK comes with some caveats as outlined in its description. Namely, it currently only works with some persistence implementations and it requires all location-based entities to have a recently-introduced field set. These locations are expected to be suffixed with /, and locations with many / may not be eligible for the optimized check.

Older Polaris deployments may not meet these requirements without a migration or backfill. Accordingly combining these two features should be considered experimental for the time being.

[MonkeyCanCode]

Nice work on this piece. The approach looks good. Here are my two cents:

1. I think the word "random" may be a bit misleading, as the underlying hash function murmur3_32_fixed is deterministic.
2. Could murmur3_32_fixed introduce hash collisions, particularly within the 20 bits used for the prefix? For example, if we have two logically distinct tables, say namespaceA.tableX and namespaceB.tableX, and their full table identifiers ("namespaceA.tableX" and "namespaceB.tableX") happen to generate the same 20-bit hash prefix, this would lead to their data being co-located under the same physical S3 directory for that prefix. While the full S3 path would still be unique due to the appended namespace/table name, is this a concern for the optimized sibling check or other aspects of data management?

[eric-maynard]

Good point @MonkeyCanCode re: the name. A previous version had a truly random prefix, but now it's deterministic based on a hash of the table identifier. I don't actually have a preference between the two or think it matters. Do you have a suggestion for a better name, though? Iceberg calls their similar feature write.object-storage.enabled, which I think is quite misleading.

Re (2), you're right to consider the risk of hash collisions esp. when we only use 20 bits. In fact, collisions are inevitable as you approach 1M identifiers. However MurmurHash exhibits strong avalanche properties, meaning even small differences in the identifier (like the A vs B) produce vastly different outputs. So while collisions are possible, the chance that structurally similar identifiers (e.g., namespaceA.tableX vs. namespaceB.tableX) collide in the 20-bit prefix is still low.

As for uniqueness, since the full path still includes the actual identifier, there's no ambiguity or risk of overwriting. The sibling check looks at the full path.

[MonkeyCanCode]

Good point @MonkeyCanCode re: the name. A previous version had a truly random prefix, but now it's deterministic based on a hash of the table identifier. I don't actually have a preference between the two or think it matters. Do you have a suggestion for a better name, though? Iceberg calls their similar feature write.object-storage.enabled, which I think is quite misleading.

Re (2), you're right to consider the risk of hash collisions esp. when we only use 20 bits. In fact, collisions are inevitable as you approach 1M identifiers. However MurmurHash exhibits strong avalanche properties, meaning even small differences in the identifier (like the A vs B) produce vastly different outputs. So while collisions are possible, the chance that structurally similar identifiers (e.g., namespaceA.tableX vs. namespaceB.tableX) collide in the 20-bit prefix is still low.

As for uniqueness, since the full path still includes the actual identifier, there's no ambiguity or risk of overwriting. The sibling check looks at the full path.

Thanks for the confirmation.

[eric-maynard]

But I really am open to changing the name @MonkeyCanCode -- I toyed with DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED but wasn't sure about it. Let me know if you have any ideas.

[MonkeyCanCode]

But I really am open to changing the name @MonkeyCanCode -- I toyed with DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED but wasn't sure about it. Let me know if you have any ideas.

Yes, I think that is good. The one used by Iceberg is generic and not mentioned about random I think (as earlier they were using some random hash function as well...based on the code context, this one is derived from that, so I am assuming they changed to this hash as well).

[dimas-b]

Why is it "DEFAULT"? It was not default in Polaris 🤔

[dimas-b]

In Iceberg code this method of computing table locations is apparently called an "object store" location provider 🤔

[MonkeyCanCode]

In Iceberg code this method of computing table locations is apparently called an "object store" location provider 🤔

Yes, it is only for the object store as they are flat and use those hashes for different compute on the backend to handle based on prefix (not a problem nor needed is backend is block storage)

[eric-maynard]

Ah, yeah, the naming is really tough here. It's DEFAULT_LOCATION ... because it alters the "default location" given to a table if there is no user-specified location.

The Iceberg feature doesn't have this component because it applies within the user-specified table location, whereas the Polaris feature applies above the table location only if there isn't a user-specified location.

[dimas-b]

How about OBJECT_STORAGE_LOCATION_PREFIX_ENABLED?

[eric-maynard]

But it's not always applied like the Iceberg option, it's only applied when using the "default location" for a table

[dimas-b]

ok