Add support for S3 request signing #2280

adutra · 2025-08-06T17:26:49Z

Fixes #32.

Design doc:

https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing

singhpk234 · 2025-08-06T23:21:18Z

Thank you so much for this @adutra !

when ready please let us know (would be really helpful if you can share some description to walk us through your thought process), happy to help with reviews :) !

snazy

This looks like good 1st step.

As all signing requests will have to access the backend for every little data file chunk being accessed by clients, this implementation will cause quite a slow experience for users. Until we don't have a faster implementation (signed access-rules), I think we should label this feature as "experimental".

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java

...service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java

snazy · 2025-08-07T08:47:34Z

...service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java

+            .method(method)
+            .headers(signingRequest.headers());
+
+    // FIXME is this correct?


nope.
See ListObjects + ListObjectsV2 REST spec.

Be careful: Not all GET /{bucket/ requests are list-objects requests!
The "v1" list-objects has no required parameter, so you cannot just check for the presence of a req param. It could also be a get-bucket-cors or get-bucket-encryption or ...-policy or ...-website or ......
Nice, no?

Maybe it's okay to assume that every client uses the v2 api, which has a unique, required request param.

...service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java

adutra · 2025-08-08T14:14:55Z

Ready for review. The test failures are unrelated to this change.

\cc @singhpk234 @snazy @dimas-b @metadaddy

adutra · 2025-08-08T14:24:01Z

api/iceberg-aws-sign-service/build.gradle.kts

+    mapOf(
+      "IcebergErrorResponse" to "org.apache.iceberg.rest.responses.ErrorResponse",
+      "S3SignRequest" to "org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest",
+      "SignS3Request200Response" to


FYI The name of the logical response type is a bit weird because it is inlined, instead of being a $ref in the OpanAPI spec.

adutra · 2025-08-08T14:25:44Z

...ests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java

  private String currentCatalogName;
  private Map<String, String> restCatalogConfig;
-  private URI externalCatalogBase;
+  private URI externalCatalogBaseLocation;


FYI I refactored this class a bit, notably to facilitate subclassing, but there is no functional change in this class.

Minor request here, but you could pull this and the new IntegerationTest Helpers out into a separate PR and reduce the size of this PR by a bit. I think that would also be a very fast review

Yup good idea!

#2384

.../src/main/java/org/apache/polaris/service/it/test/PolarisS3RemoteSigningIntegrationTest.java

adutra · 2025-08-08T14:29:04Z

runtime/server/build.gradle.kts

  }

 dependencies {
-  implementation(project(":polaris-core"))


Not directly referenced in this module.

This is another change I would try to pull out to another PR, it's not related to this changeset correct?

Correct:

#2385

adutra · 2025-08-08T14:31:06Z

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

    }
  }

-  private static boolean isStaticFacade(CatalogEntity catalog) {


Refactored to become a method of CatalogEntity.

@RussellSpitzer FYI:

#2386
#2387

Thanks! It definitely helps to keep this sort of refactor out of the larger functional change

adutra · 2025-08-08T14:32:49Z

runtime/service/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java

+   * <p>The returned provider is not meant to be vended directly to clients, but rather used with
+   * STS, unless credential subscoping is disabled.
+   */
+  default AwsCredentialsProvider awsSystemCredentials() {


Renamed to awsSystemCredentials because these are not "just" credentials for STS, they are server credentials.

adutra · 2025-08-08T14:35:30Z

...c/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogAdapter.java

+        prefix,
+        catalog -> {
+          S3SignResponse response = catalog.signS3Request(s3SignRequest, tableIdentifier);
+          return Response.status(Response.Status.OK).entity(response).build();


In theory the response should include Cache-Control headers. This is left for a later improvement.

... but that would require clients to respect those 🤷

adutra · 2025-11-04T17:05:34Z

This is now ready again for review!

\cc @snazy @singhpk234 @flyrain @dimas-b

This PR is equivalent to Milestone 1 in the original design doc. Let's try to work incrementally and get this in, we are already working on getting a new PR for Milestone 2 and especially the authz checks that are not yet implemented. Thanks!

singhpk234 · 2025-11-04T18:04:41Z

Hey Alex,
Thank a ton for all the work !
Is this understanding of M1 and M2 correct ? (take design doc above)

M1 - Simple, slow implementation with basic RBAC
- Not production ready – marked experimental
- Does not implement access checks based on S3 location
- Does not support HTTP proxies
- Corresponds to the current PR: https://github.com/apache/polaris/pull/2280 

M2 - Fast implementation with efficient RBAC
- Production-ready
- Implements [Access checks based on the S3 location](https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?tab=t.0#bookmark=id.hs05149qp3eo) 
- Supports HTTP proxies
- Implements "Nessie style" signed parameters

If yes, I am bit confused, if we know M2 is the one we eventually wanna make and M2 is a bit orthogonal to M1 why are making / investing on M1, a couple questions that would be really helpful to get your take :

How long it will take for us to get M2 post M1 ? (you mentioned M2 is already in works ?)
why are we making M1 if M1 is not production recommended and based on my understanding it will never be.
when both M2 and M1 are there when will a user use M1 over M2 ?
The route for M1 is /s3-sign/v1/{prefix}/namespaces/{namespace}/tables/{table} and for M2 its gonna be
/s3-sign/v1/{prefix}/namespaces/{namespace}/tables/{table}/{signContext} or /s3-sign/v1/{prefix}/namespaces/{namespace}/tables/{table}?signContext={signedContext} checking this commit (please correct me If i am missing something) projectnessie/nessie@2c6b931 it will be earlier ? are we gonna introduce a new endpoint for M2 ?

adutra · 2025-11-05T15:38:48Z

If yes, I am bit confused, if we know M2 is the one we eventually wanna make and M2 is a bit orthogonal to M1 why are making / investing on M1, a couple questions that would be really helpful to get your take :

It's not orthogonal, M2 builds on top of M1.

How long it will take for us to get M2 post M1 ? (you mentioned M2 is already in works ?)

It depends if we want all the features in M2 or not. I would actually personally break M2 into smaller pieces. The most important piece is "Access checks based on the S3 location", which can be done in a few man-days. The "Nessie style" signed parameters would require a bit more than that, imo.

why are we making M1 if M1 is not production recommended and based on my understanding it will never be.

For the same reason we released the Events feature in incomplete state, with "beta" status: to get users feedback, and to work incrementally.

when both M2 and M1 are there when will a user use M1 over M2 ?

These are not "flavors" of remote signing. It's just one same feature, in two levels of maturity M1: experimental, M2: production-ready.

are we gonna introduce a new endpoint for M2 ?

Yes, if we adopt Nessie's strategy for fast remote signing as proposed in the doc (it's optional). Note: this isn't a breaking change, as it's not user facing.

Happy to discuss this topic more in depth in a Slack channel or during the next sync!

dimas-b · 2025-11-06T19:40:48Z

The proposed phased approach (M1/M2) SGTM 👍

dimas-b

Lots of comments 🙂 but the PR overall looks good 👍 Thanks for implementing this, @adutra !

spec/README.md

dimas-b · 2025-11-06T20:00:59Z

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java

                        .addAll(PolarisEndpoints.getSupportedGenericTableEndpoints(realmConfig))
                        .addAll(PolarisEndpoints.getSupportedPolicyEndpoints(realmConfig))
+                        .addAll(
+                            PolarisEndpoints.getSupportedRemoteSigningEndpoints(


I'm not sure it's worth advertising in ConfigResponse because it's not a standard endpoint, for which clients might want to check the endpoints list. Clients learn about it via loadTable responses, I guess 🤔

OK let's remove.

dimas-b · 2025-11-06T20:08:54Z

runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java

+    }
+
+    String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+    URI signerUri = uriInfo.getBaseUri().resolve("api/");


I'm not sure about the api/ constant... It related to our @Path annotations in the generated REST API classes, so it's usage is a bit obscure here. Could we make it a constant in the s3-sign-service module?

I agree, but I'm not sure a constant in s3-sign-service is the best solution. The api/ path segment doesn't seem to be a first-class citizen in our APIs, rather a bi-product. It's only declared as a server variable in the OpenAPI generator config, for example:

polaris/api/iceberg-service/build.gradle.kts

Line 86 in 1f4e748

serverVariables.put("basePath", "api/catalog")

polaris/api/polaris-catalog-service/build.gradle.kts

Line 116 in b6e247d

serverVariables.put("basePath", "api/catalog")

The management APi is actually a bit different, there is not basePath variable in polaris-management-service.yml, so the base path is hard-coded:

polaris/spec/polaris-management-service.yml

Line 26 in 04c8ee6

- url: "{scheme}://{host}/api/management/v1"

But the common characteristic is that api/is the first path segment of all Polaris APIs, but that is not enforced by any constraint.

I would say the right place for this segment would be in PolarisResourcePaths, but you will notice again that none of the paths include the api/ segment. Same for Iceberg's ResourcePaths. So it's really implicit 😅

All of this to say, let's declare a constant in PolarisResourcePaths and call it a day, wdyt?

+1 to PolarisResourcePaths

dimas-b · 2025-11-06T20:15:14Z

polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisResourcePaths.java

        RESTUtil.encodeString(ident.name()));
  }
+
+  public String s3RemoteSigning(TableIdentifier ident) {


Is this a core concern? I suppose it is only relevant to the REST layer 🤔

Well the same could be said for all the methods in this class 😄
The package is org.apache.polaris.core.rest btw - so I guess REST is a core concern 🤷‍♂️

I think PolarisResourcePaths and PolarisEndpoints should not live in core ideally. But that's the way it is today.

oops, I did not actually look at the whole class, only at the diff 😅

Let's keep it for now and maybe refactor later.... In general, I'd like to untangle core from REST 😅

dimas-b · 2025-11-06T20:17:09Z

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

+  AWS_REMOTE_SIGNER_URI(
+      String.class,
+      S3V4RestSignerClient.S3_SIGNER_URI,
+      "the base URI for the remote signer service, used for signing S3 requests",


Does this have a reference Iceberg version number where the property becomes respected?

Iceberg 1.2.0. That's more than 2 years old, I'm not sure we need to mention it here. Wdyt?

That's fine... I was worries it might have been more recent ;)

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

singhpk234 · 2025-11-07T18:40:34Z

Hey Alex,
Thanks for the response, TBH I am still unclear why do we wanna introduce an endpoint for M1 only to introduce a new one in M2 ? How can an public endpoint not be a user facing change ?
Apologies but I am not able to follow, As you suggested, definitely sync will be really helpful to discuss, meanwhile i request you to please bump this in the thread https://lists.apache.org/thread/xpf60ko823c7mlg5bhnddq3btvdsq5s1 too for more eyes, in case I am missing something.

singhpk234 · 2025-11-07T19:13:22Z

runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java

+            identifier, PolarisEntityType.TABLE_LIKE, PolarisEntitySubType.ICEBERG_TABLE, true);
+
+    // If the table doesn't exist, we still need to check allowed locations from the parent
+    // namespace.


[doubt] is this a valid state ?

Yes. When creating a new table, engines write the manifest files first, then commit the table. The manifest writes must be signed too. Thus, a request to sign can arrive at the server before the table is created.

singhpk234 · 2025-11-07T19:21:29Z

CHANGELOG.md

+    2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+       and `TABLE_WRITE_DATA` privileges.


[doubt] is there an assertion when even TABLE_REMOTE_SIGN is granted these two privieldges would be there ?

nit :

Suggested change

2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`

and `TABLE_WRITE_DATA` privileges.

2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The catalog role must also be granted the `TABLE_READ_DATA`

and `TABLE_WRITE_DATA` privileges.

is there an assertion when even TABLE_REMOTE_SIGN is granted these two privieldges would be there ?

Currently no.

singhpk234 · 2025-11-08T01:39:47Z

runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java

+    }
+
+    String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+    URI signerUri = uriInfo.getBaseUriBuilder().path(PolarisResourcePaths.API_PATH_SEGMENT).build();


can we add a TODO in code to handle proxies ?

singhpk234 · 2025-11-08T01:42:33Z

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

+          realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+      boolean credentialSubscopingAllowed =
+          baseCatalog instanceof IcebergCatalog


what if its an IcebergCatalog but doesn't support STS ?

That's a good question!

It's complex to check this at this level because you need to find the AwsStorageConfigurationInfo corresponding to the resolved path in order to know if STS is unavailable.

I think we shouldn't make this logic too complex, because it's not common for a client to request more than one access delegation mode anyways.

singhpk234 · 2025-11-08T01:44:14Z

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java

+  private AccessDelegationMode selectAccessDelegationMode(
+      Set<AccessDelegationMode> delegationModes) {


does linking this part of spec https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L1859 helps in putting some context that server is allowed to decide which one to choose ?

singhpk234 · 2025-11-08T02:03:26Z