add refresh credentials property to loadTableResult #2341

jasonf20 · 2025-08-13T08:47:13Z

Rebased version of #1164

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java

dimas-b · 2025-08-19T22:39:09Z

Please address CI failures :)

dimas-b

LGTM overall 👍 Thanks for working on this @jasonf20 !

Do we have existing integration tests for the credential refresh endpoint? If not, could you add a test in RestCatalogMinIOSpecialIT (probably). I suppose we could use the test REST Client to validate refresh credential responses. I know this is specific to the S3 storage integration, but it will hit the common call path and we can validate Azure AccessConfig in unit tests. WDYT?

dimas-b · 2025-08-21T01:47:35Z

polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisResourcePaths.java

+  public String credentialsPath(TableIdentifier ident) {
+    return SLASH.join(
+        "v1",
+        prefix,
+        "namespaces",
+        RESTUtil.encodeNamespace(ident.namespace()),
+        "tables",
+        RESTUtil.encodeString(ident.name()),
+        "credentials");


Just to confirm: relative URIs are correct in this case, right?

Yes, from what I understand the recent version of the client automatically appends the base URI if a relative path is returned from the server.

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java

polaris-core/src/main/java/org/apache/polaris/core/storage/cache/StorageCredentialCache.java

...core/src/main/java/org/apache/polaris/core/storage/gcp/GcpCredentialsStorageIntegration.java

dimas-b · 2025-08-21T01:56:46Z

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

+      Boolean.class,
+      AwsClientProperties.REFRESH_CREDENTIALS_ENABLED,
+      "whether to enable automatic refresh of credentials",
+      true,


I do not think this property is a credential property (logically)

From my understanding of the client code:

The AwsClientFactory is initialized ahead of time with generic properties retrieved from the "configuration" endpoint. This doesn't include specific endpoints for credential refresh per table and/or access keys.

Later on when loading the fileIO per table, the base properties + additional table properties provided from the List<Credential> are added here.

For the property to be available in Credential.config() down the line this needs to be set to true.

Thanks for the code pointer! In this case, it does make sense to mark these properties as "credentials", but please add a code comment for future readers to be aware of what Iceberg java clients expect.

I had another look at this. I think I was wrong since it also passes in tableConf which if set initializes a new fileIO with those properties, so this can be false. I'll update those.

Thanks for your diligence! 🙂

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageIntegration.java

.../test/java/org/apache/polaris/core/storage/azure/AzureCredentialsStorageIntegrationTest.java

jasonf20 · 2025-08-21T09:22:00Z

@dimas-b I've added the integration test and unit test for azure as you suggested and made other fixes based on your comments.

dimas-b

This PR LGTM in its current state 👍 ... with some minor comments 🙂

However, I believe it is worth sending a dev email about the new feature before merging (as commented below).

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialVendor.java

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageIntegration.java

polaris-core/src/main/java/org/apache/polaris/core/storage/cache/StorageCredentialCache.java

dimas-b · 2025-08-21T23:45:36Z

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java

+      EnumSet<AccessDelegationMode> delegationModes,
+      String prefix,
+      TableIdentifier tableIdentifier) {
+    return delegationModes.contains(AccessDelegationMode.VENDED_CREDENTIALS)


This logic is ok from my POV, but I wonder if other community members prefer a feature flag to switch the new behaviour on/off.

Please send an email about this to the dev ML.

Actually, I'll send the dev email myself :)

https://lists.apache.org/thread/dvrx5lg36ovo7387pt6dm06wgmswstvc

singhpk234

Mostly LGTM as well 👍 ! we are very close :) !

singhpk234 · 2025-08-22T00:31:06Z

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

+  AWS_REFRESH_CREDENTIALS_ENDPOINT(
+      String.class,
+      AwsClientProperties.REFRESH_CREDENTIALS_ENDPOINT,
+      "the endpoint to use for refreshing credentials",


ideally we are not refreshing the credentials, we are getting brand new one how about we say something like
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L119

dimas-b · 2025-08-22T13:32:20Z

@jasonf20 : Could you add an entry about this change to CHANGELOG.md, section New Features?

…ndpoint is included

…tion

dimas-b

LGTM 👍 Thanks again for you contribution, @jasonf20 !

Since this PR was already mentioned on the dev ML on Aug 21 and no concerned were raised, I think it is good to merge. I'll wait until Aug 26 before merging, in case any last minute comments come up.

singhpk234

LGTM overall (have some small cosmetic suggestion for documentation), Thank you @jasonf20 ! much needed feature for long running jobs !

will wait for @dimas-b review and approval before merge

singhpk234 · 2025-08-25T14:03:52Z

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialVendor.java

   * @param allowedWriteLocations a set of allowed to write locations
+   * @param refreshCredentialsEndpoint an optional endpoint to use for refreshing credentials. If
+   *     supported by the storage type it will be returned to the client in the appropriate
+   *     properties. The endpoint may be relative to the base URI and the client is responsible for


can we return anything except relative path ? my understanding is base uri is always catalog uri https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java#L92 is this statement in intentionally open ended since spec is not explicit ?

At one point the client required full paths, now it works with partial paths. I'm not sure if the spec identifies one as the correct method over the other so I left it like this. If one is definitely the "correct" way I can update the comment.

@jasonf20 : do you know the Iceberg version / PR where relative paths started working?

It was linked in the previous PR. #1164 (comment)

Seems like it's since 1.8.0

Oh no this made a lot of things super tricky, i checked iceberg python for refresh keyword but i didn't any prs for that so i am assuming that it doesn't support, taking only the iceberg java sdk reference
if we always send relative url it only works for >= 1.8
fails for < 1.8
I know there is a header that the java sdk sends which has SDK version X-Client-Version header we may want to salvage this now on when to return the absolute vs relative, https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L77

seems like the only broken release would be 1.7

Its a bit surprising to me, but i think for now we can park this by just documenting this gotcha, WDYT @dimas-b ?

Getting full URI is a bit trickier - as discussed in #1164.

From my POV, let's merge this "as is" and open a GH issue that it does not work properly for Iceberg < 1.8. When someone has time, let's improve.

Getting this feature enabled for Iceberg clients >= 1.8 is valuable, I think.

Since old client version behave differently, I think we ought to have a feature flag now... just in case someone runs into client incompatibility issues and wants to disable credential refresh properties completely (i.e. revert to old Polaris behaviour). WDYT?

That would be ideal IMHO (specially now), but i think the jobs without us sending the credentials endpoint would still have failed because my understanding is we would have not made the /credentials call unless the creds in loadTable expires https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java#L63, its just we might see an HTTP error rather than the creds expired exception.
But agree, we might want to have a feature flag to fallback to the old behaviour if we unexpectedly fail.

I don't think a feature flag is needed.

The client is already broken, if they reach the need to refresh credentials it already fails today.

A feature flag will not solve the problem. Enabling it will fix the issue for some and create a different issue for others, so you'll never be able to enable it anymore than you can without the flag.

Clients can simply upgrade to 1.8.0 to get a working version. Their current version doesn't work anyway so they can upgrade at their own time when this becomes important to them.

If you would like we can remove the refresh_credentials_enabled property and let the client set that one in his catalog settings. That way he can control if this gets used or not. This would be better than a flag and is controlled client side. not server side.

I've added a commit that does 4.

CHANGELOG.md

singhpk234 · 2025-08-25T14:10:19Z

Looks like we already have @dimas-b ;) review, we were parallely reviewing, we good to go from my end !

dimas-b · 2025-08-25T14:12:20Z

Oops, CI failure 😅

jasonf20 · 2025-08-25T16:01:18Z

Oops, CI failure 😅

fixed

dimas-b

@jasonf20 : I'm a bit lost now... Why do we need to remove the "enabled" flag in 2a123d1?

In my earlier comments I mean that we need to add a new flag to FeatureConfiguration and use it to control whether to add the credential refresh endpoint and the related "enabled" flag to AccessConfig... WDYT?

dimas-b

After re-reading @jasonf20 comment, I think removing of the "enabled" properties from AccessConfig is a sound approach.

singhpk234 · 2025-08-26T14:30:02Z

CHANGELOG.md

+- Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
+automatic storage credential refresh per table on the client side.


Lets call out the client version limitation here and the suggestion to the caller to set the client side override to enable this.

good points!

since the release is imminent, if the round trip cycle is slow, I suppose we can still merge this PR and later adjust the changelog between @singhpk234 and myself :) Let's give it a few more hours.

I updated the changelog. Seems a little verbose now for a changelog but It's probably fine.

p.s. I believe you should be able to push changes to my branch yourself since I have the "Allow edits and access to secrets by maintainers" checkbox enabled. But I never tried using that myself.

singhpk234 · 2025-08-26T15:47:47Z

CHANGELOG.md

-automatic storage credential refresh per table on the client side.
+automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required.
+The endpoint path is always returned when using vended credentials, but the client must configure 
+REFRESH\_ENDPOINT\_ENABLED = "true" in the catalog properties for the client to use the endpoint.


the properties are unfortunately different for the all the cloud providers, not sure if we wanna call this out, just saying enabled never the less refresh-credentials-enabled should be endpoint

how about we say that clients must enable refresh-credentials flag for their respective cloud provider ?

Yea that's less verbose. I changed it to that.

singhpk234

LGTM, thanks @jasonf20 for all the effort !
Appreciate your contribution :), much needed capability !

jasonf20 · 2025-08-26T16:09:03Z

Thanks for your reviews @dimas-b @singhpk234 !

flyrain · 2025-08-26T16:44:14Z

CHANGELOG.md

+- Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
+automatic storage credential refresh per table on the client side. Java client version >= 1.8.0 is required.
+The endpoint path is always returned when using vended credentials, but clients must enable the 


We will need to document this. Unfortunately, the best place to doc this is a page for storage setting and credential vending, which doesn't exist yet. Here is the issue filed long time ago, #1325. Not a blocker for this PR though.

github-project-automation bot added this to Basic Kanban Board Aug 13, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Aug 13, 2025

jasonf20 mentioned this pull request Aug 13, 2025

add refresh credentials property to loadTableResult #1164

Closed

jasonf20 force-pushed the loadTableResponse-refresh-credential-properties branch from bb55a80 to eb3e360 Compare August 13, 2025 08:54

singhpk234 reviewed Aug 14, 2025

View reviewed changes

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java Outdated Show resolved Hide resolved

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java Outdated Show resolved Hide resolved

jasonf20 force-pushed the loadTableResponse-refresh-credential-properties branch from 5c92bdd to 99075da Compare August 19, 2025 07:39

singhpk234 reviewed Aug 19, 2025

View reviewed changes

.../service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java Outdated Show resolved Hide resolved

dimas-b reviewed Aug 21, 2025

View reviewed changes

singhpk234 reviewed Aug 22, 2025

View reviewed changes

dimas-b modified the milestones: 1.2.0, 1.1.0 Aug 22, 2025

jasonf20 requested review from dimas-b and singhpk234 August 25, 2025 08:32

jasonf20 added 12 commits August 25, 2025 11:33

add refresh credentials property to loadTableResult

d6066fa

IcebergCatalogAdapterTest: Added test to ensure refresh credentials e…

5c05a0d

…ndpoint is included

fixup! add refresh credentials property to loadTableResult

b9386a2

fixed lint error

0e388b5

delegate refresh credential endpoint configuration to storage integra…

149b1fc

…tion

change stirng to optional<string>

32ca5c2

added unit test for azure

d8d11b3

Add endpoint to cache key

7259b61

Added integration test

d449ccf

GCP: Add refresh credential properties'

211d315

Set isCredential to false

77d0968

cache: use endpoint from key

c93d2b3

jasonf20 force-pushed the loadTableResponse-refresh-credential-properties branch from 431de42 to 2d6dd03 Compare August 25, 2025 08:33

dimas-b previously approved these changes Aug 25, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 25, 2025

singhpk234 previously approved these changes Aug 25, 2025

View reviewed changes

jasonf20 added 3 commits August 25, 2025 18:51

changelog update

2cbfbd5

fixed unit tests

4159327

fixed integration tests

cae0632

jasonf20 dismissed stale reviews from singhpk234 and dimas-b via cae0632 August 25, 2025 16:01

Remove REFRESH_CREDENTIALS_ENABLED from server response

2a123d1

dimas-b reviewed Aug 26, 2025

View reviewed changes

dimas-b previously approved these changes Aug 26, 2025

View reviewed changes

singhpk234 previously approved these changes Aug 26, 2025

View reviewed changes

Updated changelog

d8026f2

jasonf20 dismissed stale reviews from singhpk234 and dimas-b via d8026f2 August 26, 2025 15:41

Updated changelog

a6ac0cf

singhpk234 reviewed Aug 26, 2025

View reviewed changes

jasonf20 added 2 commits August 26, 2025 18:51

Updated changelog

9b9dc41

Updated changelog

e59fc7e

dimas-b approved these changes Aug 26, 2025

View reviewed changes

singhpk234 approved these changes Aug 26, 2025

View reviewed changes

flyrain reviewed Aug 26, 2025

View reviewed changes

dimas-b merged commit 9e6d929 into apache:main Aug 28, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Aug 28, 2025

dimas-b mentioned this pull request Oct 29, 2025

Credential Vending: refresh-token-endpoint is not set #2177

Closed

		- Added refresh credentials endpoint configuration to LoadTableResponse for AWS, Azure, and GCP. Enabling
		automatic storage credential refresh per table on the client side.

add refresh credentials property to loadTableResult #2341

add refresh credentials property to loadTableResult #2341

Uh oh!

Conversation

jasonf20 commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Aug 19, 2025

Uh oh!

dimas-b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasonf20 Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasonf20 commented Aug 21, 2025

Uh oh!

dimas-b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b commented Aug 22, 2025

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

singhpk234 Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jasonf20 Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jasonf20 commented Aug 13, 2025 •

edited

Loading

dimas-b left a comment •

edited

Loading

dimas-b Aug 21, 2025 •

edited

Loading

jasonf20 Aug 21, 2025 •

edited

Loading

dimas-b Aug 21, 2025 •

edited

Loading

dimas-b left a comment •

edited

Loading

dimas-b Aug 21, 2025 •

edited

Loading

singhpk234 Aug 25, 2025 •

edited

Loading

jasonf20 Aug 25, 2025 •

edited

Loading

singhpk234 Aug 25, 2025 •

edited

Loading

dimas-b Aug 26, 2025 •

edited

Loading

jasonf20 Aug 26, 2025 •

edited

Loading