Extract interface for RequestIdGenerator #2720

adutra · 2025-09-30T13:15:18Z

Summary of changes:

Extracted an interface from RequestIdGenerator.
The generateRequestId method now returns a Uni<String> in case custom implementations need to perform I/O or other blocking calls during request ID generation.
Also addressed comments in Generate Request IDs (if not specified); Return Request ID as a Header #2602.

Summary of changes: 1. Extracted an interface from `RequestIdGenerator`. 2. The `generateRequestId` method now returns a `Uni<String>` in case custom implementations need to perform I/O or other blocking calls during request ID generation. 3. Also addressed comments in apache#2602.

adutra · 2025-09-30T13:19:27Z

cc @adnanhemani

adnanhemani · 2025-09-30T20:05:56Z

runtime/service/src/main/java/org/apache/polaris/service/tracing/RequestIdFilter.java

+            : requestIdGenerator.generateRequestId(rc))
+        .onItem()
+        .invoke(id -> rc.setProperty(REQUEST_ID_KEY, id))
+        .invoke(id -> ContextLocals.put(REQUEST_ID_KEY, id))


What is this for?

Same as we do in RealmContextFilter: this places the request ID in the current Vertx context for later retrieval by another event loop thread:

https://quarkus.io/guides/duplicated-context#context-local-data

IIUC, ContextLocals seems a global container to hold anything could be used within one request context. Another option is to use a explicit RequestScoped bean to store these, like we did in the call context bean. In that case, it's a more typed solution. How would you compare both approaches?

In a slightly orthogonal direction: I don't see us using the variables we've already stored in ContextLocals anywhere. So while I see that we are putting the information in RealmContextFilter, what is our (proposed?) use of this?

Another thought (I'm not sure if this actually answers the question) - if we were to use the request ID later downstream in the application (for example, Event Listeners), would it be better to use ContextLocals or ContainerRequestContext?

How would you compare both approaches?

Indeed, a request-scoped bean is undoubtedly better.

So while I see that we are putting the information in RealmContextFilter, what is our (proposed?) use of this?

This was done a while ago, as I thought at that time that we would be leveraging context propagation to "transfer" some information from the initial request scope to, say, the async tasks framework.

It seems though that, since that time, we've been taking the opposite direction: that of avoiding context propagation (due to some issues with bean proxies, etc.)

The situation today is that those calls to ContextLocals are probably not necessary anymore. They can be removed.

Are we OK if I remove the call to ContextLocals in RequestIdFilter in this PR, then remove the other call in RealmContextFilter in a different PR?

FYI: #2747 (removed ContextLocals and seized the opportunity for some minor cleanup).

UPDATE: I actually remember now 😄

ContextLocals is required in this specific place:

polaris/runtime/service/src/main/java/org/apache/polaris/service/metrics/RealmIdTagContributor.java

Line 41 in 20febda

context.requestContextLocalData(RealmContextFilter.REALM_CONTEXT_KEY);

I even remember asking the Quarkus devs for this feature:

quarkusio/quarkus#47887

So, TLDR: we need to keep ContextLocals in RealmContextFilter. As for RequestIdFilter we can remove it, at least for now.

flyrain · 2025-09-30T20:33:32Z

runtime/service/src/main/java/org/apache/polaris/service/tracing/RequestIdGenerator.java

+   * Generates a new request ID. IDs must be fast to generate and unique. If the generation involves
+   * I/O (which is not recommended), it should be performed asynchronously.


Can we remove these comments, if I/O involving is not recommended?

Suggested change

* Generates a new request ID. IDs must be fast to generate and unique. If the generation involves

* I/O (which is not recommended), it should be performed asynchronously.

* Generates a new request ID. IDs must be fast to generate and unique.

flyrain

Thanks @adutra for the PR — it looks good to me overall!
Would you mind elaborating a bit on the motivation behind it?
In what situation would we need a custom request ID generator?

adutra · 2025-10-01T13:54:55Z

Would you mind elaborating a bit on the motivation behind it?

I'm trying to avoid what happened with the RealmContextFilter a while ago: #1345.

IOW, we want to make sure that whatever technique is used to generate a request ID, we won't see Quarkus complaining about it being blocking.

In what situation would we need a custom request ID generator?

I could see e.g. users storing their request IDs in the database, or using Zookeeper's distributed atomic counter.

dimas-b · 2025-10-01T15:00:45Z

runtime/service/src/main/java/org/apache/polaris/service/tracing/DefaultRequestIdGenerator.java

+    @Override
+    @Nonnull
+    public String toString() {
+      return String.format("%s_%019d", uuid(), counter());


nit: if lexicographic ordering is desired (based on previous conversations), it might be worth using something like <MILLIS_SINCE_EPOCH_PADDED>_<COUNTER_PADDED>_<UUID> WDYT?

or <MILLIS_PADDED>_<SMALL_COUNTER>_<UUID> - SMALL_COUNTER meaning something in the range of 10K to avoid calling the system clock too often.

but this might be getting too much into the monotonic clock area 😅

I'm not sure lexicographic sort order is desirable for anything else than making it easier for humans to visualize and compare request IDs – so any of your suggestions is fine 😄
How about we get this in without changing the generation logic, and then, we look into ways of making the logic better?

Absolutely 👍

Personally, I think it's fine if the requests are only lexicographically sorted across all requests made per node. IMO request ID sorting doesn't mean much for API clients as they also usually log timestamps.

I'm okay to change it, if you both feel there's something better - but just my two cents 😃

adnanhemani · 2025-10-02T09:15:58Z

In what situation would we need a custom request ID generator?

I could see e.g. users storing their request IDs in the database, or using Zookeeper's distributed atomic counter.

To be clear, I am mostly fine with the changes and asking mainly to learn here - isn't the expected workflow that, in the case users want to store their request IDs in a database prior to the API call, that they would store their request IDs on client-side and pass it into Polaris via the request headers? Can you please explain further in what use case the server-side would actually want to store (and then use) pre-computed request IDs?

I think ZK's distributed atomic counter would be a good use case where making request ID asynchronous would help, although I'm sure most developers nowadays would prefer not to use ZK for a use case like creating request IDs unless they have no other option :)

adutra · 2025-10-02T10:08:48Z

FYI. One thing that annoys me a bit but I didn't want to change it without prior discussion:

The classes RequestIdFilter, RequestIdGenerator and RequestIdResponseFilter are all in the org.apache.polaris.service.tracing package – but the configuration they use is declared in the org.apache.polaris.service.logging package and the configuration option is prefixed with polaris.log.

I find this setup a little confusing, especially for developers.

I can provide a "fix" for that later if you all agree.

Summary of changes: 1. Remove the call to ContextLocals (context: apache#2720 (comment)). 2. Don't include the exception's message in the response as it can leak details about Polaris internals. 3. Add a small test for success and failure cases.

flyrain

Thanks @adutra !

github-project-automation bot added this to Basic Kanban Board Sep 30, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Sep 30, 2025

adutra force-pushed the request-id-generator-interface branch from 8f5f2b6 to 6e58c97 Compare September 30, 2025 13:18

adnanhemani reviewed Sep 30, 2025

View reviewed changes

flyrain reviewed Sep 30, 2025

View reviewed changes

review

b44573f

dimas-b previously approved these changes Oct 1, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Oct 1, 2025

Remove ContextLocals

3221913

adutra dismissed dimas-b’s stale review via 3221913 October 2, 2025 09:56

adutra mentioned this pull request Oct 2, 2025

Cleanup RealmContextFilter #2747

Merged

dimas-b approved these changes Oct 2, 2025

View reviewed changes

flyrain approved these changes Oct 2, 2025

View reviewed changes

sfc-gh-ahemani approved these changes Oct 2, 2025

View reviewed changes

adnanhemani approved these changes Oct 2, 2025

View reviewed changes

adutra merged commit d2a607a into apache:main Oct 3, 2025
16 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Oct 3, 2025

adutra deleted the request-id-generator-interface branch October 3, 2025 14:46

adutra mentioned this pull request Oct 3, 2025

Rename request ID to correlation ID #2757

Closed

		* Generates a new request ID. IDs must be fast to generate and unique. If the generation involves
		* I/O (which is not recommended), it should be performed asynchronously.

Extract interface for RequestIdGenerator #2720

Extract interface for RequestIdGenerator #2720

Uh oh!

Conversation

adutra commented Sep 30, 2025

Uh oh!

adutra commented Sep 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

adutra commented Oct 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adnanhemani commented Oct 2, 2025

Uh oh!

adutra commented Oct 2, 2025

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dimas-b Oct 1, 2025 •

edited

Loading