scheduler: fragment queue and querier pick-up coordination #6968

rubywtl · 2025-08-13T22:54:36Z

What this PR does:
This PR introduces a Fragmenter interface that splits logical query plans into fragments when distributed execution is enabled. The Fragmenter appends metadata to each fragment for tracking, which the scheduler then uses to route fragments to appropriate queriers. The scheduler maintains a mapping between fragments and querier addresses to track fragment locations across the distributed system.

Which issue(s) this PR fixes:
related to proposal #6789

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24 · 2025-08-14T03:35:57Z

pkg/cortex/modules.go

 	t.Cfg.Worker.MaxConcurrentRequests = t.Cfg.Querier.MaxConcurrent
 	t.Cfg.Worker.TargetHeaders = t.Cfg.API.HTTPRequestHeadersToLog
-	return querier_worker.NewQuerierWorker(t.Cfg.Worker, httpgrpc_server.NewServer(internalQuerierRouter), util_log.Logger, prometheus.DefaultRegisterer)
+	ipAddr, err := ring.GetInstanceAddr(t.Cfg.Alertmanager.ShardingRing.InstanceAddr, t.Cfg.Alertmanager.ShardingRing.InstanceInterfaceNames, util_log.Logger)


Why using alertmanager config here

The gPRC params I needed are under RingConfig struct, which is called ShardedRing here, but it doesn't exist under querier

[update] I will add new field (ring configs) for querier 👍

Umm I don't think we want to add a Ring for querier. We just need the configurations for the addresses and interface, etc

pkg/distributed_execution/fragment_key.go

pkg/scheduler/plan_fragments/fragmenter.go

pkg/scheduler/plan_fragments/fragment_table.go

yeya24 · 2025-08-14T03:51:13Z

pkg/scheduler/plan_fragments/fragment_table.go

+	defer f.mu.Unlock()
+
+	keysToDelete := make([]distributed_execution.FragmentKey, 0)
+	for key := range f.mappings {


Looking at the methods you have, is it easier to change mappings from mappings map[distributed_execution.FragmentKey]string to map[uint64]map[uint64]string?

You can find the map by just a lookup

True, but I made the FragmentKey struct so that it is easier to maintain (for example: if we ever want to change the types for the IDs or add more fields, we dont have to go through the codebase to fix it), and the code will be easier to understand (more literal). This fragment key type is also reused for remote nodes and child-root execution accesses to result cache in future PRs.

yeya24 · 2025-08-14T03:54:18Z

pkg/scheduler/plan_fragments/fragmenter.go

+
+import "github.com/thanos-io/promql-engine/logicalplan"
+
+type Fragmenter interface {


It is better to move the Fragmenter to distributed_execution as fragmentation is specific to remote distribution.

The fragment table can be just moved to scheduler folder

pkg/querier/worker/worker.go

yeya24 · 2025-08-26T23:17:44Z

pkg/scheduler/fragment_table/fragment_table.go

+	}
+}
+
+func (f *FragmentTable) AddMapping(queryID uint64, fragmentID uint64, addr string) {


Can we find a better name other than mapping?

yeya24 · 2025-08-26T23:17:53Z

pkg/scheduler/fragment_table/fragment_table.go

+	return "", false
+}
+
+func (f *FragmentTable) ClearMappings(queryID uint64) {


yeya24 · 2025-08-26T23:18:46Z

pkg/scheduler/scheduler.go

+
+	// queryKey <--> fragment-ids lookup table allows faster cancellation of the whole query
+	// compared to traversing through the pending requests to find matching fragments
+	queryToFragmentsLookUp map[queryKey][]uint64


Let's find a better name for this. LookUp is not a correct name

yeya24 · 2025-08-26T23:19:50Z

pkg/scheduler/scheduler.go


-type requestKey struct {
+// additional layer to improve efficiency of deleting fragments of logical query plans
+// while maintaining previous logics


The comment is confusing. Which previous logic this struct maintains?

Previous logic is to cancel a query by its queryID and frontend address, but now there are multiple fragments under one queryID, and traversing through the pending request queue and checking the queryID is in-efficient, so I added an extra layer of mapping to keep track of the fragment IDs under the same queryKey.

pkg/scheduler/scheduler.go

yeya24 · 2025-08-26T23:59:20Z

pkg/scheduler/scheduler.go

+			for _, childID := range req.fragment.ChildIDs {
+				addr, ok := s.fragmentTable.GetChildAddr(req.queryID, childID)
+				if !ok {
+					return


Do we need to return some error here if missing child addr?

I'm a bit confused as I don't see any s.fragmentTable.AddAddressByID before this check. Could you clarify when you expect us to add items to the table before we get here?

AddAddressByID is called in forwardRequestToQuerier, after fragments are picked up by queriers.

This ensures that when a querier successfully picks up a fragment, its address and fragment ID are recorded in the address table. Since we process fragments in child-to-parent order, parent fragments (which are scheduled later) can reliably look up their child fragment addresses using the fragment IDs when they need to fetch results. This ordering guarantees that child fragment addresses are already available when parent fragments need them.

Ah nice, child-to-parent order was what I was missing

pkg/scheduler/schedulerpb/scheduler.proto

justinjung04 · 2025-09-04T23:27:36Z

pkg/scheduler/scheduler.go

+
+	// Enables or disables distributed query execution functionality
+	distributedExecEnabled bool
+	fragmenter             plan_fragments.DummyFragmenter // Splits logical plans into executable fragments


Is the type meant to be Fragmenter?

justinjung04 · 2025-09-04T23:32:59Z

pkg/scheduler/scheduler.go

+			for _, childID := range req.fragment.ChildIDs {
+				addr, ok := s.fragmentTable.GetChildAddr(req.queryID, childID)
+				if !ok {
+					return


I'm a bit confused as I don't see any s.fragmentTable.AddAddressByID before this check. Could you clarify when you expect us to add items to the table before we get here?

justinjung04 · 2025-09-04T23:34:16Z

pkg/scheduler/scheduler.go

+			// if there is an error in any of the process enqueueing the fragments
+			// immediately propagate the error back
+			if err != nil {
+				return err
+			}
+
+			return nil


nit: simplify to return err?

justinjung04

Looks great, thank you

pkg/distributed_execution/config.go

pkg/scheduler/fragment_table/fragment_table.go

Signed-off-by: rubywtl <[email protected]>

yeya24

Thanks! Great work

pull-request-size bot added the size/XL label Aug 13, 2025

dosubot bot added the component/querier label Aug 13, 2025

yeya24 reviewed Aug 14, 2025

View reviewed changes

rubywtl force-pushed the scheduler/logicalplan_fragment_coordination branch 4 times, most recently from 89e8021 to 67eff93 Compare August 21, 2025 21:39

yeya24 reviewed Aug 26, 2025

View reviewed changes

yeya24 reviewed Aug 27, 2025

View reviewed changes

pkg/scheduler/schedulerpb/scheduler.proto Outdated Show resolved Hide resolved

rubywtl force-pushed the scheduler/logicalplan_fragment_coordination branch 12 times, most recently from beeb9a5 to c81acda Compare September 4, 2025 19:06

justinjung04 reviewed Sep 4, 2025

View reviewed changes

rubywtl force-pushed the scheduler/logicalplan_fragment_coordination branch from c81acda to 06acad2 Compare September 5, 2025 16:53

justinjung04 approved these changes Sep 5, 2025

View reviewed changes

yeya24 reviewed Sep 5, 2025

View reviewed changes

pkg/distributed_execution/config.go Outdated Show resolved Hide resolved

pkg/distributed_execution/config.go Outdated Show resolved Hide resolved

pkg/scheduler/fragment_table/fragment_table.go Outdated Show resolved Hide resolved

allow logical plan fragment type for scheduler queue

9a5c0af

Signed-off-by: rubywtl <[email protected]>

rubywtl force-pushed the scheduler/logicalplan_fragment_coordination branch from 06acad2 to 9a5c0af Compare September 5, 2025 18:35

yeya24 approved these changes Sep 5, 2025

View reviewed changes

yeya24 merged commit eab2099 into cortexproject:master Sep 5, 2025
18 checks passed


		import "github.com/thanos-io/promql-engine/logicalplan"

		type Fragmenter interface {

scheduler: fragment queue and querier pick-up coordination #6968

scheduler: fragment queue and querier pick-up coordination #6968

Uh oh!

Conversation

rubywtl commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rubywtl Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rubywtl Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinjung04 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rubywtl commented Aug 13, 2025 •

edited

Loading

rubywtl Aug 14, 2025 •

edited

Loading

rubywtl Aug 14, 2025 •

edited

Loading