feat(lazer): improve metadata in protobuf #2856

Riateche · 2025-07-16T10:18:41Z

Summary

This PR splits old content of FeedMetadata into two categories:

Essential feed properties that are used for feed aggregation or other functions within Lazer core system (relayer, router, history service): name, exponent, min_publishers, min_rate, expiry_time, market_schedule, state. These are now properties of the Feed message.
Extra metadata used for feed search, documentation, feed catalogs (symbol, description, asset_type, quote_currency), and for matching with external systems (cmc_id, hermes_id). These fields may be used by publishers and by history service (for feed search API), but are otherwise treated as opaque in the Lazer core system. They are now stored as a collection of dynamically typed key-value pairs. This gives the operations team more freedom to extend and change metadata.

Other relevant changes in this PR:

Rename is_activated to is_enabled_in_shard (same for other related fields) to clarify the meaning and intended usage, and to avoid confusion with Feed.state.
Add DynamicValue Rust type to deal with metadata without loss of information.

Rationale

For more flexible metadata.

How has this been tested?

Current tests cover my changes
Added new tests
Manually tested the code

vercel · 2025-07-16T10:18:46Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
api-reference	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
component-library	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
developer-hub	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
entropy-debugger	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
entropy-explorer	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
insights	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
proposals	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am
staking	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 21, 2025 10:46am

lazer/publisher_sdk/proto/state.proto

ali-behjati · 2025-07-16T12:08:19Z

lazer/publisher_sdk/proto/state.proto

+    optional FeedState state = 107;
+
+
+    // [required] Feed status in the current shard. Disabled feeds will not be visible in


sorry i got a bit lost here with feedState. what cases does it cover that feed state is not :? is it like we add a same feed to all the shards that we need this?

Feed state is a consumer-facing and publisher-facing property that's also used to limit access to the feeds. is_enabled_in_shard is a purely internal thing for moving live feeds between shards.

it kind of feels complex but got it. but how do you envision it? on my head i was like we do it as a new feed creation but i think the price feed state is replicated everywhere.

In my view the sequence should be like this:

We have a feed live in the old shard.

We add a feed with the same feed ID and metadata to the new shard and set enabled_in_shard = false.

Agent discovers the feed in the new shard and starts publishing the same data to it.

We verify that the feed in the new shard is ready to go live (publisher data and aggregation are good).

We schedule the switch using disable_in_shard_timestamp in the old shard and enable_in_shard_timestamp in the new shard.

At this timestamp aggregators will stop producing updates for the feed in the old shard and start producing updates for it in the new shard. This makes the switch transparent to consumers, as long as both shards are available to the router that they're connected to.

We delete the feed in the old shard.

lazer/publisher_sdk/rust/src/dynamic_value.rs

keyvankhademi · 2025-07-16T14:28:11Z

lazer/publisher_sdk/proto/state.proto

+    COMING_SOON = 0;
+    // A fully available feed.
    STABLE = 1;
+    // Inactive feeds are not available to consumers or publishers.


it is technically available to publishers now.

What do you mean by "technically available"? Can you suggest a proper description of this state?

I think he means publishers can publish to them regardless of the state of the feed? We should change this in the future though.

Perhaps you can specify they are feeds which can/can't be subscribed to?

Suggested change

// Inactive feeds are not available to consumers or publishers.

// Inactive feeds are not available to consumers.

I think the only difference in INACTIVE and COMING_SOON is to inform the end user that inactive feed will never become active.

we might let some people consume comin_soon ones too.

Updated comments.

keyvankhademi · 2025-07-16T14:29:39Z

lazer/publisher_sdk/proto/state.proto

+    optional string market_schedule = 106;
+    // [required] Feed state
+    optional FeedState state = 107;
+


I think we should add asset_type here too. Because we will be using asset_type to determine what kind of feed type is expected (e.g funding-rate vs regular price).

let's call it feed type.

@ali-behjati are you saying it's different from asset_type and we have both? Because I was thinking we could infer feed type based on asset_type. and multiple asset_types can have the same feed type.

I also prefer to have feed type (Price/FundingRate) because those options are what we care about in the implementation. (It's already like this in the transaction protobuf). Asset type has a lot of options and I think it's better to leave it in the dynamic metadata.

I called it FeedKind because type is annoying to use in Rust 😅

…o protocol

darunrs

LGTM but will let Keyvan/Ali give final approval since they left their own comments already. Just a handful of small questions.

darunrs · 2025-07-17T15:48:30Z

lazer/publisher_sdk/proto/governance_instruction.proto

+// Add a new feed. Refer to `Feed` message fields documentation.
 message AddFeed {
-    // [required] ID of the feed. Must be unique (within the shard).
+    // [required]


Why was this description removed? Is it no longer accurate?

If a feed ID needs to change as a result of moving to another shard which already has that ID (further, how do we prevent this from happening?), how would that impact subscribing to the feed? Is this ID separate from the customer/publisher facing feed ID?

I removed all field comments here because I don't want them to be duplicated with comments on the same fields in Feed. Otherwise they could go out of sync.

Feed IDs have to be globally unique. We don't have a way to enforce it in a shard, but we can check it in xc-admin I guess. If a feed is moved to another shard then it will always have the same ID. If the ID is not the same then it's not really the same feed anymore.

darunrs · 2025-07-17T15:51:00Z

lazer/publisher_sdk/proto/governance_instruction.proto

-    // IDs of publishers enabled for this feed.
-    repeated uint32 permissioned_publishers = 3;
+    // [required]
+    optional DynamicValue.Map metadata = 3;


Any reason we skip field ID 2 if this is not in use yet?

I use the same IDs here as in Feed, and 2 is occupied there by per_publisher field. However, none of that really matters.

darunrs · 2025-07-17T15:54:14Z

lazer/publisher_sdk/proto/state.proto

+    COMING_SOON = 0;
+    // A fully available feed.
    STABLE = 1;
+    // Inactive feeds are not available to consumers or publishers.


I think he means publishers can publish to them regardless of the state of the feed? We should change this in the future though.

Perhaps you can specify they are feeds which can/can't be subscribed to?

darunrs · 2025-07-17T15:55:20Z

lazer/publisher_sdk/proto/state.proto

+// Feed kind determines the set of data fields available in the feed.
+// It also determines the kind of data accepted from publishers for this feed
+// (`PriceUpdate` or `FundingRateUpdate`).
+enum FeedKind {


I intended to introduce this in some form. I will use the state enum to guard rail what updates a feed is able to accept.

we should use this for guard railing updates

darunrs · 2025-07-17T16:00:41Z

lazer/publisher_sdk/proto/state.proto

+    // If a feed is present in
    // multiple shards, it must only be active in one of them at each time.
-    // To enforce this, `pending_activation` and `pending_deactivation` fields
+    // To enforce this, `enable_in_shard_timestamp` and `disable_in_shard_timestamp` fields


To confirm my understanding, Aggregator pulls all shards. So, it is able to handle this switch in logic?

It's not possible to do that deterministically without timestamps because shards have separate transaction queues. Aggregator is only able to achieve deterministic results within a shard because the inputs are always arriving in the same order.

keyvankhademi · 2025-07-17T16:23:09Z

Does this break the current agent transactions?
cc: @bplatak

keyvankhademi · 2025-07-17T16:23:52Z

lazer/publisher_sdk/proto/governance_instruction.proto

+        UpdateFeedMetadata update_feed_metadata = 102;
+        EnableFeedInShard enable_feed_in_shard = 103;
+        DisableFeedInShard disable_feed_in_shard = 104;
        google.protobuf.Empty remove_feed = 199;


we probably don't need this, since we will make feeds inactive instead of deleting.

I prefer to have the option to delete them. Having lots of inactive feeds in state could become too costly.

ali-behjati

LGTM.

I don't see it breaking transaction sending part but it might affect the metadata read part but if it does it's fine as we haven't released it completely yet. @bplatak to say.

Riateche · 2025-07-21T10:45:15Z

Does this break the current agent transactions?

I believe it doesn't. I only changed the package name in the transaction protos.

Riateche added 2 commits July 15, 2025 13:44

feat(lazer): update metadata proto, rename and clarify docs

48521b5

feat(lazer): add DynamicValue parsed type

b1e808e

test(lazer): add DynamicValue tests

1cb5d89

Riateche marked this pull request as ready for review July 16, 2025 10:47

Riateche requested a review from darunrs as a code owner July 16, 2025 10:47

vercel bot deployed to Preview – staking July 16, 2025 10:50 View deployment

vercel bot deployed to Preview – developer-hub July 16, 2025 10:50 View deployment

vercel bot deployed to Preview – entropy-explorer July 16, 2025 10:50 View deployment

vercel bot deployed to Preview – insights July 16, 2025 10:51 View deployment

vercel bot deployed to Preview – entropy-debugger July 16, 2025 10:52 View deployment

vercel bot deployed to Preview – component-library July 16, 2025 10:52 View deployment

vercel bot deployed to Preview – api-reference July 16, 2025 10:53 View deployment

vercel bot deployed to Preview – proposals July 16, 2025 10:54 View deployment

ali-behjati reviewed Jul 16, 2025

View reviewed changes

keyvankhademi reviewed Jul 16, 2025

View reviewed changes

feat(lazer): add FeedKind to protocol

0129da1

vercel bot deployed to Preview – component-library July 17, 2025 09:51 View deployment

vercel bot deployed to Preview – entropy-debugger July 17, 2025 09:52 View deployment

vercel bot deployed to Preview – entropy-explorer July 17, 2025 09:52 View deployment

vercel bot deployed to Preview – insights July 17, 2025 09:53 View deployment

vercel bot deployed to Preview – api-reference July 17, 2025 09:53 View deployment

vercel bot deployed to Preview – developer-hub July 17, 2025 09:54 View deployment

vercel bot deployed to Preview – staking July 17, 2025 09:55 View deployment

vercel bot deployed to Preview – proposals July 17, 2025 09:55 View deployment

feat(lazer): add feed kind to protocol

f8c719b

vercel bot deployed to Preview – entropy-debugger July 17, 2025 10:01 View deployment

vercel bot deployed to Preview – entropy-explorer July 17, 2025 10:01 View deployment

vercel bot deployed to Preview – proposals July 17, 2025 10:02 View deployment

vercel bot deployed to Preview – staking July 17, 2025 10:04 View deployment

vercel bot deployed to Preview – api-reference July 17, 2025 10:04 View deployment

vercel bot deployed to Preview – insights July 17, 2025 10:04 View deployment

fix(lazer): expose dynamic_value protobuf as rust module, move type t…

1054759

…o protocol

vercel bot deployed to Preview – component-library July 17, 2025 12:39 View deployment

vercel bot deployed to Preview – entropy-explorer July 17, 2025 12:40 View deployment

vercel bot deployed to Preview – developer-hub July 17, 2025 12:40 View deployment

vercel bot deployed to Preview – staking July 17, 2025 12:40 View deployment

vercel bot deployed to Preview – insights July 17, 2025 12:41 View deployment

vercel bot deployed to Preview – entropy-debugger July 17, 2025 12:42 View deployment

vercel bot deployed to Preview – api-reference July 17, 2025 12:42 View deployment

vercel bot deployed to Preview – proposals July 17, 2025 12:43 View deployment

darunrs reviewed Jul 17, 2025

View reviewed changes

keyvankhademi reviewed Jul 17, 2025

View reviewed changes

ali-behjati approved these changes Jul 18, 2025

View reviewed changes

keyvankhademi approved these changes Jul 18, 2025

View reviewed changes

Riateche added 2 commits July 21, 2025 11:03

fix: add FeedKind to AddFeed

b1512a8

doc(lazer): improve proto comments

06a3661

vercel bot deployed to Preview – component-library July 21, 2025 10:42 View deployment

vercel bot deployed to Preview – staking July 21, 2025 10:44 View deployment

vercel bot deployed to Preview – proposals July 21, 2025 10:44 View deployment

vercel bot deployed to Preview – developer-hub July 21, 2025 10:44 View deployment

vercel bot deployed to Preview – entropy-explorer July 21, 2025 10:45 View deployment

vercel bot deployed to Preview – entropy-debugger July 21, 2025 10:46 View deployment

vercel bot deployed to Preview – api-reference July 21, 2025 10:46 View deployment

vercel bot deployed to Preview – insights July 21, 2025 10:46 View deployment

Riateche merged commit d6829b7 into main Jul 21, 2025
11 checks passed

Riateche deleted the update-lazer-metadata-proto branch July 21, 2025 10:51

		optional FeedState state = 107;


		// [required] Feed status in the current shard. Disabled feeds will not be visible in

	// Inactive feeds are not available to consumers or publishers.
	// Inactive feeds are not available to consumers.

feat(lazer): improve metadata in protobuf #2856

feat(lazer): improve metadata in protobuf #2856

Uh oh!

Conversation

Riateche commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Rationale

How has this been tested?

Uh oh!

vercel bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darunrs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

keyvankhademi commented Jul 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ali-behjati left a comment

Choose a reason for hiding this comment

Uh oh!

Riateche commented Jul 21, 2025

Riateche commented Jul 16, 2025 •

edited

Loading

vercel bot commented Jul 16, 2025 •

edited

Loading