feat: Adds a streaming snapshot paser #259

buddhisthead · 2025-10-17T19:13:45Z

Implement a complete streaming parser for Cardano Conway-era snapshots that parses the NewEpochState CBOR structure without loading everything into memory.

Key features:

Callback-based streaming architecture for UTXOs, pools, accounts, and DReps
Parses 11.2M UTXOs at ~2M UTXOs/second
Extracts 3,095 stake pools with full metadata (pledge, cost, margin, relays)
Parses 1.41M stake accounts with rewards and delegations (SPO & DRep)
Extracts 278 DReps with deposits and anchor metadata
Uses minicbor Decode trait for type-safe CBOR parsing
Adds some documentation about the snapshot file structure. See stuff under docs

Components added:

streaming_snapshot.rs: Main streaming parser with callback traits
pool_params.rs: Stake pool parameter types with CBOR decoding
hash.rs: Generic hash types for pool IDs, VRF keys, addresses (stolen from Amaru)
AccountState/StakeAddressState: Shared types for account state. AccountState can be sent on the msg bus.
test_streaming_parser.rs: Example showing callback usage
Makefile: snap-test-streaming target for testing

TESTING

Run make snap-test-streaming at the top level to see it in action. That invokes a new integration test under examples which reads the giant Epoch 505 snapshot file. There is a README.md file there too for instructions.

- Add streaming_snapshot.rs with callback-based parser - Add parser.rs with manifest validation - Add hash.rs and pool_params.rs for snapshot types - Move Account/AccountDRep types into streaming_snapshot.rs (internal only) - Add AccountState to stake_addresses.rs for external API - Add sha2 dependency for integrity checking - Include test fixtures for validation

…enerates a snapshot manifest file

Copilot

Pull Request Overview

Adds a callback-based streaming snapshot parser for Cardano Conway-era snapshots, plus manifest tooling, fixtures, and documentation.

Introduces a streaming parser that navigates NewEpochState and invokes callbacks for UTXOs, pools, accounts, DReps, and proposals
Adds manifest parsing and integrity validation utilities and example/Makefile for running the parser against large fixtures
Documents Amaru snapshot structure and the parser’s architecture and usage

Reviewed Changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
common/src/snapshot/streaming_snapshot.rs	Core streaming parser, data types, and callback traits; includes UTXO streaming, pools/accounts/DReps extraction
common/src/snapshot/pool_params.rs	Pool parameter CBOR encode/decode types used by the streaming parser
common/src/snapshot/parser.rs	Manifest parsing and integrity validation utilities
common/src/snapshot/mod.rs	Module wiring and re-exports
common/src/hash.rs	Generic hash utilities and minicbor (de)serialization helpers used by parser types
common/examples/test_streaming_parser.rs	Example harness to run the streaming parser and print summaries
scripts/generate_manifest.py	Standalone manifest generation for snapshots (filename/header-based), streaming SHA256
docs/streaming-snapshot-parser.md	Parser architecture, callbacks, features, and usage
docs/amaru-snapshot-structure.md	Detailed description of Amaru snapshot (NewEpochState) structure
tests/fixtures/*	LFS pointer files and sample manifests for testing
tests/fixtures/.gitattributes	LFS settings for large fixture
common/Cargo.toml	Adds sha2 dependency used by integrity validation
common/src/lib.rs	Exposes new snapshot and hash modules
Makefile	Convenience targets to run parser example and generate manifests

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-17T19:17:25Z

common/src/snapshot/streaming_snapshot.rs

+            let tx_hash_bytes = match decoder.bytes() {
+                Ok(b) => b,
+                Err(_e) => {
+                    errors += 1;
+                    decoder.skip().ok(); // skip remaining TxIn fields and value
+                    continue;
+                }
+            };
+
+            let output_index = match decoder.u64() {
+                Ok(idx) => idx,
+                Err(_e) => {
+                    errors += 1;
+                    decoder.skip().ok(); // skip value
+                    continue;
+                }
+            };


On key-parse errors, only one skip() is issued, which leaves the map value (TxOut) unconsumed and desynchronizes the decoder. When tx_hash decoding fails, you need to skip the remaining element(s) of the TxIn array and then also skip the corresponding TxOut value; similarly, when output_index fails, you must skip the TxOut value as well. Example fix: call skip() for the missing array element and then an additional skip() for the value.

Copilot · 2025-10-17T19:17:25Z

common/src/snapshot/streaming_snapshot.rs

+        let mut buffer = Vec::new();
+        file.read_to_end(&mut buffer).context("Failed to read snapshot file")?;
+
+        let mut decoder = Decoder::new(&buffer);


The parser reads the entire snapshot into memory, which contradicts the streaming goal and will blow memory for multi‑GB snapshots. Prefer a zero‑copy or low‑copy approach such as memory-mapping (memmap2) or a reader-based decoder to keep memory bounded, and stream UTXOs directly from the file.

Good point. We do read the whole damn thing into memory first. I had a previous version with read the UTXOs in by 16M at a time. I'll leave that for another PR so we don't keep pushing this out.

Agreed - I thought the benefit of streaming/callback was we could read it progressively, at least at the overall section level and with chunks of UTXOs / addresses.

Yep. I had planned to do this and thought I'd move things along faster by pushing this out to a later version. But I can go a head and address it now.

Wow. This is way way harder than I thought. My previous "16 MB streamer" wasn't really doing that at all. I'm trying an approach now with ciborium but may just punt this for later since we just need to get boot from snapshot working.

UTXO element is a CBOR MAP with 11.2M entries

Average: 194 bytes per UTXO
Minimum: 92 bytes
Maximum: 22,656 bytes (~22KB)
Most common: 100-199 bytes (9.2M entries)

So it should be good for ciborium streaming. Will give it a try.

Done. 4.7 seconds to parse the whole file, using max of 256MB (meta data) + 64MB buffer for streaming UTXO parsing.

common/src/snapshot/streaming_snapshot.rs

common/examples/test_streaming_parser.rs

Copilot · 2025-10-17T19:17:26Z

common/examples/test_streaming_parser.rs

+                    "  Account #{}: {} (utxo: {}, rewards: {}, pool: {:?}, drep: {:?})",
+                    i + 1,
+                    &account.stake_address[..32],
+                    account.address_state.utxo_value,
+                    account.address_state.rewards,
+                    account.address_state.delegated_spo.as_ref().map(|s| &s[..16]),
+                    account.address_state.delegated_drep


Slicing delegated_spo bytes with &s[..16] can panic when s.len() < 16. Consider formatting as hex and truncating on the string: let h = hex::encode(s); let shown = &h[..h.len().min(16)];

Copilot · 2025-10-17T19:17:27Z

common/examples/test_streaming_parser.rs

+                eprintln!(
+                    "  Proposal #{}: {} (deposit: {}, action: {}, by: {})",
+                    i + 1,
+                    proposal.gov_action_id,
+                    proposal.deposit,
+                    proposal.gov_action,
+                    &proposal.reward_account[..32]


Fixed-length slicing of reward_account (&[..32]) risks panic if the string is shorter than 32. Use a safe truncation (e.g., let s = &proposal.reward_account; let shown = &s[..s.len().min(32)];).

Suggested change

eprintln!(

" Proposal #{}: {} (deposit: {}, action: {}, by: {})",

i + 1,

proposal.gov_action_id,

proposal.deposit,

proposal.gov_action,

&proposal.reward_account[..32]

let s = &proposal.reward_account;

let shown = &s[..s.len().min(32)];

eprintln!(

" Proposal #{}: {} (deposit: {}, action: {}, by: {})",

i + 1,

proposal.gov_action_id,

proposal.deposit,

proposal.gov_action,

shown

scripts/generate_manifest.py

docs/amaru-snapshot-structure.md

common/src/stake_addresses.rs

- Defer on_metadata callback until after parsing deposits from UTxOState[1] - Parse deposits field correctly (previously hardcoded to 0) - Fix stream_utxos to consume break marker for indefinite-length maps - Return UTXO count from stream_utxos for accurate metadata - Update callback order: metadata now called after all data, before on_complete - Update documentation to reflect new callback invocation order Metadata now reports accurate pot balances: - Deposits: 4,612,238,000,000 lovelace (~4.6M ADA) - UTXO count: 11,199,911 This ensures correct accounting of the deposits pot which is critical for bootstrap state distribution.

buddhisthead · 2025-10-17T20:13:30Z

common/src/stake_addresses.rs

    }

    #[inline]
-    pub fn entry(&mut self, stake_key: KeyHash) -> Entry<KeyHash, StakeAddressState> {


These are fixes for warnings so we can eventually run cargo check --warnings and pass.

Copilot

Pull Request Overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 10 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

common/src/snapshot/streaming_snapshot.rs

Copilot · 2025-10-17T20:15:58Z

common/src/snapshot/streaming_snapshot.rs

+                    reward_account: hex::encode(&params.reward_account.0),
+                    pool_owners: params.owners.iter().map(|h| h.to_string()).collect(),


PoolInfo.reward_account is documented as Bech32 but is emitted as hex. Convert to Bech32 here or revise the API docs/field name to reflect hex output to avoid consumer confusion.

common/src/snapshot/streaming_snapshot.rs

Copilot · 2025-10-17T20:15:59Z

common/src/hash.rs

+impl<const BYTES: usize> From<&[u8]> for Hash<BYTES> {
+    fn from(value: &[u8]) -> Self {
+        let mut hash = [0; BYTES];
+        hash.copy_from_slice(value);
+        Self::new(hash)
+    }
+}


copy_from_slice will panic if value.len() != BYTES. Prefer TryFrom<&[u8]> returning a Result to avoid panics, or at minimum assert the length and map to a descriptive error.

Copilot · 2025-10-17T20:15:59Z

Makefile

+	BH_FLAG=$${BLOCK_HASH:+--block-hash $$BLOCK_HASH}; \
+	BHGT_FLAG=$${BLOCK_HEIGHT:+--block-height $$BLOCK_HEIGHT}; \
+	if [ -f scripts/generate_manifest.py ]; then \
+		$(PYTHON) scripts/generate_manifest.py $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG $< > $@; \


The script expects the positional snapshot argument before flags (./generate_manifest.py [flags]); current order passes flags first. Fix ordering: $(PYTHON) scripts/generate_manifest.py $< $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG > $@

Suggested change

$(PYTHON) scripts/generate_manifest.py $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG $< > $@; \

$(PYTHON) scripts/generate_manifest.py $< $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG > $@; \

Copilot · 2025-10-17T20:15:59Z

tests/fixtures/.gitattributes

@@ -0,0 +1 @@
+134092758.670ca68c3de580f8469677754a725e86ca72a7be381d3108569f0704a5fca327.cbor filter=lfs diff=lfs merge=lfs -text


Only the large CBOR is tracked by LFS; snapshot-small.cbor is an LFS pointer but is not covered here, leading to inconsistent local behavior (e.g., SHA256 checks on pointer text). Track all CBOR fixtures with a pattern like *.cbor or add an explicit line for snapshot-small.cbor.

Suggested change

134092758.670ca68c3de580f8469677754a725e86ca72a7be381d3108569f0704a5fca327.cbor filter=lfs diff=lfs merge=lfs -text

*.cbor filter=lfs diff=lfs merge=lfs -text

sandtreader · 2025-10-20T14:39:30Z

common/src/snapshot/mod.rs

@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: Apache-2.0


This may well be the right thing to do but let's discuss - is it implicit in the repo licence?

It's what Amaru does and I think we're on the same footing there. Also, since we want to be able to use some Amaru code, it makes sense to use the same license.

sandtreader · 2025-10-20T14:42:40Z

common/src/snapshot/error.rs

+
+/// Errors that can occur during snapshot parsing
+#[derive(Debug)]
+pub enum SnapshotError {


Nice - I've been wondering for a while whether we should standardise on error enums instead of convenient but slapdash anyhow!()

I think thiserror is the way to go since it's hidden without our libraries but generates all the nice things for Error automatically if you annotate your errors slightly.

sandtreader · 2025-10-20T14:44:33Z

common/src/snapshot/pool_params.rs

+}
+
+// Serialize implementation requires pallas_addresses which is not currently a dependency
+// TODO: Add pallas_addresses or implement Bech32 encoding differently


I think we should have everything you need in address.rs - if not, it could be added there. If it's Pallas specific, it could go in codec/

common/src/snapshot/streaming_snapshot.rs

sandtreader · 2025-10-20T14:47:59Z

common/src/snapshot/streaming_snapshot.rs

+}
+
+/// Maybe type (optional with explicit encoding)
+#[derive(Debug, Clone, PartialEq, Eq)]


Do we need this instead of Option because we want to add a custom decoder?

sandtreader · 2025-10-20T14:53:53Z

common/src/snapshot/streaming_snapshot.rs

+        let epoch = decoder.u64().context("Failed to parse epoch number")?;
+
+        // Skip blocks_previous_epoch [1] and blocks_current_epoch [2]
+        decoder.skip().context("Failed to skip blocks_previous_epoch")?;


I think we need these block counts to bootstrap the rewards calcs in AccountsState

Alright! Thanks for noting that.

What kind of information do we need from the blocks?

I've added block parsing, but all we really get are the number of blocks.

sandtreader · 2025-10-20T14:55:43Z

common/src/snapshot/streaming_snapshot.rs

+
+                // Extract rewards from rewards_and_deposit (first element of tuple)
+                let rewards = match &account.rewards_and_deposit {
+                    StrictMaybe::Just((reward, _deposit)) => *reward,


Interesting, we don't track per-account deposit at the moment - this makes me wonder whether there's some case that requires us to do so.

sandtreader · 2025-10-20T14:56:55Z

common/src/snapshot/streaming_snapshot.rs

+                    stake_address,
+                    address_state: StakeAddressState {
+                        registered: false, // Accounts are registered by SPOState
+                        utxo_value: 0, // Not available in DState, would need to aggregate from UTxOs


AccountsState module will do this for you if you just feed the UTXOState raw UTXOs

…ainder

- Implement optimized streaming with 64MB parse buffer and 16MB read chunks - Add batch processing for multiple UTXOs per buffer cycle Performance Results: - Memory: 256MB meta-data + 64MB peak (vs 2.1GB original) - Speed: 2.4M UTXOs/sec in release build (254K in debug) - Total time: 4.7s for 11.2M UTXOs (release build)

…d the order of it's elements to match Haskell Node CBOR format

buddhisthead · 2025-10-21T19:03:42Z

@sandtreader I'm inclined to merge this now so I can start working on snapshot booting. I think I've addressed all the main concerns except perhaps the question about utxo_value. Perhaps I'll just remove that element when I work on sending UTXOs on the message bus.

buddhisthead added 2 commits October 17, 2025 11:55

Update documentation and add remanining test driver and script that g…

cbf1c58

…enerates a snapshot manifest file

Copilot AI review requested due to automatic review settings October 17, 2025 19:13

Copilot AI reviewed Oct 17, 2025

View reviewed changes

buddhisthead requested a review from lowhung October 17, 2025 19:20

buddhisthead added 3 commits October 17, 2025 12:24

Fix formatting and also a comment that was misleading about the type

087606c

Fix a couple of nits from co-pilot

35e8d94

buddhisthead requested a review from Copilot October 17, 2025 20:11

buddhisthead commented Oct 17, 2025

View reviewed changes

Copilot AI reviewed Oct 17, 2025

View reviewed changes

Apply some small suggestions from co-pilot reviewer

65e5618

sandtreader reviewed Oct 20, 2025

View reviewed changes

common/src/snapshot/streaming_snapshot.rs Show resolved Hide resolved

sandtreader reviewed Oct 20, 2025

View reviewed changes

buddhisthead added 6 commits October 20, 2025 11:53

WIP to convert to streaming snapshot parser in chunks

bdc357f

WIP: read meta-data first in 256MB chunk, seek to UTXOs and ready rem…

b1e1013

…ainder

Cleanup unused code and dependency

9e4eaef

Add block pool production stats for previous and current epochs

fd7d2ac

Swapped out the hash::StakeCredential for the one in types and change…

017fbe8

…d the order of it's elements to match Haskell Node CBOR format

Replaced printlns with info tracing calls

053cad2

buddhisthead merged commit 9cd06a7 into main Oct 21, 2025
2 checks passed

github-actions bot mentioned this pull request Oct 27, 2025

📊 Weekly Status - Week of 2025-10-27 #300

Closed

SupernaviX deleted the cet-snap-parser branch October 28, 2025 17:17

github-actions bot mentioned this pull request Oct 31, 2025

📊 Weekly Status - Week of 2025-10-31 #331

Closed

		reward_account: hex::encode(&params.reward_account.0),
		pool_owners: params.owners.iter().map(\|h\| h.to_string()).collect(),

	$(PYTHON) scripts/generate_manifest.py $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG $< > $@; \
	$(PYTHON) scripts/generate_manifest.py $< $$ERA_FLAG $$BH_FLAG $$BHGT_FLAG > $@; \

		@@ -0,0 +1 @@
		134092758.670ca68c3de580f8469677754a725e86ca72a7be381d3108569f0704a5fca327.cbor filter=lfs diff=lfs merge=lfs -text

	134092758.670ca68c3de580f8469677754a725e86ca72a7be381d3108569f0704a5fca327.cbor filter=lfs diff=lfs merge=lfs -text
	*.cbor filter=lfs diff=lfs merge=lfs -text

feat: Adds a streaming snapshot paser #259

feat: Adds a streaming snapshot paser #259

Uh oh!

Conversation

buddhisthead commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

buddhisthead commented Oct 17, 2025 •

edited

Loading