-
Notifications
You must be signed in to change notification settings - Fork 412
Make full-scan/sync flow easier to reason about. #1838
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make full-scan/sync flow easier to reason about. #1838
Conversation
LLFourn
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left a couple of documentation comments but this LGTM.
64b4f14 to
6bb7afa
Compare
ValuedMammal
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approach ACK
abf2bc1 to
8682487
Compare
LLFourn
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 8682487
nymius
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 8682487
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: typo in commit message.
This may, if we introduce new fields to
TxUdpate, they can be minor
non-breaking updates.
Otherwise LGTM.
68f668b to
61bcd12
Compare
|
Rebased on #1860 and removed the changes to |
61bcd12 to
0214401
Compare
If we introduce new fields to `TxUpdate`, they can be minor non-breaking updates.
* Change `TxUpdate::seen_ats` to be a `HashSet<(Txid, u64)>` so we can
introduce multiple timestamps per tx. This is useful to introduce both
`first_seen` and `last_seen` timestamps to `TxGraph`. This is also a
better API for chain-sources as they can just insert timestamps into
the field without checking previous values.
* Change sync/full-scan flow to have the request structure introduce the
sync time instead of introducing the timestamp when apply the
`TxUpdate`. This simplifies the `apply_update{_at}` logic and makes
`evicted_at` easier to reason about (in the future).
0214401 to
800f358
Compare
notmandatory
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 800f358
Note to other reviewers: bdk_wallet changes will be added to a new PR once the new bdk_wallet repo is ready. But the PR won't be merged until we're ready for a bdk_wallet 2.0.
e125014 feat(examples): Change `example_wallet_rpc` to detect evictions. (志宇) 12b00e0 feat(wallet): Add method `apply_evicted_txs` (valued mammal) a247215 deps!: update bdk_chain to 0.22.0 (valued mammal) f882945 ci: Unpin some dependencies (valued mammal) Pull request description: This PR updates `bdk_chain` dependency to the latest version 0.22.0 ### Notes to the reviewers This PR doesn't make use of `CanonicalizationParams` for any advanced use cases - instead we rely on the default params for all tx-graph queries. fixes #202 fixes #224 ### Changelog notice - deps: bump `bdk_chain` to 0.22.0 - deps: bump `bdk_file_store` to 0.20.0 #### Added - Added `start_sync_with_revealed_spks_at` - Added `start_full_scan_at` - Added `apply_evicted_txs` #### Changed - `start_sync_with_revealed_spks` is feature gated by "std", as it uses the system time to provide the time of the sync and also the tx last-seen. - `apply_update` no longer requires std, as the timestamp is provided in the sync / full scan request - `start_full_scan` is also gated behind "std" - `start_sync_*` now includes the expected SPK history (via `list_expected_spk_txids`) #### Removed - Removed `apply_update_at` #### BREAKING - `bdk_core::TxUpdate` is non-exhaustive bitcoindevkit/bdk#1838 - For the optional file_store feature, the Load variant of the `FileStoreError` is changed to hold a new inner type bitcoindevkit/bdk#1684 #### Changes to persisted data The `tx_graph::ChangeSet` gained a new default-able field `last_evicted` to persist the fact that a tx is no longer part of the canonical chain. ### Checklists #### All Submissions: * [x] I've signed all my commits * [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md) * [x] I ran `cargo fmt` and `cargo clippy` before committing * [x] I've added docs for the new feature * [x] This pull request breaks the existng API * [x] I'm linking the issue being fixed by this PR ACKs for top commit: notmandatory: ACK e125014 Tree-SHA512: 91f77e4268aac0aaee2503a3fdca13754fc291159b9f37b0997be8c011b02b8ac16b4b6a7edbbb731005c83b37d438666fb803dbe18a6b9496b11f7e3e1ab155
Description
#1811 introduces the
evicted-atconcept. While adding this toTxUpdate, I realized there were some shortcomings in our full-scan & sync flow:TxUpdateto convey updates cannot choose to add transactions without aseen_attimestamp as theTxGraph::apply_update_atlogic adds this timestamp for all unanchored txs if theseen_atparameter is specified. Purposefully adding uncanonical txs is useful for wallets that want to know about replaced txs.TxGraph::apply_update_atlogic is hard to reason about.TxUpdate::seen_atsalready has theseen_attimestamp per tx, butapply_update_at()also takes in aseen_attimestamp`.TxUpdate::seen_atscurrently forces us to only have oneseen-atper tx as it's a map oftxid -> seen_at. However, in the future we want to add afirst-seentimestamp toTxGraphwhich is basically the earliestseen-attimestamp introduced so we may want to have multipleseen_ats per tx. The other issue is if we mergeTxUpdates, we will loose data. I.e. we can only keep thelast_seenor thefirst_seen.These problems were exacerbated when introducing
evicted-at. In the old design, the chain-source has no concept of sync time (as sync time was introduced after-the-fact when applying theTxUpdate). Therefore the introducedTxUpdate::evictedfield was aHashSet<Txid>(no timestamps) and relied on theTxGraph::apply_upate_atlogic to introduce theevicted-attimestamp. All this makes the sync logic harder to understand. What happens if theseen_atinput isNone? What happens if updates were applied out of order? What happens when we mergeTxUpdatesbefore applying?The following changes are made in this PR to simplify the sync/full-scan logic to be easier to understand and robust:
sync_timeis added to theSyncRequestandFullScanRequest.sync_timeis mandatory and is added as an input ofbuilder(). If thestdfeature is enabled, thebuilder_now()is available which uses the current timestamp. The chain source is now responsible to add thissync_timetimestamp asseen_atfor mempool txs. Non-canonical and non-anchored txs can be added without theseen_attimestamp.TxUpdate::seen_atsis now aHashSetof(Txid, u64). This allows multipleseen_ats per tx. This is also a much easier to use API as the chain source can just insert into thisHashSetwithout checking previous data.TxGraph::apply_update_atis removed as we no longer needs to introduce a fallbackseen_attimestamp after-the-fact.TxGraph::apply_updateis no longerstd-only and the logic of applying updates is greatly simplified.Additionally,
TxUpdateis updated to be#[non_exhaustive]for backwards-compatibility protection.Notes to the reviewers
This is based on #1837. Merge that first.
These are breaking changes to
bdk_core. It needs to be breaking to properly fix all the issues.As mentioned by @notmandatory,
bdk_walletchanges will be added to a new PR once the newbdk_walletrepo is ready. But the PR won't be merged until we're ready for abdk_wallet2.0.Changelog notice
FullScanRequest::builderandSyncRequest::buildermethods to depend onfeature = "std". This is because requests now have astart_time, instead of specifying aseen_atwhen applying the update.FullScanRequest::builder_atandSyncRequest::builder_atmethods which are the non-std version of the..Request::buildermethods.TxUpdate::seen_atsfield to be aHashSetof(Txid, u64). This allows a single update to have multipleseen_ats per tx.TxUpdateto benon-exhaustive.apply_update_atas we no longer need to apply with a timestamp after-the-fact.Checklists
All Submissions:
cargo fmtandcargo clippybefore committingNew Features:
* [ ] I've added tests for the new featureNo tests needed as it's more of a refactor.