TQ: Add support for "Expunge" messages #8874
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR builds on #8801 |
andrewjstone
force-pushed
the
tq-expunged
branch
2 times, most recently
from
ddfc395 to
665ef30
Compare
andrewjstone
changed the title
When a node requests a share for an old configuration it will receive an `Expunged` message if the share request receiving node has a later committed configuration where the node is not a member. When the `Expunged` message is receieved, the expunged node persists this fact and stops replying to peer messages. We also fix a bug in the test where nexus wasn't actually committing configurations at all. So far invariants really slow the test down. On my machine checking the invariants after every applied event makes the test take ~120s. Without invariant checking, it takes about 14s. I definitely want to add more invariants for correctness, but maybe not check them when they aren't applicable. Some may become postconditions on certain events instead.
andrewjstone
force-pushed
the
tq-expunged
branch
from
0ca88b1 to
d85c5d9
Compare
andrewjstone
added a commit
that referenced
this pull request
Aug 28, 2025
This builds on #8874 To make using using tqdb easier, we put all necessary information inside events. This works because of deterministic replay using the same infrastucture as the proptest itself. To ensure we are truly deterministically replaying we must assert that any Event in our log has actually been generated when we go to apply it in tqdb. In the common case case this is equivalent to asserting that the `DeliveredEnvelope` is actually the same as the one pulled off the `bootstrap_network` vec during runs. In order to guarantee this a few changes needed to be made: 1. Determinism exists, except for in the crypto code because we don't seed the various random number generators, and therefore different key shares and rack secrets get generated in different runs. We could seed them, but then this makes it possible that we accidentally seed them in production code. More importantly for our current purposes, though, is that it's tedious and unnecessary. Instead we just implement comparison methods for messages that ignore things like key shares. This works fine because if the shares are not self-consistent with the rack secret and the parameters such as threshold that we don't change the crypto code itself will fail immediately. 2. We had a few actions that generated multiple events. Unfortunately, applying these events would result in mutating the test networks multiple times, resulting in a difference between which message was recorded and which was actually pulled out to be delivered. This was fixed by changing each action to only do a single thing at a time, like deliver one envelope or commit a configuration at one node. This also allows for finer grain interleaving and matches better with our TLA+ spec/model checking. This was observable by looking at the event logs. The change to an action typically generating a single event means, however, that we need to generate more actions per run to get the equivalent functionality. A single action won't result in N commits or N envelopes delivered. Therefore, each test run now needs to have significantly more actions generated. Unfortunately this can make test runs even longer. In order to help alleviate this we made three other changes: 1. We change the invariant checks to only look at nodes that could possibly be mutated by an event application. We call these the "affected nodes". This prevents looping over every node in each check. 2. We reduce the member universe, and hence the size of the state space. 3. We reduce the total number of test cases per run.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a node requests a share for an old configuration it will receive
an
Expungedmessage if the share request receiving node has a latercommitted configuration where the node is not a member. When the
Expungedmessage is receieved, the expunged node persists this factand stops replying to peer messages.
We also fix a bug in the test where nexus wasn't actually committing
configurations at all.
So far invariants really slow the test down. On my machine checking the
invariants after every applied event makes the test take ~120s. Without
invariant checking, it takes about 14s.
I definitely want to add more invariants for correctness, but maybe not
check them when they aren't applicable. Some may become postconditions
on certain events instead.