Only notify ready global checkpoint listeners #33690
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we add a global checkpoint listener, it is also carries along with it a value that it thinks is the current global checkpoint. This value can be above the actual global checkpoint on a shard if the listener knows the global checkpoint from another shard copy (e.g., the primary), and the current shard copy is lagging behind. Today we notify the listener whenever the global checkpoint advances, regardless if it goes above the current global checkpoint known to the listener. This commit reworks this implementation. Rather than thinking of the value associated with the listener as the current global checkpoint known to the listener, we think of it as the value that the listener is waiting for the global checkpoint to advance to (inclusive). Now instead of notifying all waiting listeners when the global checkpoint advances, we only notify those that are waiting for a value not larger than the actual global checkpoint that we advanced to.
jasontedor
added
review
v7.0.0
:Distributed Indexing/Distributed
Collaborator
|
Pinging @elastic/es-distributed |
dnhatn
approved these changes
Sep 14, 2018
server/src/main/java/org/elasticsearch/index/shard/GlobalCheckpointListeners.java
Show resolved
Hide resolved
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Sep 14, 2018
* master: (24 commits) Only notify ready global checkpoint listeners (elastic#33690) Don't count hits via the collector if the hit count can be computed from index stats. (elastic#33701) Expose retries for CCR fetch failures (elastic#33694) Test fix - Graph vertices could appear in different orders based on map insertion sequence (elastic#33709) Structured audit logging (elastic#31931) Core: Add DateFormatter interface for java time parsing (elastic#33467) [CCR] Check whether the rejected execution exception has the shutdown flag set (elastic#33703) Mute ClusterDisruptionIT#testSendingShardFailure Revert "Mute FullClusterRestartSettingsUpgradeIT" Adjust BWC version on settings upgrade test (elastic#33650) [ML] Allow overrides for some file structure detection decisions (elastic#33630) Adapt skip version for doc_values format deprecation [TEST] wait for no initializing shards [Docs] Minor fix in `has_child` javadoc comment (elastic#33674) Mute FullClusterRestartSettingsUpgradeIT [Kerberos] Add realm name & UPN to user metadata (elastic#33338) [TESTS] Disable specific locales for RestrictedTrustManagerTest (elastic#33299) SQL: Return functions in JDBC driver metadata (elastic#33672) SCRIPTING: Move terms_set Context to its Own Class (elastic#33602) AwaitsFix testRestoreMinmal ...
jasontedor
added a commit
that referenced
this pull request
Sep 14, 2018
When we add a global checkpoint listener, it is also carries along with it a value that it thinks is the current global checkpoint. This value can be above the actual global checkpoint on a shard if the listener knows the global checkpoint from another shard copy (e.g., the primary), and the current shard copy is lagging behind. Today we notify the listener whenever the global checkpoint advances, regardless if it goes above the current global checkpoint known to the listener. This commit reworks this implementation. Rather than thinking of the value associated with the listener as the current global checkpoint known to the listener, we think of it as the value that the listener is waiting for the global checkpoint to advance to (inclusive). Now instead of notifying all waiting listeners when the global checkpoint advances, we only notify those that are waiting for a value not larger than the actual global checkpoint that we advanced to.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we add a global checkpoint listener, it is also carries along with it a value that it thinks is the current global checkpoint. This value can be above the actual global checkpoint on a shard if the listener knows the global checkpoint from another shard copy (e.g., the primary), and the current shard copy is lagging behind. Today we notify the listener whenever the global checkpoint advances, regardless if it goes above the current global checkpoint known to the listener. This commit reworks this implementation. Rather than thinking of the value associated with the listener as the current global checkpoint known to the listener, we think of it as the value that the listener is waiting for the global checkpoint to advance to (inclusive). Now instead of notifying all waiting listeners when the global checkpoint advances, we only notify those that are waiting for a value not larger than the actual global checkpoint that we advanced to.