Reduce likelihood of a race between rules evaluation and alerts scheduling #484
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Identified by @bboreham, all credits go to him 👍 -- #482.
This is a quick change to (hopefully) help identify an issue with alerts periodically going back to pending, and not firing; and reduce the likelihood of the race condition interfering with this investigation.
Alternatives considered:
Note that the first interval is used by a timer, and the second in a
Deferinstruction, so one would think the execution of the scheduled work would add enough noise to avoid races (?).A proper fix and the implementation of
TODO: Separate configuration for polling interval.should follow, once the incident is resolved.