Skip to content

Conversation

@marccarre
Copy link
Contributor

@marccarre marccarre commented Jul 6, 2017

Identified by @bboreham, all credits go to him 👍 -- #482.

This is a quick change to (hopefully) help identify an issue with alerts periodically going back to pending, and not firing; and reduce the likelihood of the race condition interfering with this investigation.

Alternatives considered:

  • add half-an-interval to, but this would still lead to a race 50% of the time,
  • add one second, but this would still lead to a race every 15 seconds (the current default interval),
  • randomise, but could actually make it harder to identify issues,
  • add 10 ms (current state): this should be long enough to avoid a race condition, and short enough to maximise the time before the two schedules overlap again (if the race is happening 100% of the time every 15 seconds, then, with 10 ms offset, it would only happen 1/1500 of the time).

Note that the first interval is used by a timer, and the second in a Defer instruction, so one would think the execution of the scheduled work would add enough noise to avoid races (?).

A proper fix and the implementation of TODO: Separate configuration for polling interval. should follow, once the incident is resolved.

@marccarre marccarre requested review from bboreham and jml July 6, 2017 10:02
@marccarre marccarre force-pushed the periodicity-for-eval-and-scheduling branch from eee965d to 66fad0c Compare July 6, 2017 10:32
@marccarre marccarre merged commit 5e2c7f7 into master Jul 6, 2017
@marccarre marccarre deleted the periodicity-for-eval-and-scheduling branch July 6, 2017 10:46
bboreham added a commit that referenced this pull request Nov 23, 2017
…ts scheduling. (#484)"

No longer needed after 3281b7d which
fixed the problem this was working around.

This reverts commit 5e2c7f7.
bboreham added a commit that referenced this pull request Dec 5, 2017
…ts scheduling. (#484)" (#614)

No longer needed after 3281b7d which
fixed the problem this was working around.

This reverts commit 5e2c7f7.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants