Do not mark late success as "okay"

[evanpurkhiser]

Currently there exists a scenario where you may start a job far before the expected check in time, and if it completes near the checkin time OR after the check in time (after it was marked missed) the status of the monitor will be updated to successful.

Imagine a crontab monitor configured as 0 * * * *. This cron monitor should run on the hour every hour. You have also configured a 5 minute checkin margin.

In this scenario...

1. A checkin has happened at 12:40 (a good 20 minutes before expected)
2. At 01:05 the expected checkin at 01:00 is marked as missed since no checkin happened within the 5 minute checkin margin.
3. Finally the job started at 12:40 completes, and now the monitor is marked as having been "successful"

Existing world

This is not quite right, as the monitor actually started much earlier than it should have. If it had run faster and completed BEFORE 1:00 it would not have been accounted for the 1:00 checkin at all.

Proposed new world

We should not update a monitors due to checkins that started before the expected checkin time.

[evanpurkhiser]

Example of some real usage confusion because of this #42283 (comment)

[getsantry]

Routing to @getsentry/product-owners-crons for triage ⏲️

[evanpurkhiser]

This is somewhat related to #44387