processbot is unresponsive to pull request comments (25 Mar 2022 incident report)

[joao-paulo-parity]

On 25 Mar 2022 we received multiple reports of the GitHub App being unresponsive.

The main cause of unresponsiveness stems from the app apparently not receiving issue_comment.created events. This is corroborated by

• The feedback from https://github.com/paritytech/internal_it/issues/67 where we were not able to find any recent deliveries of issue_comment.created
• One experiment where a comment was created and deleted, but the app only got the issue_comment.deleted event

At this point I still have no clue why issue_comment.created events are apparently not reaching the server, but other events are. It might have to do with GitHub's recent instability spree.

I've opened a GitHub Support ticket in the meantime but, until when we get some semblance of a clue, this is unactionable.

[joao-paulo-parity]

As confirmed by the GitHub Support Ticket, it was a failure on GitHub's side

GitHub (GitHub Support)

Mar 25, 2022, 6:17 PM UTC

Hi João,

Thanks for reaching out!

Our engineering teams did spot this issue and I understand a fix has just been deployed to resolve the problem. Could you please let us know if you continue to see cases of these missing webhooks event?

Regards,

Oluwaseun
GitHub Support

Now the bot is working again with no changes made on our side: paritytech/substrate#11125 (comment)