Enable alerting on OOMs

[rocketraman]

The next version of kube-state-metrics should have a metric called kube_pod_container_status_last_terminated_reason. See kubernetes/kube-state-metrics#535.

Can an OOM alert based on this metric be setup in this mixin?

[metalmatze]

Yes. I think this should be possible and the right place is within this repository.

[BrianChristie]

Currently it appears to be difficult to create a good alert on Out Of Memory crash looping pods, because kube_pod_container_status_last_terminated_reason is a binary gauge, but should be a counter.

This has been brought up with sig-node and sig-instrumentation, and it appears a fix requires changes to the Kubelet to expose this counter (as well as a Pod event). This will probably take a significant period of time to get done and merged, see: kubernetes/kubernetes#69676 (comment) .

As far as I can see, in the interim the best that could be done is an alert if the Pod has a status of OOMKilled a single time, but there's no way to only alert based on the Pod crashing with that status on multiple occasions over a period of time.

[wking]

#510 and #800 are both opened to poke at this. I haven't read either closely enough to compare and contrast, but thought I'd drop some cross-references.

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!