Feature request: Drain on Rebalance Recommendation Notification

[dongheeJeong]

Spot interruption termination Notification occurs in two minutes before instance termination. Two minutes may not sufficient time to drain node on some circumstance. (e.g. When new worker node creation needed when all existing worker nodes are too busy to accept pods).

If option to drain on Rebalance Recommendation Notification(which arrives sooner than interruption termination Notification) provided, it would offer more time to pod to move to the other node.

Is there a plan on the roadmap to support something like this?

[haugenj]

We have vague plans of doing a refactor at some point to allow for better customization for any event type, this would help in other cases too such as #361 for scheduled events. But for now draining on a rebalance recommendation is not something we plan on supporting in a configuration option.

I'm not sure I understand your situation, do you mean the node is unable to completely drain within two minutes? Unless the drain is failing to complete I don't see how draining sooner than the spot itn will increase the speed of provisioning a new node

[dongheeJeong]

I use Cluster Autoscaler which add new worker node when there are pods that failed to run in the cluster due to insufficient resources. When drain needed and if there are insufficient resources in cluster for moving pods, new Worker node would provisioned (by Cluster Autoscaler) before starting drain. So, it takes time of new node provisioning + drain.

I do not mean that time of new node provisioning + drain is always more than 2 minutes, but it could take longer than general situation.

[opalmer]

But for now draining on a rebalance recommendation is not something we plan on supporting in a configuration option.

@haugenj, would you be open to an MR to add this or would you consider supporting this feature?

Right now, based on what I've observed in a couple of clusters, when using the spot termination handler to handle rebalancing notifications the handler picks up the event and cordons the node. It does not drain the node to remove pods. In this scenario the next thing the ASG does is launch a new instance and then finally terminate the instance that got the rebalance recommendation. The AWS termination handler does not drain the node as a result because it never receives a "spot instance interruption" signal at least based on the logs I've seen from the service.

Essentially, operating in this mode currently is not much better than running without the spot termination handler because it it causes workloads to be left on the node until termination. It would be awesome if the termination handler would support this as a feature.

[haugenj]

👋 @opalmer

Thanks for your patience, I wanted to talk to the team about this before getting back to you. I think it'll take us a bit longer to settle on our approach, so in the meantime you can go ahead and add an option for draining on a rebalance recommendation event. If you need any help with it just open a draft pr and I'll help out. Thanks!

[trutx]

I have a similar use case. We use Cluster Autoscaler to scale up and down our ASGs. Our ASGs use the new Capacity Rebalance feature that will proactively spin up a new spot instance when another one is at "elevated risk of interruption" and gets the capacity rebalance signal. This doesn't necessarily mean the existing instance will be interrupted, but no matter what we already got a new instance so we want to gracefully get rid of the old one by draining it. Once drained, Cluster Autoscaler will see that node as not used and will terminate it.

I went ahead and pushed some changes to implement this in a non-breaking way. However, I agree with @haugenj that the code needs some refactoring, i.e. to add the ability to drain and/or cordon on a per-monitor basis.

[bwagner5]

released