Rebalance Notification not Triggered in IMDS Mode

[MattyKuzyk]

Setup

Termination handler app version 1.13 is deployed successfully via Helm with the following options:

2021/04/21 11:14:29 DBG Trying to get token from IMDSv2
2021/04/21 11:14:29 DBG Got token from IMDSv2
2021/04/21 11:14:29 INF Startup Metadata Retrieved metadata={"accountId":"REDACTED","availabilityZone":"us-west-2c","instanceId":"REDACTED","instanceType":"t3.large","localHostname":"REDACTED","privateIp":"REDACTED","publicHostname":"","publicIp":"","region":"us-west-2"}
2021/04/21 11:14:29 INF aws-node-termination-handler arguments:
	dry-run: false,
	node-name: REDACTED,
	metadata-url: http://169.254.169.254,
	kubernetes-service-host: 172.20.0.1,
	kubernetes-service-port: 443,
	delete-local-data: true,
	ignore-daemon-sets: true,
	pod-termination-grace-period: 40,
	node-termination-grace-period: 300,
	enable-scheduled-event-draining: false,
	enable-spot-interruption-draining: true,
	enable-sqs-termination-draining: false,
	enable-rebalance-monitoring: true,
	enable-rebalance-draining: true,
	metadata-tries: 3,
	cordon-only: false,
	taint-node: true,
	json-logging: false,
	log-level: DEBUG,
	webhook-proxy: ,
	webhook-headers: <not-displayed>,
	webhook-url: ,
	webhook-template: <not-displayed>,
	uptime-from-file: ,
	enable-prometheus-server: false,
	prometheus-server-port: 9092,
	aws-region: us-west-2,
	queue-url: ,
	check-asg-tag-before-draining: false,
	managed-asg-tag: aws-node-termination-handler/managed,
	aws-endpoint: ,

2021/04/21 11:14:29 INF Started watching for interruption events
2021/04/21 11:14:29 INF Kubernetes AWS Node Termination Handler has started successfully!
2021/04/21 11:14:29 INF Started watching for event cancellations
2021/04/21 11:14:29 INF Started monitoring for events event_type=SPOT_ITN
2021/04/21 11:14:29 INF Started monitoring for events event_type=REBALANCE_RECOMMENDATION

Reproduction steps

To test, on ASG inus-west-2c private subnet, starting with 6 desired nodes (max 20):

1. Deploy ~6 pod replicas with pod anti affinity to spread across nodes, and Pod Disruption Budget w/ maxUnavailable: 1
2. Edit ASG in AWS console to add us-west-2a and us-west-2b AZs to trigger rebalancing
3. Rebalance is triggered, following ASG activity occurs multiple times (in AWS console):

At 2021-04-20T18:54:02Z a user request added the zones us-west-2a, us-west-2b to this AutoScalingGroup and the group may require rebalancing. At 2021-04-20T18:54:26Z availability zones us-west-2a us-west-2b us-west-2c had 0 0 6 instances respectively. An instance was launched to aid in balancing the group's zones.

At 2021-04-20T18:54:27Z instances were launched to balance instances in zones us-west-2a us-west-2b us-west-2c with other zones resulting in more than desired number of instances in the group. At 2021-04-20T18:55:08Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 10 to 6. At 2021-04-20T18:55:09Z instance REDACTED was selected for termination. At 2021-04-20T18:55:09Z instance REDACTED was selected for termination. At 2021-04-20T18:55:09Z instance REDACTED was selected for termination. At 2021-04-20T18:55:09Z instance REDACTED was selected for termination.

Result

No additional aws-node-termination-handler logs other than the startup logs from the Setup step. Neither taints nor cordons applied to nodes that were selected for termination by AWS. Nodes ungracefully terminated by AWS without draining pods.

PDB reports no disruptions (i.e the pods were not drained)

[MattyKuzyk]

I see now that there may have been confusion in my understanding of AZ rebalancing in AWS. I can see in this doc that a "rebalance recommendation" is something for Spot instances only.

However, in this doc, it mentions AZ rebalancing for ASGs. This second scenario is what applies to me, as I am not using Spot instances.

Am I to understand that aws-node-termination-handler only supports Spot instance AZ rebalance recommendations, not generic ASG EC2 AZ rebalancing?

[haugenj]

Node Termination Handler supports AZ rebalancing, but only in Queue Processor mode

[MattyKuzyk]

Node Termination Handler supports AZ rebalancing, but only in Queue Processor mode

Is that through the ASG Lifecycle Hooks feature, separate from EC2 Instance Rebalance Recommendation in the README?

[haugenj]

Yeah, the Rebalance Recommendation is an early indicator for Spot instances to say you're more likely to get interrupted soon. For AZ rebalances the instances are just terminated, so the lifecycle hook + eventbridge rule for EC2 Instance-terminate Lifecycle Action should handle that case for OD instances

[MattyKuzyk]

Yeah, the Rebalance Recommendation is an early indicator for Spot instances to say you're more likely to get interrupted soon. For AZ rebalances the instances are just terminated, so the lifecycle hook + eventbridge rule for EC2 Instance-terminate Lifecycle Action should handle that case for OD instances

Makes sense. May I suggest some doc enhancement in the README to make this distinction of rebalancing more clear. The doc linked for ASG AZ Rebalance specifically mentions AZ rebalancing for EC2, and it is easy to confuse for the EC2 Instance Rebalance Recommendation which is only for spot instances. Multiple engineers on my team thought that this tool would serve our usecase because of this.

I'll be watching the Queue processor mode with great interest :)

[haugenj]

@MattyKuzyk 👍 I'll take that up for sure

Mind me asking what your use case is? Is the problem that we're still designating Queue Processor mode as 'beta' ?

[MattyKuzyk]

The usecase is trying to handle interruptions to application pods when ASG notices an AZ imbalance. Annoyingly, when cluster-autoscaler handles our scale down it happens gracefully, but it is not graceful when AWS handles it via AZ rebalance.

Yes, we are skittish for now about going with Queue Processor because it's marked Beta.