generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 278
Add AWS Health event support for QP mode #510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
b113a54
Add AWS Health event support for QP mode
AustinSiu a3ee476
Update QP Scheduled Change event drain time, README, taint type
AustinSiu 8efb322
Add AWS Health event support for QP mode
AustinSiu 814a35a
Update QP Scheduled Change event drain time, README, taint type
AustinSiu d621abd
Update AWS Health event handler names, update e2e run command with AS…
AustinSiu f0997c2
Add AWS Health event support for QP mode
AustinSiu 9df853b
Update QP Scheduled Change event drain time, README, taint type
AustinSiu acfa0bb
Update AWS Health event handler names, update e2e run command with AS…
AustinSiu 23a7c6f
Improve comments related to Health events and scheduled changes
AustinSiu 5ad7ffc
Reduce scope of several variables
AustinSiu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"). You may | ||
| // not use this file except in compliance with the License. A copy of the | ||
| // License is located at | ||
| // | ||
| // http://aws.amazon.com/apache2.0/ | ||
| // | ||
| // or in the "license" file accompanying this file. This file is distributed | ||
| // on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| // express or implied. See the License for the specific language governing | ||
| // permissions and limitations under the License. | ||
|
|
||
| package sqsevent | ||
|
|
||
| import ( | ||
| "encoding/json" | ||
| "fmt" | ||
| "time" | ||
|
|
||
| "github.com/aws/aws-node-termination-handler/pkg/monitor" | ||
| "github.com/aws/aws-node-termination-handler/pkg/node" | ||
| "github.com/aws/aws-sdk-go/service/sqs" | ||
| "github.com/rs/zerolog/log" | ||
| ) | ||
|
|
||
| /* Example AWS Health Scheduled Change EC2 Event: | ||
| { | ||
| "version": "0", | ||
| "id": "7fb65329-1628-4cf3-a740-95fg457h1402", | ||
| "detail-type": "AWS Health Event", | ||
| "source": "aws.health", | ||
| "account": "account id", | ||
| "time": "2016-06-05T06:27:57Z", | ||
| "region": "us-east-1", | ||
| "resources": ["i-12345678"], | ||
| "detail": { | ||
| "eventArn": "arn:aws:health:region::event/id", | ||
| "service": "EC2", | ||
| "eventTypeCode": "AWS_EC2_DEDICATED_HOST_NETWORK_MAINTENANCE_SCHEDULED", | ||
| "eventTypeCategory": "scheduledChange", | ||
| "startTime": "Sat, 05 Jun 2016 15:10:09 GMT", | ||
| "eventDescription": [{ | ||
| "language": "en_US", | ||
| "latestDescription": "A description of the event will be provided here" | ||
| }], | ||
| "affectedEntities": [{ | ||
| "entityValue": "i-12345678", | ||
| "tags": { | ||
| "stage": "prod", | ||
bwagner5 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "app": "my-app" | ||
| } | ||
| }] | ||
| } | ||
| } | ||
| */ | ||
|
|
||
| // AffectedEntity holds information about an entity that is affected by a Health event | ||
| type AffectedEntity struct { | ||
| EntityValue string `json:"entityValue"` | ||
| } | ||
|
|
||
| // ScheduledChangeEventDetail holds the event details for AWS Health scheduled EC2 change events from Amazon EventBridge | ||
| type ScheduledChangeEventDetail struct { | ||
| EventTypeCategory string `json:"eventTypeCategory"` | ||
| Service string `json:"service"` | ||
| AffectedEntities []AffectedEntity `json:"affectedEntities"` | ||
| } | ||
|
|
||
| func (m SQSMonitor) scheduledEventToInterruptionEvents(event *EventBridgeEvent, message *sqs.Message) []InterruptionEventWrapper { | ||
| scheduledChangeEventDetail := &ScheduledChangeEventDetail{} | ||
| interruptionEventWrappers := []InterruptionEventWrapper{} | ||
|
|
||
| if err := json.Unmarshal(event.Detail, scheduledChangeEventDetail); err != nil { | ||
| return append(interruptionEventWrappers, InterruptionEventWrapper{nil, err}) | ||
| } | ||
|
|
||
| if scheduledChangeEventDetail.Service != "EC2" { | ||
| err := fmt.Errorf("events from Amazon EventBridge for service (%s) are not supported", scheduledChangeEventDetail.Service) | ||
| return append(interruptionEventWrappers, InterruptionEventWrapper{nil, err}) | ||
| } | ||
|
|
||
| if scheduledChangeEventDetail.EventTypeCategory != "scheduledChange" { | ||
| err := fmt.Errorf("events from Amazon EventBridge with EventTypeCategory (%s) are not supported", scheduledChangeEventDetail.EventTypeCategory) | ||
| return append(interruptionEventWrappers, InterruptionEventWrapper{nil, err}) | ||
| } | ||
|
|
||
| for _, affectedEntity := range scheduledChangeEventDetail.AffectedEntities { | ||
| nodeInfo, err := m.getNodeInfo(affectedEntity.EntityValue) | ||
| if err != nil { | ||
| interruptionEventWrappers = append(interruptionEventWrappers, InterruptionEventWrapper{nil, err}) | ||
| continue | ||
| } | ||
|
|
||
| // Begin drain immediately for scheduled change events to avoid disruptions in cases such as degraded hardware | ||
bwagner5 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| interruptionEvent := monitor.InterruptionEvent{ | ||
| EventID: fmt.Sprintf("aws-health-scheduled-change-event-%x", event.ID), | ||
| Kind: SQSTerminateKind, | ||
| AutoScalingGroupName: nodeInfo.AsgName, | ||
| StartTime: time.Now(), | ||
| NodeName: nodeInfo.Name, | ||
| InstanceID: nodeInfo.InstanceID, | ||
| Description: fmt.Sprintf("AWS Health scheduled change event received. Instance %s will be interrupted at %s \n", nodeInfo.InstanceID, event.getTime()), | ||
| } | ||
| interruptionEvent.PostDrainTask = func(interruptionEvent monitor.InterruptionEvent, n node.Node) error { | ||
| if errs := m.deleteMessages([]*sqs.Message{message}); errs != nil { | ||
| return errs[0] | ||
| } | ||
| return nil | ||
| } | ||
| interruptionEvent.PreDrainTask = func(interruptionEvent monitor.InterruptionEvent, n node.Node) error { | ||
| if err := n.TaintScheduledMaintenance(interruptionEvent.NodeName, interruptionEvent.EventID); err != nil { | ||
| log.Err(err).Msgf("Unable to taint node with taint %s:%s", node.ScheduledMaintenanceTaint, interruptionEvent.EventID) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| interruptionEventWrappers = append(interruptionEventWrappers, InterruptionEventWrapper{&interruptionEvent, nil}) | ||
| } | ||
|
|
||
| return interruptionEventWrappers | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason this event pattern isn't more specific? Should this be limited to EC2 scheduled changes since the code ignores all other events?
{ "source": [ "aws.health" ], "detail-type": [ "AWS Health Event" ], "detail": { "service": [ "EC2" ], "eventTypeCategory": [ "scheduledChange" ] } }Apologies for commenting on a closed PR. I'm trying to upgrade NTH and need to understand the new event rules that should be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should also add "region": ["us-east-1"] or whatever region you're using for the cluster. Since AWS Health events are global and at least want the filter to EC2 scheduled changes for the region you're using.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gabegorelick can you open an issue with your thoughts on how we should constrain the rule? Also, if you'd like to PR an update to the README and run some tests, that would be great!