Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CFN: Add resilient service rolling update mechanism #5955

Merged
merged 1 commit into from
Jan 23, 2024
Merged

CFN: Add resilient service rolling update mechanism #5955

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions aws-cfn/cross-service/resilient-workflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,11 @@ aws elbv2 describe-target-health --target-group-arn "arn:aws:elasticloadbalancin

All stack outputs:

| OutputKey | OutputValue | Usage |
| --------- | ----------------------------------------- | ------------------------------------------------- |
| `LB` | The DNS Name of the primary load balancer | `curl` or Browser |
| OutputKey | OutputValue | Usage |
| --------- | ------------------------------------------ | ------------------------------------------------- |
| `LB` | The DNS Name of the primary load balancer | `curl` or Browser |
| `Key` | The ID of a .pem format private key in SSM | `ssh` after downloading from SSM |
| `TGArn` | The ARN of the target group of instances | Check various additional information from the CLI |
| `TGArn` | The ARN of the target group of instances | Check various additional information from the CLI |

#### Demonstrate resiliency

Expand All @@ -182,6 +182,7 @@ aws cloudformation update-stack \

1. **Initial state: healthy** — Sends requests to the endpoint to get recommendations and verify that instances
are healthy.

2. **Broken dependency** — Sets a parameter that specifies a nonexistent DynamoDB table name. This simulates a
failure of the recommendation service. Requests for recommendations now return a failure
code. All instances still report as healthy because they only implement shallow health checks. For this
Expand Down Expand Up @@ -231,7 +232,8 @@ aws cloudformation update-stack \

Using the AWS Management Console, open the CloudFormation page. Navigate to the `resilience-demo` stack. Choose the `Resources`
tab. Find the `DocExampleRecommendationServiceTargetGroup` line. Choose the `Physical Resource ID` link.
From this EC2 page, find the list of instances in the target group. Select one and navigate to it. Choose `Actions`, `Terminate instance`. See EC2 terminate the instance, and watch the Auto Scaling group start a new instance.
From this EC2 page, find the list of instances in the target group. Select one and navigate to it. Choose `Actions`, `Terminate instance`.
See EC2 terminate the instance, and watch the Auto Scaling group start a new instance.

7. **Fail open** — Sets the table name parameter so the recommendations service fails for all instances.
Because all instances are using deep health checks, they all report as unhealthy. In this
Expand All @@ -242,6 +244,10 @@ aws cloudformation update-stack \
Edit `params.json`. Add a new entry with `ParameterKey` as `SSMTableName` and `ParameterValue` as `unknown`.
After updating, the service should report unhealthy but return static responses.

8. **Rolling Update** If necessary, you can trigger rolling updates to all instances by changing the Launch Template.
To change a non-functional aspect of the Launch Template, which will trigger a rolling update without needing to modify any functional configuration, change the `LaunchTemplateVersion` parameter.
This has a default value of `1.0.0`, but can be any string. Any change to this string will trigger an `AutoScalingRollingUpdate` in the `DocExampleRecommendationServiceAutoScalingGroup`.

##### Destroy resources

Use AWS CloudFormation to clean up all resources created for this example.
Expand Down
12 changes: 12 additions & 0 deletions aws-cfn/cross-service/resilient-workflow/resilient-service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ Parameters:
SSMTableName:
Type: String
Default: ""
LaunchTemplateVersion:
Type: String
Default: "1.0.0"

Conditions:
EmptySSMTableName: !Equals ["", !Ref SSMTableName]
Expand Down Expand Up @@ -197,6 +200,11 @@ Resources:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: doc-example-resilience-template
TagSpecifications:
- ResourceType: launch-template
Tags:
- Key: InternalVersion
Value: !Ref LaunchTemplateVersion # Increment this value & update the stack to trigger a rolling update of the group
LaunchTemplateData:
InstanceType: !Ref InstanceType
ImageId: resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
Expand All @@ -219,6 +227,10 @@ Resources:
# 4. An Auto Scaling group that starts EC2 instances, one in each of three Availability Zones.
DocExampleRecommendationServiceAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
UpdatePolicy:
AutoScalingRollingUpdate:
MaxBatchSize: 2
MinInstancesInService: 1
Properties:
AutoScalingGroupName: doc-example-resilience-group
AvailabilityZones: { "Fn::GetAZs": { "Ref": "AWS::Region" } }
Expand Down
Loading