Skip to content

Commit

Permalink
CFN: Add resilient service rolling update mechanism (awsdocs#5955)
Browse files Browse the repository at this point in the history
Add CFN resilient service rolling update mechanism
  • Loading branch information
DavidSouther authored and max-webster committed Mar 15, 2024
1 parent 18a2802 commit f35bb37
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 5 deletions.
16 changes: 11 additions & 5 deletions aws-cfn/cross-service/resilient-workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,11 @@ aws elbv2 describe-target-health --target-group-arn "arn:aws:elasticloadbalancin

All stack outputs:

| OutputKey | OutputValue | Usage |
| --------- | ----------------------------------------- | ------------------------------------------------- |
| `LB` | The DNS Name of the primary load balancer | `curl` or Browser |
| OutputKey | OutputValue | Usage |
| --------- | ------------------------------------------ | ------------------------------------------------- |
| `LB` | The DNS Name of the primary load balancer | `curl` or Browser |
| `Key` | The ID of a .pem format private key in SSM | `ssh` after downloading from SSM |
| `TGArn` | The ARN of the target group of instances | Check various additional information from the CLI |
| `TGArn` | The ARN of the target group of instances | Check various additional information from the CLI |

#### Demonstrate resiliency

Expand All @@ -182,6 +182,7 @@ aws cloudformation update-stack \

1. **Initial state: healthy** — Sends requests to the endpoint to get recommendations and verify that instances
are healthy.

2. **Broken dependency** — Sets a parameter that specifies a nonexistent DynamoDB table name. This simulates a
failure of the recommendation service. Requests for recommendations now return a failure
code. All instances still report as healthy because they only implement shallow health checks. For this
Expand Down Expand Up @@ -231,7 +232,8 @@ aws cloudformation update-stack \

Using the AWS Management Console, open the CloudFormation page. Navigate to the `resilience-demo` stack. Choose the `Resources`
tab. Find the `DocExampleRecommendationServiceTargetGroup` line. Choose the `Physical Resource ID` link.
From this EC2 page, find the list of instances in the target group. Select one and navigate to it. Choose `Actions`, `Terminate instance`. See EC2 terminate the instance, and watch the Auto Scaling group start a new instance.
From this EC2 page, find the list of instances in the target group. Select one and navigate to it. Choose `Actions`, `Terminate instance`.
See EC2 terminate the instance, and watch the Auto Scaling group start a new instance.

7. **Fail open** — Sets the table name parameter so the recommendations service fails for all instances.
Because all instances are using deep health checks, they all report as unhealthy. In this
Expand All @@ -242,6 +244,10 @@ aws cloudformation update-stack \
Edit `params.json`. Add a new entry with `ParameterKey` as `SSMTableName` and `ParameterValue` as `unknown`.
After updating, the service should report unhealthy but return static responses.

8. **Rolling Update** If necessary, you can trigger rolling updates to all instances by changing the Launch Template.
To change a non-functional aspect of the Launch Template, which will trigger a rolling update without needing to modify any functional configuration, change the `LaunchTemplateVersion` parameter.
This has a default value of `1.0.0`, but can be any string. Any change to this string will trigger an `AutoScalingRollingUpdate` in the `DocExampleRecommendationServiceAutoScalingGroup`.

##### Destroy resources

Use AWS CloudFormation to clean up all resources created for this example.
Expand Down
12 changes: 12 additions & 0 deletions aws-cfn/cross-service/resilient-workflow/resilient-service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ Parameters:
SSMTableName:
Type: String
Default: ""
LaunchTemplateVersion:
Type: String
Default: "1.0.0"

Conditions:
EmptySSMTableName: !Equals ["", !Ref SSMTableName]
Expand Down Expand Up @@ -197,6 +200,11 @@ Resources:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: doc-example-resilience-template
TagSpecifications:
- ResourceType: launch-template
Tags:
- Key: InternalVersion
Value: !Ref LaunchTemplateVersion # Increment this value & update the stack to trigger a rolling update of the group
LaunchTemplateData:
InstanceType: !Ref InstanceType
ImageId: resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
Expand All @@ -219,6 +227,10 @@ Resources:
# 4. An Auto Scaling group that starts EC2 instances, one in each of three Availability Zones.
DocExampleRecommendationServiceAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
UpdatePolicy:
AutoScalingRollingUpdate:
MaxBatchSize: 2
MinInstancesInService: 1
Properties:
AutoScalingGroupName: doc-example-resilience-group
AvailabilityZones: { "Fn::GetAZs": { "Ref": "AWS::Region" } }
Expand Down

0 comments on commit f35bb37

Please sign in to comment.