diff --git a/README.md b/README.md index cdd464e..3bec757 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,65 @@ -Lifecycled - Gracefully handle EC2 scaling events -======================================================= +# Lifecycled - Gracefully handle EC2 scaling events Lifecycled is designed to run on an AWS EC2 instance and listen for various state change mechanisms: * [AWS AutoScaling](https://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/lifecycle-hooks.html) * [Spot Instance Termination Notices](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) -A handler script is provided to gracefully handle these actions before shutdown. +When a termination notice is received, lifecycled runs a user-provided script (called a handler) and then proceeds with the shutdown. This script can be used to gracefully terminate any daemons you have running. -Installing ----------- +## Installing with Systemd -Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases). +Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases). Install into `/usr/bin/lifecycled`. -Running -------- +```bash +# Install the binary +curl -Lf -o /usr/bin/lifecycled \ + https://github.com/buildkite/lifecycled/releases/download/${VERSION}/lifecycled-linux-amd64 +chmod +x /usr/bin/lifecycled -Check out the [Cloudformation Template](cloudformation/template.yml) for an example of how to setup an autoscaling group with the right permissions and an SNS topic. Once you have the SNS topic, run lifecycled on your aws instance: - -``` -lifecycled --sns-topic arn:aws:sns:us-east-1:11111111:lifecycled-test-1501806648-LifecycleTopic-UTAZ7PQOA32Q +# Install the systemd service +touch /etc/lifecycled +curl -Lf -o /etc/systemd/system/lifecycled.service \ + https://raw.githubusercontent.com/buildkite/lifecycled/${VERSION}/init/systemd/lifecycled.unit ``` -Autoscaling Hooks ------------------ +Assuming your custom handler script is in `/usr/local/bin/my_graceful_shutdown.sh` and you've got an SNS topic for your EC2 Lifecycle Hooks, you would configure `/etc/lifecycled` with: + +```bash +LIFECYCLED_HANDLER=/usr/local/bin/my_graceful_shutdown.sh +LIFECYCLED_SNS_TOPIC=arn:aws:sns:us-east-1:11111111:my-lifecycle-topic +``` -Lifecycle events are consumed from an SQS queue and a corresponding hook is executed. Whilst the hook is executing `lifecycled` sends heartbeats to the Autoscaling group to stall further action. When the hook completes executing, if it completes successfully the lifecycle action is completed successfully, if the hook returns a non-zero exit code then the lifecycle action is abandoned. +Then start the daemon with: -The handler script gets passed the event and the instance id, e.g: `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12` +```bash +systemctl daemon-reload +systemctl enable lifecycled +systemctl start lifecycled +systemctl status lifecycled +``` -Spot Termination ----------------- +## Handler script + +Handler scripts are used for things like shutting down services that need some time to shutdown. Any example script that shuts down a service and waits for it to shutdown might look like: + +```bash +#!/bin/bash +set -euo pipefail +function await_shutdown() { + echo -n "Waiting for $1..." + while ! systemctl is-active $1 > /dev/null; do + sleep 1 + done + echo "Done!" +} +systemctl stop myservice.service +await_unit myservice.service +``` -These notices are consumed by polling the local metadata url every 5 seconds. They are passed to the handler script as a custom event, the instance id and the timestamp, e.g `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z` +The handler script is passed the event that was received and the instance id, e.g `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12` for lifecycle events, or `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z` in the case of a spot termination. -Cleaning up Leftover SQS Queues -------------------------------- +## Cleaning up Leftover SQS Queues The `lifecycled` daemon should clean up the per-instance SQS queues that are created when it shuts down, but there has been a bug where this does not happen (See https://github.com/buildkite/lifecycled/issues/12). To mitigate this, you can run a cleanup tool: @@ -43,3 +67,7 @@ The `lifecycled` daemon should clean up the per-instance SQS queues that are cre go get -u github.com/buildkite/lifecycled/tools/lifecycled-queue-cleaner lifecycled-queue-cleaner ``` + +## Licence + +See [Licence.md](Licence.md) (MIT)