Skip to content

Commit

Permalink
Merge pull request #46 from buildkite/improve-readme
Browse files Browse the repository at this point in the history
Improve the README documentation
  • Loading branch information
lox authored Sep 7, 2018
2 parents b1f84bf + ca2f034 commit 4e7951c
Showing 1 changed file with 49 additions and 21 deletions.
70 changes: 49 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,73 @@
Lifecycled - Gracefully handle EC2 scaling events
=======================================================
# Lifecycled - Gracefully handle EC2 scaling events

Lifecycled is designed to run on an AWS EC2 instance and listen for various state change mechanisms:

* [AWS AutoScaling](https://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/lifecycle-hooks.html)
* [Spot Instance Termination Notices](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html)

A handler script is provided to gracefully handle these actions before shutdown.
When a termination notice is received, lifecycled runs a user-provided script (called a handler) and then proceeds with the shutdown. This script can be used to gracefully terminate any daemons you have running.

Installing
----------
## Installing with Systemd

Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases).
Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases). Install into `/usr/bin/lifecycled`.

Running
-------
```bash
# Install the binary
curl -Lf -o /usr/bin/lifecycled \
https://github.com/buildkite/lifecycled/releases/download/${VERSION}/lifecycled-linux-amd64
chmod +x /usr/bin/lifecycled

Check out the [Cloudformation Template](cloudformation/template.yml) for an example of how to setup an autoscaling group with the right permissions and an SNS topic. Once you have the SNS topic, run lifecycled on your aws instance:

```
lifecycled --sns-topic arn:aws:sns:us-east-1:11111111:lifecycled-test-1501806648-LifecycleTopic-UTAZ7PQOA32Q
# Install the systemd service
touch /etc/lifecycled
curl -Lf -o /etc/systemd/system/lifecycled.service \
https://raw.githubusercontent.com/buildkite/lifecycled/${VERSION}/init/systemd/lifecycled.unit
```

Autoscaling Hooks
-----------------
Assuming your custom handler script is in `/usr/local/bin/my_graceful_shutdown.sh` and you've got an SNS topic for your EC2 Lifecycle Hooks, you would configure `/etc/lifecycled` with:

```bash
LIFECYCLED_HANDLER=/usr/local/bin/my_graceful_shutdown.sh
LIFECYCLED_SNS_TOPIC=arn:aws:sns:us-east-1:11111111:my-lifecycle-topic
```

Lifecycle events are consumed from an SQS queue and a corresponding hook is executed. Whilst the hook is executing `lifecycled` sends heartbeats to the Autoscaling group to stall further action. When the hook completes executing, if it completes successfully the lifecycle action is completed successfully, if the hook returns a non-zero exit code then the lifecycle action is abandoned.
Then start the daemon with:

The handler script gets passed the event and the instance id, e.g: `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12`
```bash
systemctl daemon-reload
systemctl enable lifecycled
systemctl start lifecycled
systemctl status lifecycled
```

Spot Termination
----------------
## Handler script

Handler scripts are used for things like shutting down services that need some time to shutdown. Any example script that shuts down a service and waits for it to shutdown might look like:

```bash
#!/bin/bash
set -euo pipefail
function await_shutdown() {
echo -n "Waiting for $1..."
while ! systemctl is-active $1 > /dev/null; do
sleep 1
done
echo "Done!"
}
systemctl stop myservice.service
await_unit myservice.service
```

These notices are consumed by polling the local metadata url every 5 seconds. They are passed to the handler script as a custom event, the instance id and the timestamp, e.g `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z`
The handler script is passed the event that was received and the instance id, e.g `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12` for lifecycle events, or `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z` in the case of a spot termination.

Cleaning up Leftover SQS Queues
-------------------------------
## Cleaning up Leftover SQS Queues

The `lifecycled` daemon should clean up the per-instance SQS queues that are created when it shuts down, but there has been a bug where this does not happen (See https://github.com/buildkite/lifecycled/issues/12). To mitigate this, you can run a cleanup tool:

```
go get -u github.com/buildkite/lifecycled/tools/lifecycled-queue-cleaner
lifecycled-queue-cleaner
```

## Licence

See [Licence.md](Licence.md) (MIT)

0 comments on commit 4e7951c

Please sign in to comment.