-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #46 from buildkite/improve-readme
Improve the README documentation
- Loading branch information
Showing
1 changed file
with
49 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,45 +1,73 @@ | ||
Lifecycled - Gracefully handle EC2 scaling events | ||
======================================================= | ||
# Lifecycled - Gracefully handle EC2 scaling events | ||
|
||
Lifecycled is designed to run on an AWS EC2 instance and listen for various state change mechanisms: | ||
|
||
* [AWS AutoScaling](https://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/lifecycle-hooks.html) | ||
* [Spot Instance Termination Notices](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) | ||
|
||
A handler script is provided to gracefully handle these actions before shutdown. | ||
When a termination notice is received, lifecycled runs a user-provided script (called a handler) and then proceeds with the shutdown. This script can be used to gracefully terminate any daemons you have running. | ||
|
||
Installing | ||
---------- | ||
## Installing with Systemd | ||
|
||
Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases). | ||
Either install with `go get -u github.com/buildkite/lifecycled` or download a [binary release for Linux or Windows](https://github.com/buildkite/lifecycled/releases). Install into `/usr/bin/lifecycled`. | ||
|
||
Running | ||
------- | ||
```bash | ||
# Install the binary | ||
curl -Lf -o /usr/bin/lifecycled \ | ||
https://github.com/buildkite/lifecycled/releases/download/${VERSION}/lifecycled-linux-amd64 | ||
chmod +x /usr/bin/lifecycled | ||
|
||
Check out the [Cloudformation Template](cloudformation/template.yml) for an example of how to setup an autoscaling group with the right permissions and an SNS topic. Once you have the SNS topic, run lifecycled on your aws instance: | ||
|
||
``` | ||
lifecycled --sns-topic arn:aws:sns:us-east-1:11111111:lifecycled-test-1501806648-LifecycleTopic-UTAZ7PQOA32Q | ||
# Install the systemd service | ||
touch /etc/lifecycled | ||
curl -Lf -o /etc/systemd/system/lifecycled.service \ | ||
https://raw.githubusercontent.com/buildkite/lifecycled/${VERSION}/init/systemd/lifecycled.unit | ||
``` | ||
|
||
Autoscaling Hooks | ||
----------------- | ||
Assuming your custom handler script is in `/usr/local/bin/my_graceful_shutdown.sh` and you've got an SNS topic for your EC2 Lifecycle Hooks, you would configure `/etc/lifecycled` with: | ||
|
||
```bash | ||
LIFECYCLED_HANDLER=/usr/local/bin/my_graceful_shutdown.sh | ||
LIFECYCLED_SNS_TOPIC=arn:aws:sns:us-east-1:11111111:my-lifecycle-topic | ||
``` | ||
|
||
Lifecycle events are consumed from an SQS queue and a corresponding hook is executed. Whilst the hook is executing `lifecycled` sends heartbeats to the Autoscaling group to stall further action. When the hook completes executing, if it completes successfully the lifecycle action is completed successfully, if the hook returns a non-zero exit code then the lifecycle action is abandoned. | ||
Then start the daemon with: | ||
|
||
The handler script gets passed the event and the instance id, e.g: `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12` | ||
```bash | ||
systemctl daemon-reload | ||
systemctl enable lifecycled | ||
systemctl start lifecycled | ||
systemctl status lifecycled | ||
``` | ||
|
||
Spot Termination | ||
---------------- | ||
## Handler script | ||
|
||
Handler scripts are used for things like shutting down services that need some time to shutdown. Any example script that shuts down a service and waits for it to shutdown might look like: | ||
|
||
```bash | ||
#!/bin/bash | ||
set -euo pipefail | ||
function await_shutdown() { | ||
echo -n "Waiting for $1..." | ||
while ! systemctl is-active $1 > /dev/null; do | ||
sleep 1 | ||
done | ||
echo "Done!" | ||
} | ||
systemctl stop myservice.service | ||
await_unit myservice.service | ||
``` | ||
|
||
These notices are consumed by polling the local metadata url every 5 seconds. They are passed to the handler script as a custom event, the instance id and the timestamp, e.g `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z` | ||
The handler script is passed the event that was received and the instance id, e.g `autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12` for lifecycle events, or `ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z` in the case of a spot termination. | ||
|
||
Cleaning up Leftover SQS Queues | ||
------------------------------- | ||
## Cleaning up Leftover SQS Queues | ||
|
||
The `lifecycled` daemon should clean up the per-instance SQS queues that are created when it shuts down, but there has been a bug where this does not happen (See https://github.com/buildkite/lifecycled/issues/12). To mitigate this, you can run a cleanup tool: | ||
|
||
``` | ||
go get -u github.com/buildkite/lifecycled/tools/lifecycled-queue-cleaner | ||
lifecycled-queue-cleaner | ||
``` | ||
|
||
## Licence | ||
|
||
See [Licence.md](Licence.md) (MIT) |