diff --git a/.buildkite/steps/cleanup.sh b/.buildkite/steps/cleanup.sh index 149bcf705..0feff4b16 100755 --- a/.buildkite/steps/cleanup.sh +++ b/.buildkite/steps/cleanup.sh @@ -1,13 +1,15 @@ #!/bin/bash # shellcheck disable=SC2016 -set -uo pipefail +set -uxo pipefail if [[ $OSTYPE =~ ^darwin ]] ; then - cutoff_date=$(gdate --date='-2 days' +%Y-%m-%d) + cutoff_date=$(gdate --date='-1 days' +%Y-%m-%d) else - cutoff_date=$(date --date='-2 days' +%Y-%m-%d) + cutoff_date=$(date --date='-1 days' +%Y-%m-%d) fi +echo "--- Cleaning up resources older than ${cutoff_date}" + if [[ -n "${AWS_STACK_NAME:-}" ]] ; then echo "--- Deleting stack $AWS_STACK_NAME" aws cloudformation delete-stack --stack-name "$AWS_STACK_NAME" @@ -18,8 +20,8 @@ aws s3api list-buckets \ --output text \ --query "$(printf 'Buckets[?CreationDate<`%s`].Name' "$cutoff_date" )" \ | xargs -n1 \ - | grep -E 'buildkite-aws-stack-test-managedsecrets' \ - | xargs -n1 -t -I% aws s3 rb s3://% --force + | grep -E 'buildkite-aws-stack-test-(\d+-)?managedsecrets' \ + | xargs -n1 -t -I% aws s3 rb s3://% --force echo "--- Deleting old cloudformation stacks" aws cloudformation describe-stacks \ diff --git a/README.md b/README.md index 4ff9f1546..2a0aa2fa2 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ Here's an example that shows how to generate a private SSH key, and upload it wi ssh-keygen -t rsa -b 4096 -f id_rsa_buildkite pbcopy < id_rsa_buildkite.pub # paste this into your github deploy key -aws s3 cp --acl private --sse aws:kms id_rsa_buildkite "s3://${SecretsBucket}/private_ssh_key" +aws s3 cp --acl private --sse aws:kms id_rsa_buildkite "s3://${SecretsBucket}/private_ssh_key" ``` If you want to set secrets that your build can access, create a file that sets environment variables and upload it: @@ -96,10 +96,10 @@ If you want to set secrets that your build can access, create a file that sets e ```bash echo "export MY_ENV_VAR=something secret" > myenv aws s3 cp --acl private --sse aws:kms myenv "s3://${SecretsBucket}/env" -rm myenv +rm myenv ``` -**Note: Currently only using the default KMS key for s3 can be used, follow [#235](https://github.com/buildkite/elastic-ci-stack-for-aws/issues/235) for progress on using specific KMS keys** +**Note: Currently only using the default KMS key for s3 can be used, follow [#235](https://github.com/buildkite/elastic-ci-stack-for-aws/issues/235) for progress on using specific KMS keys** If you really want to store your secrets unencrypted, you can disable it entirely with `BUILDKITE_USE_KMS=false`. @@ -107,7 +107,7 @@ If you really want to store your secrets unencrypted, you can disable it entirel * [Amazon Linux 2017.03.1](https://aws.amazon.com/amazon-linux-ami/) * [Buildkite Agent](https://buildkite.com/docs/agent) -* [Docker 17.05.0-ce](https://www.docker.com) +* [Docker 17.06.0-ce](https://www.docker.com) * [Docker Compose 1.14.0](https://docs.docker.com/compose/) * [aws-cli](https://aws.amazon.com/cli/) - useful for performing any ops-related tasks * [jq](https://stedolan.github.io/jq/) - useful for manipulating JSON responses from cli tools such as aws-cli or the Buildkite API @@ -124,7 +124,7 @@ By following these simple conventions you get a scaleable, repeatable and source ## Multiple Instances of the Stack -If you need to different instances sizes and scaling characteristics between pipelines, you can create multiple stack. Each can run on a different [Agent Queue](https://buildkite.com/docs/agent/queues), with it's own configuration, or even in a different AWS account. +If you need to different instances sizes and scaling characteristics between pipelines, you can create multiple stack. Each can run on a different [Agent Queue](https://buildkite.com/docs/agent/queues), with it's own configuration, or even in a different AWS account. Examples: @@ -134,9 +134,9 @@ Examples: ## Autoscaling -If you have provided `BuildkiteApiAccessToken` and your `MinSize` < `MaxSize`, the stack will automatically scale up and down based on the number of scheduled jobs. +If you have provided `BuildkiteApiAccessToken` and your `MinSize` < `MaxSize`, the stack will automatically scale up and down based on the number of scheduled jobs. -This means you can scale down to zero when idle, which means you can use larger instances for the same cost. +This means you can scale down to zero when idle, which means you can use larger instances for the same cost. Metrics are collected with a Lambda function, polling every minute. @@ -160,19 +160,19 @@ If you want to login to an ECR server on another AWS account, you can set `AWS_E We recommend running the latest release, which is available at `https://s3.amazonaws.com/buildkite-aws-stack/aws-stack.json`, or on the [releases page](https://github.com/buildkite/elastic-ci-stack-for-aws/releases). -The latest build of the stack is published to `https://s3.amazonaws.com/buildkite-aws-stack/master/aws-stack.json`, along with a version for each commit in the form of `https://s3.amazonaws.com/buildkite-aws-stack/master/${COMMIT}.aws-stack.json`. +The latest build of the stack is published to `https://s3.amazonaws.com/buildkite-aws-stack/master/aws-stack.json`, along with a version for each commit in the form of `https://s3.amazonaws.com/buildkite-aws-stack/master/${COMMIT}.aws-stack.json`. -Branches are published in the form of `https://s3.amazonaws.com/buildkite-aws-stack/${BRANCH}/aws-stack.json`. +Branches are published in the form of `https://s3.amazonaws.com/buildkite-aws-stack/${BRANCH}/aws-stack.json`. ## Updating Your Stack To update your stack to the latest version use CloudFormation’s stack update tools with one of the urls in the [Versions](#versions) section. -Prior to updating, it's a good idea to set the desired instance size on the AutoscalingGroup to 0 manually. +Prior to updating, it's a good idea to set the desired instance size on the AutoscalingGroup to 0 manually. ## CloudWatch Metrics -Metrics are calculated every minute from the Buildkite API using a lambda function. +Metrics are calculated every minute from the Buildkite API using a lambda function. cloudwatch diff --git a/packer/scripts/install-docker.sh b/packer/scripts/install-docker.sh index b67544f28..f7d5e392e 100755 --- a/packer/scripts/install-docker.sh +++ b/packer/scripts/install-docker.sh @@ -1,7 +1,7 @@ #!/bin/bash set -eu -o pipefail -DOCKER_VERSION=17.05.0-ce +DOCKER_VERSION=17.06.0-ce DOCKER_COMPOSE_VERSION=1.14.0 # This performs a manual install of Docker. The init.d script is from the @@ -17,7 +17,7 @@ sudo groupadd docker sudo usermod -a -G docker ec2-user # Manual install ala https://docs.docker.com/engine/installation/binaries/ -curl -Lsf https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz > docker.tgz +curl -Lsf https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VERSION}.tgz > docker.tgz tar -xvzf docker.tgz sudo mv docker/* /usr/bin rm docker.tgz diff --git a/plugins/ecr b/plugins/ecr index 02c9f8c80..c2438d180 160000 --- a/plugins/ecr +++ b/plugins/ecr @@ -1 +1 @@ -Subproject commit 02c9f8c8032103b5f9ef6c6a90bdb75d9da03b08 +Subproject commit c2438d180be1fb55f6595e223ea2b913481d2c34