Harvey

The lightweight Docker Compose deployment runner.

Why Docker Compose Deployments

I've long been a fan of the simplicity of a docker-compose file and its usage. Deploying with systems such as Rancher or a self-hosted GitLab seemed too daunting with uneccessary overhead for simple projects. Why can't I use the same setup in production as I do in development? Skip the environment variable injection and key stores, configuring production machines, and separate workflow configuration by having Harvey spin up my project by using Docker Compose in production, just like I do locally.

How it Works

Harvey receives a webhook from GitHub, pulls in the changes, and deploys them. If you have Slack enabled, Harvey can send you the deployment summary along with running healthchecks to ensure the deployment was successful.

1. GitHub webhook fires and is received by Harvey stating that a new commit hit an enabled repo on an allowed branch to be deployed
2. Harvey pulls in your changes from GitHub and builds your Docker image locally
3. Harvey spins up the new Docker container and tears down the old one once it's up and running (read: not zero downtime)
4. (Optional) Harvey will then run a container healthcheck to ensure your new container is up and running
5. (Optional) Harvey can send a message via Slack to notify users about the status of the deployment

Install

Because Harvey interacts directly with the Docker daemon (using sockets), to build and orchestrate Docker images and containers, Harvey cannot run in a Docker container itself and must be run on your bare-metal OS.

1. Install Docker
2. Ensure you've added your ssh key to the ssh agent: ssh-add followed by your password (alternatively, see USE_HTTPS_AUTH to use HTTPS URLs instead with something like Git Credential Manager)
3. Setup enviornment variables as needed in the .env file
4. Enable GitHub webhooks for all your repositories you want to use Harvey with (point them to http://example.com:5000/deploy, send the payload as JSON)
   • You could alternatively setup a GitHub Action or other CI flow to only trigger a webhook event once tests pass for instance (must include all the details as if it were generated by GitHub, see Workflow Webhook for an example on how to do this.)

# Install Harvey via GitHub
git clone https://github.com/Justintime50/harvey.git
cp .env-example .env # alternatively add your .env file to the $HARVEY_PATH
just install

You can alternatively download a release and follow similar steps to the above instead of cloning the repo.

Usage

# Run locally for development (runs via Flask)
just run

# Run in production (runs via uWSGI)
just prod

Things to Know

• Cloning
  • Harvey will shallow clone your project to the most recent commit
• Naming
  • Harvey expects the container name to match the GitHub repository name exactly, otherwise healthchecks will fail
  • Harvey does not handle renamed or transferred repos for you. If you rename a repo, you may need to intervene manually to shut down the old container, remove it from Harvey, and startup the new one initially on your own
• Deployments
  • Initial deployments are not gracefully handled. Because Harvey requires no configuration for a project, it assumes everything is already setup on the server. This means that on an initial deploy, you will need to set environment variables on your server, migrate databases, and whatever else may be required, at which point you may need to redeploy the project for the changes to take affect
    • Future deploys should then work without additional intervention unless you have specific manual steps like updating env vars or future database migrations
  • Because we use threads, you cannot kill an ongoing deployment because you cannot reliably kill a thread
• Logs
  • Harvey automatically rotates log files and keeps them for 2 weeks before purging them

Harvey Configuration

Configuration Criteria

• Each repo either needs a committed .harvey.yaml file in the root directory which will be used whenever a GitHub webhook fires, or a data key passed into the webhook delivered to Harvey (via something like GitHub Actions). This can be accomplished by using something like workflow-webhook or another homegrown solution (requires the entire webhook payload from GitHub. Harvey will always fallback to the .harvey.yaml file if there is no data key present)
• You can specify one of deploy or pull as the deployment_type (deploy is the default)
• Optional: prod_compose: true json can be passed to instruct Harvey to use a prod docker-compose file in addition to the base compose file. This will run the equivelant of the following when deploying: docker-compose -f docker-compose.yml -f docker-compose-prod.yml and is useful to allow both local and production compose setups in a single project.

.harvey.yaml Example

deployment_type: deploy
prod_compose: true
healthcheck:
  - container_name_1
  - container_name_2

GitHub Action Example

deploy:
    needs: ["test", "lint"]
    runs-on: ubuntu-latest
    steps:
        - name: Deploy to Harvey
        if: github.ref == 'refs/heads/main'
        uses: distributhor/workflow-webhook@v2
        env:
            webhook_type: "json-extended"
            webhook_url: ${{ secrets.WEBHOOK_URL }}
            webhook_secret: ${{ secrets.WEBHOOK_SECRET }}
            data: '{ "deployment_type": "deploy", "prod_compose" : true, "healthcheck": ["container_name_1", "container_name_2"] }'

Harvey's entrypoint (eg: http://127.0.0.1:5000/deploy) accepts a webhook from GitHub. If you'd like to simulate a GitHub webhook, pass JSON like the following example to the Harvey webhook endpoint:

{
  "ref": "refs/heads/main",
  "repository": {
    "name": "justinpaulhammond",
    "full_name": "Justintime50/justinpaulhammond",
    "html_url": "https://github.com/Justintime50/justinpaulhammond",
    "ssh_url": "[email protected]:Justintime50/justinpaulhammond.git",
    "owner": {
      "name": "Justintime50"
    }
  },
  "commits": [
    {
      "id": "1",
      "author": {
        "name": "Justintime50"
      }
    }
  ],
  "data": {
    "deployment_type": "deploy"
  }
}

App Configuration

Environment Variables:
    ALLOWED_BRANCHES  A comma separated list of branch names that are allowed to trigger deployments from a webhook event. Default: "main,master"
    DEPLOY_ON_TAG     A boolean specifying if a tag pushed will trigger a deploy. Default: True
    HARVEY_PATH       The path where Harvey will store projects, logs, and the SQLite databases. Default: ~/harvey
    HOST              The host Harvey will run on. Default: 127.0.0.1
    LOG_LEVEL         The logging level used for the entire application. Default: INFO
    OPERATION_TIMEOUT The number of seconds any given operation (git command, deploy pipeline) can take before timing out. Default: 300
    PAGINATION_LIMIT  The number of records to return via API. Default: 20
    PORT              The port Harvey will run on. Default: 5000
    SENTRY_URL        The URL authorized to receive sentry alerts.
    SLACK_BOT_TOKEN   The Slackbot token to use to authenticate each request to Slack.
    SLACK_CHANNEL     The Slack channel to send messages to.
    USE_HTTPS_AUTH    Use HTTPS URLs instead of SSH URLs to authenticate with Git. Default: False
    USE_SLACK         Set to "true" to send slack messages.
    WEBHOOK_SECRET    The Webhook secret required by GitHub (if enabled, leave blank to ignore) to secure your webhooks. Default: None

API

Endpoints

• /deployments (GET) - Retrieve a list of deployments
• /deployments/{deployment_id} (GET) - Retrieve the details of a single deployment
• /deploy (POST) - Deploy a project with data from a GitHub webhook
• /projects (GET) - Retrieve a list of projects
• /projects/{project_name}/lock (PUT) - Locks the deployments of a project
• /projects/{project_name}/unlock (PUT) - Unlocks the deployments of a project
• /projects/{project_name}/webhook (GET) - Retrieves the current webhook of a project
• /locks (GET) - Retrieve a list of locks
• /locks/{project_name} (GET) - Retrieve the lock status of a project
• /threads (GET) - Retrieves a list of threads (named after projects) running via Harvey. A thread indicates an ongoing deployment

Authentication

The Harvey API can be secured via Basic Auth by setting the WEBHOOK_SECRET env var (this secret is used for authenticating a webhook came from GitHub in addition to securing the remaining endpoints).

At this time, there is no multi-user authentication options. A single secret (password) secures the entire service. Additionally, there are no permissions. Harvey should be deployed as a runner where admins are the only ones expected to interact with it. If you want multi-user authentication, you can use Harvey UI that allows multiple users to have their own credentials (no permissions). Each of those users then authenticates with Harvey under the hood with the same WEBHOOK_SECRET.

Examples

# Retrieve a list of deployments
curl -X GET http://127.0.0.1:5000/deployments

# Retrieve a deployment from Harvey using the full repo name where slashes are replaced with dashes
curl -X GET http://127.0.0.1:5000/deployments/justintime50-justinpaulhammond

Development

# Get a comprehensive list of development tools
just --list