docker-airflow

This repository contains Dockerfile of apache-airflow for Docker's automated build published to the public Docker Hub Registry.

Pre-Requisites / Installation

• Based on Python (3.7-slim-stretch) official Image python:3.7-slim-stretch and uses the official Postgres as backend and Redis as queue
• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Build

Run < make build > to tag the current build as hubble/docker-airflow:latest.

Usage

For CeleryExecutor :

make compose

To kill the deployed containers:

make kill

For encrypted connection passwords (in Local or Celery Executor), you must have the same fernet_key. By default docker-airflow generates the fernet_key at startup, you have to set an environment variable in the docker-compose (ie: docker-compose-LocalExecutor.yml) file to set the same key accross containers. To generate a fernet_key :

docker run puckel/docker-airflow python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)"

Configurating Airflow

It's possible to set any configuration value for Airflow from environment variables, which are used over values from the airflow.cfg.

The general rule is the environment variable should be named AIRFLOW__<section>__<key>, for example AIRFLOW__CORE__SQL_ALCHEMY_CONN sets the sql_alchemy_conn config option in the [core] section.

Check out the Airflow documentation for more details

You can also define connections via environment variables by prefixing them with AIRFLOW_CONN_ - for example AIRFLOW_CONN_POSTGRES_MASTER=postgres://user:password@localhost:5432/master for a connection called "postgres_master". The value is parsed as a URI. This will work for hooks etc, but won't show up in the "Ad-hoc Query" section unless an (empty) connection is also created in the DB

Global variables or secrets can be mounted onto the environment as Environmental Variables in Kubernetes to be accessed such as AWS Secret Keys etc under extraEnv in value.yaml file to be deployed in Kubernetes. For Docker-compose files, simply mount the variables as a volume.

Running other airflow commands

If you want to enter the containers in Bash Mode:

make webserver / worker / scheduler

Tag Docker in AWS ECR Docker Repository

The purpose of this section is to tag local docker builds into AWS ECR Docker Repository so as to be able to deploy on AWS Kubernetes (EKS).
We are also assuming that you have the needed AWS credentials to access the repositories.

To check the current tag of repository:

make tag

To push the next tag number for the local build:

make pushnext

For example, if the current tag number is 1.0.1, the next tag will be 1.0.2. The current repository will be built and be pushed to hubble/docker-airflow:1.0.2.
Change the tag number in values.yaml to be updated to the next tag on Kubernetes.

Local UI Links

• Airflow: localhost:8080
• Flower: localhost:5555

Scale the number of workers

Easy scaling using docker-compose:

docker-compose -f docker-compose-CeleryExecutor.yml scale worker=5