Update Docker provisioning and docker compose command (#953)

* Update Docker provisioning and docker compose command Updates the Docker installation to: - Not use the azavea.docker role (which is no longer maintained), inlining the relevant steps instead. - Use a newer version - Invoke Docker Compose with 'docker compose' (rather than 'docker-compose') since that's how it works in newer Docker versions. I also updated the Vagrant base box, mainly because we might as well not be out of date if we don't have to be, and renamed the shared 'django' docker-compose config to 'django-common', since apparently docker compose now treats it as a name collision to have services in different files with the same name. (Also added a changelog entry for PR #951) Resolves #952
azavea · Feb 28, 2024 · 0b75fde · 0b75fde
1 parent d5fe3b8
commit 0b75fde
Show file tree

Hide file tree

Showing 20 changed files with 79 additions and 64 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Upcoming release]
 
+- Update OSM tags for doctors, dentists, hospitals, and pharmacies
+- Update Vagrant and Docker provisioning
+
 ## [0.19.0] - 2023-08-30
 
 - Update OSM tags for crossings and traffic islands (PR #948)

diff --git a/README.LOCAL-ANALYSIS.md b/README.LOCAL-ANALYSIS.md
@@ -7,7 +7,7 @@ run `vagrant ssh` from the project directory on your host machine.
 
 The container is built by `scripts/update`, but can also be rebuilt separately by running:
 ```
-docker-compose build analysis
+docker compose build analysis
 ```
 
 ## Example

diff --git a/README.md b/README.md
@@ -135,23 +135,23 @@ Build the docker container for the verification tool within the VM:
 
 ```
 cd src/verifier
-docker-compose build
+docker compose build
 ```
 
 Ensure the exported output from the analysis to check exists in the `data/output` directory. It will be there by default if the `data` directory was used for the neighborhood input shapefile.
 
 To compare the analysis output for Boulder, run the verification tool with:
 
 ```
-docker-compose run verifier boulder.csv
+docker compose run verifier boulder.csv
 ```
 
 Any output in the `verified_output` directory may be used for comparison.
 
 To compare to analysis output that has a non-default filename (`analysis_neighborhood_score_inputs.csv`), run the verification tool with the name of the file in `data/output` as the second argument:
 
 ```
-docker-compose run verifier boulder.csv my_output_to_verify.csv
+docker compose run verifier boulder.csv my_output_to_verify.csv
 ```
 
 If there are any differences in the outputs, a summary of the differences will be output to console.

diff --git a/Vagrantfile b/Vagrantfile
@@ -25,7 +25,7 @@ ROOT_VM_DIR = "/vagrant"
 
 Vagrant.configure("2") do |config|
 
-  config.vm.box = "bento/ubuntu-20.04"
+  config.vm.box = "bento/ubuntu-22.04"
   config.vm.hostname = "pfb-network-connectivity"
   config.vm.network :private_network, ip: ENV.fetch("PFB_PRIVATE_IP", "192.168.56.12")
 
@@ -59,7 +59,6 @@ Vagrant.configure("2") do |config|
 
   config.vm.provision "ansible_local" do |ansible|
     ansible.playbook = "deployment/ansible/pfb.yml"
-    ansible.galaxy_role_file = "deployment/ansible/roles.yml"
     ansible.verbose = true
     ansible.raw_arguments = ["--timeout=60",
                              "--extra-vars",
@@ -70,7 +69,7 @@ Vagrant.configure("2") do |config|
     ansible.install_mode = "pip"
     # Install pip3 for the system version of python3, and make sure 'pip3' works as well
     ansible.pip_install_cmd = "sudo apt-get install -y python3-pip && sudo ln -s -f /usr/bin/pip3 /usr/bin/pip"
-    ansible.version = "2.8.7"
+    ansible.version = "2.10.7"
   end
 
   config.vm.provider :virtualbox do |v|

diff --git a/common.yml b/common.yml
@@ -19,7 +19,7 @@ services:
     volumes:
       - $HOME/.aws:/root/.aws:ro
 
-  django:
+  django-common:
     extends:
       service: base
     build:

diff --git a/deployment/README.md b/deployment/README.md
@@ -166,7 +166,7 @@ and they will be passed in during deployment.
 Use the Tilegarden Node scripts in the VM to deploy a new Tilegarden instance:
 
 ```
-vagrant@pfb-network-connectivity:/vagrant$ docker-compose \
+vagrant@pfb-network-connectivity:/vagrant$ docker compose \
                                              -f docker-compose.yml \
                                              -f docker-compose.test.yml \
                                              run --rm --entrypoint yarn \

diff --git a/deployment/ansible/group_vars/all.example b/deployment/ansible/group_vars/all.example
@@ -7,8 +7,3 @@ aws_profile: pfb
 # AWS Batch development settings
 pfb_aws_batch_analysis_job_queue_name: "name_of_batch_analysis_job_queue"
 pfb_aws_batch_analysis_job_definition_name: "staging-pfb-analysis-run-job"
-
-# azavea.docker
-docker_version: 5:19.*
-docker_options: "--storage-opt dm.basesize=20G"
-docker_compose_version: 1.23.*
diff --git a/deployment/ansible/roles/pfb.docker/defaults/main.yml b/deployment/ansible/roles/pfb.docker/defaults/main.yml
@@ -0,0 +1,9 @@
+---
+docker_repository_arch: "amd64"
+docker_packages:
+  - "docker-ce={{ docker_version }}"
+  - "docker-ce-cli={{ docker_version }}"
+  - "containerd.io={{ docker_containerd_version }}"
+docker_version: "5:25.*"
+docker_containerd_version: "1.6.*"
+docker_options: "--storage-opt dm.basesize=20G"
diff --git a/deployment/ansible/roles/pfb.docker/meta/main.yml b/deployment/ansible/roles/pfb.docker/meta/main.yml
diff --git a/deployment/ansible/roles/pfb.docker/tasks/main.yml b/deployment/ansible/roles/pfb.docker/tasks/main.yml
@@ -1,10 +1,22 @@
 ---
-- name: Install docker-compose
-  pip: name=docker-compose
-       version={{ docker_compose_version }}
-       extra_args=--ignore-installed
+- name: Download Docker APT key
+  apt_key:
+    url: https://download.docker.com/linux/{{ ansible_distribution | lower }}/gpg
+    id: 0EBFCD88
+    state: present
 
-- name: Add Ansible user to Docker group
-  user: name="{{ ansible_user }}"
+- name: Configure the Docker APT repository
+  apt_repository:
+    repo: |
+      deb [arch={{ docker_repository_arch }}] https://download.docker.com/linux/{{ ansible_distribution | lower }} {{ ansible_distribution_release }} stable
+    state: present
+
+- name: Install Docker
+  apt:
+    pkg: "{{ docker_packages }}"
+    state: present
+
+- name: Add Vagrant user to Docker group
+  user: name=vagrant
         groups=docker
         append=yes
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -22,7 +22,7 @@ services:
   django:
     image: pfb-app
     extends:
-      service: django
+      service: django-common
       file: common.yml
     ports:
       - "9202:9202"
@@ -39,7 +39,7 @@ services:
   django-q:
     image: django-q
     extends:
-      service: django
+      service: django-common
       file: common.yml
     entrypoint: ./manage.py
     command: qcluster
@@ -72,7 +72,7 @@ services:
   analysis:
     image: pfb-analysis
     extends:
-      service: django
+      service: django-common
       file: common.yml
     build:
       context: ./src

diff --git a/scripts/cibuild b/scripts/cibuild
@@ -27,7 +27,7 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
         echo "Building container image"
         # Make sure Tilegarden config files exist
         touch "${DIR}/../src/tilegarden/.env" "${DIR}/../src/tilegarden/claudia.json"
-        GIT_COMMIT="${GIT_COMMIT}" docker-compose \
+        GIT_COMMIT="${GIT_COMMIT}" docker compose \
                   -f "${DIR}/../docker-compose.yml" \
                   -f "${DIR}/../docker-compose.test.yml" \
                   build database django angularjs analysis
@@ -46,14 +46,14 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
 
         echo "Running Django collectstatic..."
         GIT_COMMIT="${GIT_COMMIT}" \
-        docker-compose \
+        docker compose \
             -f docker-compose.yml \
             -f docker-compose.test.yml \
             run --rm --no-deps --entrypoint "./manage.py" \
             django collectstatic --noinput
 
         echo "Building nginx image"
-        GIT_COMMIT="${GIT_COMMIT}" docker-compose \
+        GIT_COMMIT="${GIT_COMMIT}" docker compose \
                   -f "${DIR}/../docker-compose.yml" \
                   -f "${DIR}/../docker-compose.test.yml" \
                   build nginx

diff --git a/scripts/clear-tile-cache b/scripts/clear-tile-cache
@@ -2,7 +2,7 @@
 
 set -e
 
-DEFAULT_BUCKET_NAME=$(docker-compose run --rm tilegarden bash -c 'echo -n $PFB_TILEGARDEN_CACHE_BUCKET')
+DEFAULT_BUCKET_NAME=$(docker compose run --rm tilegarden bash -c 'echo -n $PFB_TILEGARDEN_CACHE_BUCKET')
 
 function usage() {
     echo -n \

diff --git a/scripts/console b/scripts/console
@@ -31,11 +31,11 @@ esac
 pushd ..
 
 if [ -n "$NORMAL_CONTAINER" ]; then
-    docker-compose exec "${1}" /bin/bash
+    docker compose exec "${1}" /bin/bash
 fi
 
 if [ -n "$DATABASE_CONTAINER" ]; then
-    docker-compose exec database gosu postgres psql -d pfb
+    docker compose exec database gosu postgres psql -d pfb
 fi
 
 popd
diff --git a/scripts/django-manage b/scripts/django-manage
@@ -25,7 +25,7 @@ then
     else
         pushd ..
 
-        docker-compose exec django python3 manage.py "${@}"
+        docker compose exec django python3 manage.py "${@}"
 
         popd
     fi

diff --git a/scripts/infra b/scripts/infra
@@ -24,10 +24,10 @@ function update_batch_definitions() {
     #       and reuse old version
     # TODO: This is a bit awkward. If we `plan` but never `apply`, we'll have created
     #       an orphaned job definition that never gets used.
-    docker-compose build
+    docker compose build
 
     ENVIRONMENT="${ENVIRONMENT:-staging}"
-    BATCH_ANALYSIS_JOB_NAME_REVISION=$(docker-compose run --rm \
+    BATCH_ANALYSIS_JOB_NAME_REVISION=$(docker compose run --rm \
         update-job-defs \
         "${ENVIRONMENT}-pfb-analysis-run-job.json" \
         "${PFB_AWS_ECR_ENDPOINT}/pfb-analysis:${GIT_COMMIT}")
@@ -58,24 +58,24 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
 
                 update_batch_definitions
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     --entrypoint rm \
                     terraform -rf .terraform/ terraform.tfstate*
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     terraform init \
                     -backend-config="bucket=${PFB_SETTINGS_BUCKET}" \
                     -backend-config="key=terraform/state"
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     terraform plan \
                           -var-file="${PFB_SETTINGS_BUCKET}.tfvars" \
                           -var="git_commit=${GIT_COMMIT}" \
                           -var="batch_analysis_job_definition_name_revision=${BATCH_ANALYSIS_JOB_NAME_REVISION}" \
                           -out="${PFB_SETTINGS_BUCKET}.tfplan"
                 ;;
             apply)
-                docker-compose run --rm terraform apply "${PFB_SETTINGS_BUCKET}.tfplan"
+                docker compose run --rm terraform apply "${PFB_SETTINGS_BUCKET}.tfplan"
 
                 # TODO (#843): Restore this once we've gotten Tilegarden deployment working again
                 # popd
@@ -84,24 +84,24 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
                 # aws s3 cp "s3://${PFB_SETTINGS_BUCKET}/tilegarden/.env" "./src/tilegarden/.env"
                 # aws s3 cp "s3://${PFB_SETTINGS_BUCKET}/tilegarden/claudia.json" "./src/tilegarden/claudia/claudia.json"
 
-                # docker-compose \
+                # docker compose \
                 #     -f docker-compose.yml \
                 #     -f docker-compose.test.yml \
                 #     run --rm --entrypoint yarn tilegarden deploy
                 ;;
             plan-mgmt)
                 update_batch_definitions
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     --entrypoint rm \
                     terraform -rf .terraform/ terraform.tfstate*
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     terraform init \
                     -backend-config="bucket=${PFB_SETTINGS_BUCKET}" \
                     -backend-config="key=terraform/state"
 
-                docker-compose run --rm \
+                docker compose run --rm \
                     terraform plan \
                           -var-file="${PFB_SETTINGS_BUCKET}.tfvars" \
                           -var="git_commit=${GIT_COMMIT}" \
@@ -110,7 +110,7 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
                           -out="${PFB_SETTINGS_BUCKET}-mgmt.tfplan"
                 ;;
             apply-mgmt)
-                docker-compose run --rm terraform apply "${PFB_SETTINGS_BUCKET}-mgmt.tfplan"
+                docker compose run --rm terraform apply "${PFB_SETTINGS_BUCKET}-mgmt.tfplan"
                 ;;
             update-tfvars)
                 aws s3 cp "${PFB_SETTINGS_BUCKET}.tfvars" \

diff --git a/scripts/run-local-analysis b/scripts/run-local-analysis
@@ -20,7 +20,7 @@ The first argument can be either the path to a shapefile (within the container,
 which to download the zipped shapefile. Anything that starts with 'http' will be
 treated as a URL.
 
-'extra args' should be a string that will be fed through to the `docker-compose run`
+'extra args' should be a string that will be fed through to the `docker compose run`
 command, e.g. '-T' to disable pseudo-TTY allocation.
 
 The `NB_POSTGRESQL_*` variables can point to a remote DB instance. Be sure you've
@@ -96,7 +96,7 @@ Results available locally at ${NB_OUTPUT_DIR}
         PFB_S3_RESULTS_PATH=""
     fi
 
-    docker-compose run --rm $EXTRA_ARGS \
+    docker compose run --rm $EXTRA_ARGS \
         -e AWS_PROFILE="${AWS_PROFILE}" \
         -e AWS_STORAGE_BUCKET_NAME="${AWS_STORAGE_BUCKET_NAME}" \
         -e NB_BOUNDARY_BUFFER=$NB_BOUNDARY_BUFFER \

diff --git a/scripts/server b/scripts/server
@@ -25,7 +25,7 @@ then
     else
         pushd ..
 
-        docker-compose up --build nginx django django-q angularjs tilegarden
+        docker compose up --build nginx django django-q angularjs tilegarden
 
         popd
     fi

diff --git a/scripts/test b/scripts/test
@@ -14,18 +14,18 @@ if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
     if [ "${1:-}" = "--help" ]; then
         usage
     else
-    	echo "running python tests..."
-    	docker-compose -f "${DIR}/../docker-compose.yml" run \
-			--rm --entrypoint "python3 manage.py test --noinput" django
+        echo "running python tests..."
+        docker compose -f "${DIR}/../docker-compose.yml" run \
+            --rm --entrypoint "python3 manage.py test --noinput" django
 
         echo "running Tilegarden tests..."
-        docker-compose run --rm --no-deps tilegarden yarn test
+        docker compose run --rm --no-deps tilegarden yarn test
 
         echo "running angularjs linter..."
-        docker-compose run --rm --no-deps angularjs gulp lint
+        docker compose run --rm --no-deps angularjs gulp lint
 
         echo "running angularjs build..."
-        docker-compose run --rm --no-deps angularjs gulp build
+        docker compose run --rm --no-deps angularjs gulp build
 
         echo "tests finished"
     fi