Skip to content

Docker Workflow for Development

Max Patiiuk edited this page Jul 31, 2024 · 12 revisions

Instructions for setting up the development environment

NOTE: Development Docker composition is not suitable for Production use.

This is a Development Docker composition of Specify 7

It has nice features like hot reloading and debugging support

However, it is not suited for production use due to memory leaks and security issues

Though, you can use it for evaluation of Specify 7

Members of Specify Collections Consortium get access to production-ready Docker compositions. Please contact us at [email protected] if you would like to get access

This tutorial assumes basic knowledge of Docker and Docker Compose.

You can read these tutorials on Docker and Docker Compose to get started.

Installation

  1. Install Docker and Docker compose:

    On macOS (assuming brew is installed):

    brew install --cask docker

    On Linux:

    sudo apt-get install \
      apt-transport-https \
      ca-certificates \
      curl \
      gnupg \
      lsb-release
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
    echo \
      "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
      $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io
    sudo groupadd docker
    sudo usermod -aG docker $USER
    newgrp docker
    sudo systemctl enable docker.service
    sudo systemctl enable containerd.service
    curl -L https://raw.githubusercontent.com/docker/compose-cli/main/scripts/install/install_linux.sh | sh
    docker plugin install --grant-all-permissions vieux/sshfs

    On Windows:

    a. Install and run Windows Subsystem For Linux (WSL) b. On your Linux Subsystem, add the following to /etc/wsl.conf (If the file does not exist, create it)

    [boot]
    systemd=true
    

    c. Repeat the steps from the previous On Linux section d. To run Specify, port 80 will need to be open on your computer. By default windows already has a service listening to port 80. (You can check which services are listening on which ports with netstat -aon | findstr <port>) As the Internet Information Services feature listens on port 80, there are two options

    • Disable Windows Internet Information Services

      • In the Start Menu, search for "features"
      • In the Windows Features window, find and uncheck "Internet Information Services"
      • Click OK
    • Change the port of the default home page

      • With Internet Information Services enabled, search for Internet Information Services Manager in the start menu
      • Under the Actions toolbar (located on the right), click on View Sites
      • Click on "Default Web Site". Under the Actions toolbar, click on "Bindings..." under Edit Site
      • Change the port of http to an unused port on your computer (If you need a port, 3200 is available by default)
  2. Checkout the repository (assuming git is installed)

    git clone https://github.com/specify/specify7/
    cd specify7
  3. Put a MySQL dump of the database you want to use into the ./seed-database folder (name of the file does not matter)

  4. Build the images and run the containers:

    docker compose up --build
    # or, if you are on a legacy version:
    docker-compose up --build

    On subsequent runs, you don't have to provide the "--build" argument, unless the list of front-end or back-end dependencies has changed.

  5. Make the code changes you want. Both the backend and the frontend would be updated in real-time.

    If you want to receive audible notifications when the rebuild process is finished, here are the instructions for setting that up

Afterward, you can stop the containers by pressing Ctrl+C or typing docker compose down in a separate terminal window.

Changing the schema version

  1. Open the ./docker-compose.yml file

  2. Find the following lines:

    specify6:                                                             
      image: specifyconsortium/specify6-service:6.8.01   
  3. Replace "6.8.01" with the desired version. Full list of available versions

  4. Rebuild the containers:

    docker compose up --build

Changing the branch

  1. Checkout the branch you want:

    git switch production

    Full list of branches

  2. Get the latest changes from GitHub:

    git pull
  3. If the branch you are switcing from and the new one have the same dependencies, you don't have to stop the containers and rebuild them. Simply run docker compose up if containers are not yet running.

    If there were some dependency changes, or you are not sure, run docker compose up --build to rebuild the containers.

Changing the database

  1. Put the database dump into ./seed-database. More information

  2. SSH into the MariaDB container. You can do so by pressing the >_ button next to the container name in Docker Dashboard, or by executing the following command in the terminal:

    # Assumming your container is named "specify7-mariadb-1":
    docker exec -it specify7-mariadb-1 env TERM=xterm-256color /bin/sh
  3. Run the following commands in order:

    # Navigate into the mounted volume in the container that holds the database dump
    cd /docker-entrypoint-initdb.d/
    # List the files in that directory. If everything is correct, you should see the name of your database dump file
    ls
    # Connect to the database
    mysql -uroot -proot
    # Delete current database from the MariaDB server. This will delete all data. If you need to preserver data, make a database dump using `mysqldump` first
    DROP DATABASE `specify`;
    # Create a blank database
    CREATE DATABASE `specify`;
    # Select the database
    USE `specify`;
    # Import the database from the dump. Assuming your dump file is called "dump.sql":
    SOURCE dump.sql;
    # Disconnect from MySQL
    EXIT;
    # Disconnect from the container
    exit;
  4. Run database migrations. This step is needed only once for a database before it is used with Specify 7 for the first time or after a Specify 7 update.

    docker exec -it specify7-specify7-1 ve/bin/python manage.py migrate

After this, refresh the "Specify 7" window in your browser and log in again if prompted.

A note of caution:

Make sure there is at most one file with the .sql extension in your seed-database directory at any given point. If you need to store multiple databases there just in case, than rename the files to not end with .sql (i.e, change extension to be .sql_)

This is because every time docker starts up, it will execute all of those SQL files. If you have more than one, they will conflict and may lead to unexpected circumstances

Connecting to database from outside the container

  1. Open ./docker-compose.yml

  2. Find the section that begins with mariadb:

  3. Add a new line:

    mysql:
       # Add these two lines line:
       ports:
       - '127.0.0.1:3306:3306'
  4. Rebuild the containers:

    docker compose up --build

This would make database accessible at localhost with port 3306.

Running the server on a remote machine

A common use case for using Docker is having the containers run on a dedicated machine. This frees up resources on your laptop for other tasks.

This can be atchived in Docker by using code syncing and Docker contexts.

Configuring Docker Contexts

Docker Contexts allows to manage remote Docker servers.

Read more about them here

And here is how to use them

For example:

# Create a context
docker context create maxpatiiuk-pc --default-stack-orchestrator=swarm --docker "host=tcp://192.168.50.165:2375" --description "Home Ubuntu PC"

Then, you can define default context like docker context use maxpatiiuk-pc or define an alias and prepend it to your commands:

alias dd="DOCKER_CONTEXT=maxpatiiuk-pc "
dd docker container ls

Configuring Code Sync

We need to update the remove server in real time as soon as any code changes are made. mirror is designed just for that task.

Instructions for setting up Code Sync with Docker

The commands I used:

# Add this to your .zshrc file:
alias sync_client='docker run --rm --init -it -u $(id -u):$(id -g) -v $(pwd):/data \
  quay.io/stephenh/mirror client \
  --local-root /data \
  --remote-root /data \
  --host 192.168.50.165 \
  --include "./seed-database" --exclude "./idea" --exclude "nohup.out"'
alias sync_server='docker run \
  --rm --init -it -u $(id -u):$(id -g) -v $(pwd):/data -p 49172:49172 \
  quay.io/stephenh/mirror server'

Then just run sync_server on the server and sync_client on the client machine.

Overwriting default preferences

The easiest way currently to override Specify settings that aren't connected to an environment variable in a Docker container is:

  1. Copy the local_specify_settings.py out of the container docker-compose exec specify7 cat /opt/specify7/settings/local_specify_settings.py > override_specify_settings.py
  2. Make the desired changes in override_specify_settings.py
  3. Mount the modified settings file on top of the file in the container by adding - "./override_specify_settings.py:/opt/specify7/settings/local_specify_settings.py" to the volumes section of specify7 service.
Clone this wiki locally