Skip to content

Commit

Permalink
⚡ 🐳 Use multi-stage docker builds: split tests
Browse files Browse the repository at this point in the history 
This is the improvement to the Dockerfile to split tests and runtime.
See details on the README updated with instructions.
  • Loading branch information
@marcellodesales
marcellodesales committed Mar 8, 2023
1 parent 6ae50bd commit 7ae6f3c
Show file tree
Hide file tree
Showing 3 changed files with 138 additions and 71 deletions.
36 changes: 30 additions & 6 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,45 @@
FROM python:3.8.16-alpine3.17
FROM python:3.8.16-alpine3.17 AS builder

# The requirements to build SqlAlchemy includes compiling from source-code, at least in Alpine
# It does require GCC to compile also the MySQL client from sources.
# https://stackoverflow.com/questions/59554493/unable-to-fire-a-docker-build-for-django-and-mysql/59554601#59554601
# https://stackoverflow.com/questions/8878676/compile-error-g-error-trying-to-exec-cc1plus-execvp-no-such-file-or-dir
RUN apk add musl-dev python3-dev mariadb-dev gcc build-base bash

WORKDIR /viasat/minitwit

COPY requirements*.txt .
COPY requirements.txt .

RUN pip install -r requirements.txt

ENV FLASK_APP=minitwit.py
ENV LC_ALL=en_US.utf-8
ENV LANG=en_US.utf-8

COPY . /viasat/minitwit
COPY ./static /viasat/minitwit/static
COPY ./templates /viasat/minitwit/templates

COPY *.py /viasat/minitwit
COPY *.sql /viasat/minitwit

####
#### tester image that installs the dev dependencies and executes the test cases during build
#### and during a container execution! The build will fail if any test case fails.
#### The test image is good for inspection of test environments and ca be used in CI/CD envs.
####
FROM builder AS tester

COPY requirements-dev.txt .
RUN pip install -r requirements-dev.txt

# Note that we are executing the tests here because we don't want to build
# A runtime image before the tests are executed successfully!
RUN pytest test_minitwit.py
CMD ["pytest", "test_minitwit.py"]

###
### runtime environment containing only the runtime dependencies.
###
FROM builder AS runtime

# All the other resources are there, so let's add the run-app as it's part of the runtime
COPY run-app .

ENTRYPOINT ["bash", "/viasat/minitwit/run-app"]
153 changes: 89 additions & 64 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,20 +53,44 @@ Press CTRL+C to quit
2. Initialize a container with `docker compose up`

```console
$ docker compose up
[+] Running 1/1
⠿ Container minitwit-minitwit-1 Recreated 0.5s
Attaching to minitwit-minitwit-1
minitwit-minitwit-1 | [2023-03-07 21:27:11,511] INFO in minitwit: Using local db sqlite:////var/minitwit/minitwit.db
minitwit-minitwit-1 | * Serving Flask app 'minitwit.py'
minitwit-minitwit-1 | * Debug mode: off
minitwit-minitwit-1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
minitwit-minitwit-1 | * Running on all addresses (0.0.0.0)
minitwit-minitwit-1 | * Running on http://127.0.0.1:5000
minitwit-minitwit-1 | * Running on http://192.168.192.2:5000
minitwit-minitwit-1 | Press CTRL+C to quit
^CGracefully stopping... (press Ctrl+C again to force)
[+] Running 0/1
$ docker compose up --build minitwit-runtime
[+] Building 13.5s (17/17) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.39kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8.16-alpine3.17 1.1s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [builder 1/9] FROM docker.io/library/python:3.8.16-alpine3.17@sha256:8518dd6657131d938f283ea97385b1db6724e35d45ddab6cd1c583796e35566a 0.0s
=> => resolve docker.io/library/python:3.8.16-alpine3.17@sha256:8518dd6657131d938f283ea97385b1db6724e35d45ddab6cd1c583796e35566a 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 767B 0.0s
=> CACHED [builder 2/9] RUN apk add musl-dev python3-dev mariadb-dev gcc build-base bash 0.0s
=> CACHED [builder 3/9] WORKDIR /viasat/minitwit 0.0s
=> CACHED [builder 4/9] COPY requirements.txt . 0.0s
=> CACHED [builder 5/9] RUN pip install -r requirements.txt 0.0s
=> CACHED [builder 6/9] COPY ./static /viasat/minitwit/static 0.0s
=> CACHED [builder 7/9] COPY ./templates /viasat/minitwit/templates 0.0s
=> CACHED [builder 8/9] COPY *.py /viasat/minitwit 0.0s
=> CACHED [builder 9/9] COPY *.sql /viasat/minitwit 0.0s
=> CACHED [runtime 1/1] COPY run-app . 0.0s
=> exporting to oci image format 12.2s
=> => exporting layers 0.0s
=> => exporting manifest sha256:961d6ad7ce1a276bca5785193769284de530c6fab4af3993d79a25212b222df6 0.0s
=> => exporting config sha256:ff6eb4ca21b161d3a710f8302313e4506042511aa611356e95f5144897a22acb 0.0s
=> => sending tarball 12.2s
=> importing to docker 0.2s
[+] Running 1/0
⠿ Container minitwit-minitwit-runtime-1 Created 0.0s
Attaching to minitwit-minitwit-runtime-1
minitwit-minitwit-runtime-1 | [2023-03-08 01:37:42,142] INFO in minitwit: Using local db sqlite:////var/minitwit/minitwit.db
minitwit-minitwit-runtime-1 | * Serving Flask app 'minitwit.py'
minitwit-minitwit-runtime-1 | * Debug mode: off
minitwit-minitwit-runtime-1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
minitwit-minitwit-runtime-1 | * Running on all addresses (0.0.0.0)
minitwit-minitwit-runtime-1 | * Running on http://127.0.0.1:5000
minitwit-minitwit-runtime-1 | * Running on http://192.168.192.2:5000
minitwit-minitwit-runtime-1 | Press CTRL+C to quit
```

3. Go to the browser at http://127.0.0.1:5000.
Expand Down Expand Up @@ -142,38 +166,63 @@ test_minitwit.py ....

1. Build a docker image with the runtime needed

> **NOTE**: The docker image `viasat/minitwit` is created locally.
> **NOTE**: The docker image `viasat/minitwit-test` is created locally.
The tests are executed within the docker build process and are set as CMD to execute as containers.

```console
$ docker compose build
[+] Building 13.0s (11/12)
[+] Building 13.2s (11/12)
[+] Building 13.3s (11/12)
[+] Building 20.9s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 767B 0.1s
=> [internal] load .dockerignore 0.1s
$ docker compose build minitwit-test
[+] Building 63.0s (18/18) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.08kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8.16-alpine3.17 4.0s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [1/6] FROM docker.io/library/python:3.8.16-alpine3.17@sha256:8518dd6657131d938f283ea97385b1db6724e35d45ddab6cd1c583796e35566a 0.0s
=> [internal] load metadata for docker.io/library/python:3.8.16-alpine3.17 0.4s
=> [builder 1/9] FROM docker.io/library/python:3.8.16-alpine3.17@sha256:8518dd6657131d938f283ea97385b1db6724e35d45ddab6cd1c583796e35566a 0.0s
=> => resolve docker.io/library/python:3.8.16-alpine3.17@sha256:8518dd6657131d938f283ea97385b1db6724e35d45ddab6cd1c583796e35566a 0.0s
=> [internal] load build context 0.3s
=> => transferring context: 31.23kB 0.3s
=> CACHED [2/6] RUN apk add musl-dev python3-dev mariadb-dev gcc build-base bash 0.0s
=> CACHED [3/6] WORKDIR /viasat/minitwit 0.0s
=> CACHED [4/6] COPY requirements*.txt . 0.0s
=> CACHED [5/6] RUN pip install -r requirements.txt 0.0s
=> [6/6] COPY . /viasat/minitwit 0.3s
=> exporting to oci image format 16.1s
=> => exporting layers 0.2s
=> => exporting manifest sha256:2219c217928079623597bf133b4ca8a41ae8fe8574dcc8dc5216498402bd801d 0.0s
=> => exporting config sha256:31e750b170b03b392f698b3dd115a69251ee8a3e5cca127f8d0bace682023210 0.0s
=> => sending tarball 15.9s
=> [internal] load build context 0.0s
=> => transferring context: 490B 0.0s
=> CACHED [builder 2/9] RUN apk add musl-dev python3-dev mariadb-dev gcc build-base bash 0.0s
=> CACHED [builder 3/9] WORKDIR /viasat/minitwit 0.0s
=> CACHED [builder 4/9] COPY requirements.txt . 0.0s
=> CACHED [builder 5/9] RUN pip install -r requirements.txt 0.0s
=> CACHED [builder 6/9] COPY ./static /viasat/minitwit/static 0.0s
=> CACHED [builder 7/9] COPY ./templates /viasat/minitwit/templates 0.0s
=> CACHED [builder 8/9] COPY *.py /viasat/minitwit 0.0s
=> CACHED [builder 9/9] COPY *.sql /viasat/minitwit 0.0s
=> [tester 1/3] COPY requirements-dev.txt . 0.0s
=> [tester 2/3] RUN pip install -r requirements-dev.txt 12.2s
=> [tester 3/3] RUN pytest test_minitwit.py 30.5s
=> exporting to oci image format 19.5s
=> => exporting layers 0.3s
=> => exporting manifest sha256:20a82b1dfbe5398ebb6e8f65917221f2a5bf2a932fb5f19d9e62e19b97ab4450 0.0s
=> => exporting config sha256:64a9ebf40a546f12f97d3d74903846dc170bc776f110feb0cde2a508a51b300b 0.0s
=> => sending tarball 19.0s
=> importing to docker
```

2. Initialize a docker container with the current sources mounted as a volume.
2. You can re-execute the test cases

```console
$ docker run -ti viasat/minitwit-test
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
==================================================================== test session starts =====================================================================
platform linux -- Python 3.8.16, pytest-5.3.2, py-1.11.0, pluggy-0.13.1
rootdir: /viasat/minitwit
collected 4 items

test_minitwit.py .... [100%]

====================================================================== warnings summary ======================================================================
test_minitwit.py::test_register
/viasat/minitwit/minitwit.py:200: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
the_db.execute(query.strip() + ';')

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============================================================== 4 passed, 1 warning in 29.42s ================================================================
```

3. You can inspect the tests by initializing a docker container with the current sources mounted as a volume.

> **NOTE**: You need to override the entrypoint with bash.
Expand All @@ -187,31 +236,7 @@ $ docker run --platform linux/amd64 -ti -w $(pwd) -v $(pwd):$(pwd) --entrypoint
93078096ab1b:/Users/mdesales/dev/github.com/marcellodesales/minitwit#
```

> **NOTE**: If you get a warning while running on Mac M1, use the `--platform linux/amd64` to pull a linux image.
At this point, you are inside the runtime container.

2. Install dev dependencies inside the container.

```console
6e25763052d9:/Users/mdesales/dev/github.com/marcellodesales/minitwit# pip install -r requirements-dev.txt
Collecting pytest==5.3.2
Downloading pytest-5.3.2-py3-none-any.whl (234 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 234.5/234.5 KB 1.5 MB/s eta 0:00:00
Collecting attrs>=17.4.0
Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 KB 5.5 MB/s eta 0:00:00
Collecting more-itertools>=4.0.0
Downloading more_itertools-9.1.0-py3-none-any.whl (54 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.2/54.2 KB 4.3 MB/s eta 0:00:00
Collecting py>=1.5.0
Downloading py-1.11.0-py2.py3-none-any.whl (98 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.7/98.7 KB 7.1 MB/s eta 0:00:00
Collecting packaging
Downloading packaging-23.0-py3-none-any.whl (42 kB)
```

3. Execute the test cases from within the container
4. Execute the test cases from within the container

```console
602f533b1fa6:/Users/mdesales/dev/github.com/marcellodesales/minitwit# pytest test_minitwit.py
Expand Down
20 changes: 19 additions & 1 deletion docker-compose.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,29 @@ version: "3.8"

services:

minitwit:
####
#### Executes the test cases locally within a test image and prepares the container for execution.
####
#### docker compose build minitwit-test
####
minitwit-test:
image: viasat/minitwit-test
platform: linux/amd64
build:
context: .
target: tester

####
#### Runs the runtime server locally with the port mapping local:container
####
#### docker compose up --build minitwit-runtime
####
minitwit-runtime:
image: viasat/minitwit
platform: linux/amd64
build:
context: .
target: runtime
ports:
- 4000:5000
volumes:
Expand Down

0 comments on commit 7ae6f3c

Please sign in to comment.