Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move root-required bits of drama-free-django build into Dockerfile #5145

Merged
merged 16 commits into from
Jul 29, 2019
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docker/drama-free-django/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FROM centos:6

ENV SCL_PYTHON_VERSION python27

# Disables pip cache, which reduces build time, and suppresses warnings when run a non-root.
hkeeler marked this conversation as resolved.
Show resolved Hide resolved
ENV PIP_NO_CACHE_DIR true

ENV DFD_DIR /src/cfgov-refresh

# Must be world writable since alternate uid:gid may be patched in at `docker run` time.
RUN mkdir -p ${DFD_DIR} && chmod 777 ${DFD_DIR}
WORKDIR ${DFD_DIR}

# Sets a consistent $HOME no matter which user the container runs under. This prevents
# permissions issues caused by Docker's default `/` home directory.
ENV HOME /tmp/dfd-home
RUN mkdir -p ${HOME} && chmod 777 ${HOME}

# Install dependencies
# NOTE: You MUST upgrade pip before using it further. The version packaged with SCL has issues
# with both setuptools and the PIP_NO_CACHE_DIR envvar (hence the --no-cache-dir override).
RUN yum install -y centos-release-scl && \
curl -sL https://rpm.nodesource.com/setup_10.x | bash - && \
curl -sL https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo && \
yum install -y ${SCL_PYTHON_VERSION} gcc git nodejs yarn && \
echo "source scl_source enable ${SCL_PYTHON_VERSION}" > /etc/profile.d/scl_python.sh && \
source /etc/profile && \
pip install --no-cache-dir -U pip && \
pip install -U git+https://github.com/cfpb/drama-free-django.git

COPY _build.sh _test.sh docker-entrypoint.sh ./

ENTRYPOINT ["./docker-entrypoint.sh"]
16 changes: 16 additions & 0 deletions docker/drama-free-django/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Docker-based drama-free-django build and test tools

## Build

Run the `build.sh` script from the project root:

```sh
Expand All @@ -6,6 +10,8 @@ docker/drama-free-django/build.sh

This will run a CentOS 6 container to generate a [drama-free-django](https://github.com/cfpb/drama-free-django) release artifact in the project root named `cfgov_current_build.zip`.

## Test

To run a basic test of the artifact:

```sh
Expand All @@ -14,3 +20,13 @@ docker/drama-free-django/test.sh

This will run a CentOS 6 container to validate the built artifact by extracting it and running Django
[`collectstatic`](https://docs.djangoproject.com/en/1.11/ref/contrib/staticfiles/#collectstatic).

## Notes

1. When running the container as a user that exists on the host, but not in the container, you may notice a warning similar to:

```
/usr/bin/id: cannot find name for user ID 502
```
This is not anything to worry about. It simply means the uid/gid don't match any users/groups setup in the container.
13 changes: 0 additions & 13 deletions docker/drama-free-django/_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,6 @@ if [ ! -d "$cfgov_refresh_volume" ]; then
exit 1
fi

# Install build requirements.
yum install -y centos-release-scl
yum install -y gcc git python27

source /opt/rh/python27/enable

pip install -U pip
pip install -U git+https://github.com/cfpb/drama-free-django.git

curl -sL https://rpm.nodesource.com/setup_10.x | bash -
curl -sL https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo
yum install -y nodejs yarn

# Run the frontend build.
pushd "$cfgov_refresh_volume"
./frontend.sh production
Expand Down
11 changes: 4 additions & 7 deletions docker/drama-free-django/_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ set -x

artifact_filename=cfgov_current_build.zip
artifact_volume=/cfgov
dfd_test_dir=/tmp/dfd-test/release

# Verify that the artifact volume has been mapped.
if [ ! -d "$artifact_volume" ]; then
Expand All @@ -16,15 +17,11 @@ if [ ! -d "$artifact_volume" ]; then
exit 1
fi

# Install runtime requirements.
yum install -y centos-release-scl
yum install -y python27

source /opt/rh/python27/enable

# Extract the artifact in /tmp.
cp "$artifact_volume/$artifact_filename" /tmp
cd /tmp
mkdir -p $dfd_test_dir
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but I'm curious why you made a new directory for this. I figured that working in /tmp was fine since the container is ephemeral.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue was that by having it in /tmp, it resulted in directories being created in the / root directory, and the non-root user didn't have permission to create those directories. An alternative would be to open up permissions on /, but this seems like a better option.

cp "$artifact_volume/$artifact_filename" $dfd_test_dir
cd $dfd_test_dir
python "./$artifact_filename"

cd current
Expand Down
10 changes: 9 additions & 1 deletion docker/drama-free-django/build.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
#!/usr/bin/env bash

docker run -v `pwd`:/cfgov centos:6 /cfgov/docker/drama-free-django/_build.sh
set -e

docker build -t cfgov-dfd-builder docker/drama-free-django

docker run \
--rm \
-u $(id -u):$(id -g) \
-v $(pwd):/cfgov \
cfgov-dfd-builder ./_build.sh
7 changes: 7 additions & 0 deletions docker/drama-free-django/docker-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash --login
# This entrypoint is used primarily as means of setting up a consistent
# shell environment no matter which user the process runs as. By using
# --login, it guarantees /etc/profile is always sourced, unlike the
# non-login, non-interactive shell you get by default with `docker run`.

exec "$@"
10 changes: 9 additions & 1 deletion docker/drama-free-django/test.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
#!/usr/bin/env bash

docker run -v `pwd`:/cfgov centos:6 /cfgov/docker/drama-free-django/_test.sh
set -e

docker build -t cfgov-dfd-builder docker/drama-free-django

docker run \
--rm \
-u $(id -u):$(id -g) \
-v $(pwd):/cfgov \
cfgov-dfd-builder ./_test.sh
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't hold up this PR, but, if we are going to maintain this DFD testing capability, I wonder if it's worth entertaining the idea of a distinct Dockerfile just for that purpose. We don't really need everything in the "cfgov-dfd-builder" image just to run the DFD image, and it would be nice to actually determine what is needed. But probably thinking more about that should wait until/if we want to think about migrating Ansible code here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it does feel a little awkward to have the two scripts that both build the same image, but my assumption was that generally when you'd run test.sh, you would have run build.sh just before it, and so the docker build part would be fully cached and be pretty instant, but I didn't want to make that a requirement to running test.sh, so it's duplicated in both scripts.

As for a separate image, yeah we could. It just seemed like it'd be better to only maintain one Dockerfile that could do both. But if there's a scenario where we'd be running just test.sh, maybe that starts to make more sense.