Add ability to create deployable zipfiles

[chosak]

This adds a new Python package to this repository that allows for the creation of deployable zipfiles that automatically create a virtual environment with all project requirements when they are unpacked.

This is functionality that today is provided by the drama-free-django repository; however, that project supports quite a few other optional behaviors that are no longer needed by cf.gov. This has motivated the migration of this deployable zipfile behavior into cfgov-refresh.

The main interface with this new behavior is a create.py script that can be invoked like this from the project root:

cfgov/deployable_zipfile/create.py
    ./cfgov                        # path to project code being deployed
    ./requirements/deployment.txt  # requirements to include in zipfile
    output_zipfile_name            # output filename, appended with .zip
    --extra-static=path/to/fonts   # optional, see below
    --extra-python=python3         # optional, see below

This is functionally similar to the no-drama build command in drama-free-django. This script has no external (non-Python) dependencies and thus does not need to be run inside of a virtual environment with regular cfgov-refresh requirements.

The optional --extra-static argument can be used to include additional static files in the zip, in a static.in subdirectory. This can be used to store the private font files used by cf.gov, documented here.

Like drama-free-django, the deployable zipfile creation process involves creating Python wheels for all requirements. This uses the version of Python that you use to invoke the script.

You can optionally provide a second version of Python (e.g. Python 3) so that the zip contains wheels for multiple versions. This makes it possible to use the same zipfile to run cf.gov in Python 2 and 3. This code brings in logic from an existing drama-free-django PR by @rosskarchner.

The deployable zipfile can be deployed in a way that is almost completely compatible with how drama-free-django zipfiles work today, like this:

path/to/deployable.zip -d /path/to/deploy/zip

If the destination directory does not exist, it will be created. To maintain backwards compatibility, the zip is extracted to a current subdirectory of the specified location.

In this subdirectory, a venv subdirectory is created to hold a Python virtual environment with all requirements.

This virtual environment is configured with a special behavior related to environment variables, consistent with drama-free-django: if an environment.json file exists within the root of the deployed location (the current subdirectory), it will be loaded as environment variables on Python startup. This means that even a regular invocation of /path/to/venv/bin/python will include those variables in the environment.

A new Docker-based setup for generating these deployable zipfiles has been created under docker/deployable-zipfile, comparable to the existing docker/drama-free-django setup.

This can be invoked locally with:

docker/deployable-zipfile/build.sh

This will generate a cfgov_current_build.zip file in the root of this project.

NOTE: The major difference between zipfiles generated with this approach and those generated with drama-free-django are that DFD zips include a wsgi.py file, and these do not. In order to use these zipfiles with our current deployment process, you'll need to make an additional change to our Apache wsgi.conf file here to point to ${CFGOV_CURRENT}/cfgov/cfgov/wsgi.py:

diff --git a/cfgov/apache/conf.d/wsgi.conf b/cfgov/apache/conf.d/wsgi.conf
index 0dc0222..aaa56a4 100644
--- a/cfgov/apache/conf.d/wsgi.conf
+++ b/cfgov/apache/conf.d/wsgi.conf
@@ -3,7 +3,7 @@ ServerName consumerfinance.gov
 WSGIApplicationGroup %{GLOBAL}
 WSGIDaemonProcess django home=${CFGOV_CURRENT} python-home=${CFGOV_CURRENT}/venv processes=${APACHE_PROCESS_COUNT} threads=15 display-name=%{GROUP}
 WSGIProcessGroup django
-WSGIScriptAlias / ${CFGOV_CURRENT}/wsgi.py
+WSGIScriptAlias / ${CFGOV_CURRENT}/cfgov/cfgov/wsgi.py
 
 <Directory ${CFGOV_PATH}>
     Require all granted

I haven't included that change in this PR as it would break the existing DFD-based process.

Testing

The best way to test this is using the Docker setup mentioned above, after making the Apache config change also mentioned above. The build generated with that setup can be used successfully with our internal Ansible/Docker process.

Todos

• These changes need some additional test coverage; I will add this.
• If we plan to move to this, we should add a page about this to the cfgov-refresh documentation.
• There are a number of things added here to maintain backwards compatibility with drama-free-django to smooth the transition; these would be good to remove if we migrate to this. These include:
  • Stop using a current subdirectory as part of deploying the zipfile.
  • Stop including all of the zipfile contents in its final deploy location; for example we don't need to keep the wheels around once we've installed them.
• Although the file generated here is Python 3 compatible, some additional work is needed to make this work properly with our Ansible setup.

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows the standards laid out in the CFPB development guidelines
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated
• Reviewers requested with the Reviewers tool ➡️

[hkeeler]

Took a first pass. Overall, looks good, though I'm still working through some of the details. Just the one question about the file permissions on the zip file so far.

[rosskarchner]

Thinking about the wsgi.conf stuff: what if the extract script created a symlink from current/wsgi.py to cfgov/cfgov/wsgi.py? This serves no long-term value, but if my mental model of this stuff is correct, it would avoid that breakage you talk about. Once we've switched to this new deployment format everywhere, we could remove that function and THEN make the change to wsgi.conf

[rosskarchner]

Maybe setting --login via SHELL (see our top-level Dockerfile) is better than a dummy entrypoint?

[chosak]

I copied this behavior from @hkeeler's work with the DFD setup in #5145. I defer to the two of you as to the best approach.

[rosskarchner]

I could imagine arguments for both. SHELL is kind of an obscure Dockerfile feature(especially compared to custom entrypoints), so perhaps this approach would be more obvious to a reader.

I defer to @hkeeler and @wpears ;)

[hkeeler]

I agree, it seems like it should work that way, and I tried for a while to make that work on #5144. Unfortunately, SHELL in Dockerfile only affects ENTRYPOINT and CMD when used in shell form, and that was causing other issues with the shell env (can't remember the exact issue). So yeah, for now, probably easiest to just keep it as is.

[chosak]

the extract script created a symlink from current/wsgi.py to cfgov/cfgov/wsgi.py

That's a good suggestion, as it'll smooth the migration path. I've added that behavior in d5ea0d6.

[willbarton]

This script has no external (non-Python) dependencies and thus does not need to be run inside of a virtual environment with regular cfgov-refresh requirements.