[POP-7251] Add entrypoint for using env vars to init dbt profile #2
Conversation
POP-7251 Add entrypoint to handle env vars setup dbt container profiles.yml
What We want to allow passing an environment variable to our dbt containers, and then the container's entrypoint will write those to the requisite files. This will allow us to more easily dynamically create/run the containers across different targets, without having to juggle |
| docker build \ | ||
| --build-arg REQUIREMENTS_FILE="${requirements_file}" \ | ||
| --tag "ghcr.io/popsql/${image_name}:${version}" \ | ||
| --build-arg DBT_VERSION="${version}" \ |
There was a problem hiding this comment.
Adding this was forgotten as part of f83c4f0, adding it here as I was using this script to test the entrypoint.
| "${BASE_DIR}" | ||
|
|
||
| echo "Successfully built image ${tag}" |
There was a problem hiding this comment.
Added the echo to just make copy/paste of the tag name generated from this script easier
|
|
||
| if [ -n "${BQ_KEYFILE}" ]; then | ||
| echo "${BQ_KEYFILE}" > /.dbt/bq_keyfile.json | ||
| fi |
There was a problem hiding this comment.
No fallback for BQ as the reference to it is in profiles.yml and so up to the user to make sure the path is correct to a file that exists within the container, be it via mounting or this env var.
There was a problem hiding this comment.
This is probably fine. The ENVs should be passed together (either both or neither). If the ENVs are passed, we can assume they agree about the location. Otherwise, we can assume the mounted files agree.
There was a problem hiding this comment.
Sure, simple enough to only prefer one or the other, not a combination. Done in 7fd3e54.
| @@ -1,5 +1,8 @@ | |||
| FROM python:3.8-slim-bullseye | |||
|
|
|||
| ENV DBT_PROFILES_DIR=/.dbt | |||
| ENV AWS_SHARED_CREDENTIALS_FILE=/.dbt/aws_credentials | |||
There was a problem hiding this comment.
This env var is documented as being supported by both aws cli as well as boto3 (what underlies the athena adapter).
There was a problem hiding this comment.
This is supported by the Redshift adapter, too, right?
There was a problem hiding this comment.
Believe so. Looks like the adapter relies on boto3 as well as the redshift_connector relies on it.
MasterOdin
deleted the
matt/pop-7251-add-entrypoint-to-handle-env-vars-setup-dbt-container
branch
PR modifies our images such that we now support the following three environment variables:
DBT_PROFILES->/.dbt/profiles.ymlAWS_CREDENTIALS->/.dbt/aws_credentialsBQ_KEYFILE->/.dbt/bq_keyfile.jsonIf the environment variable is not set, then we'll fallback to seeing if we can find the files as mounted (e.g.
/home/dbt/.dbt/profiles.yml). We use the custom directory (with appropriate env vars set so dbt/aws look there) to host these files to allow easy migration from path based mounting to env vars without worrying about overwriting anything if for a period of time both are used.