Skip to content

Latest commit

 

History

History
61 lines (47 loc) · 4.88 KB

README.md

File metadata and controls

61 lines (47 loc) · 4.88 KB
Raw

Pangeo Docker Images

Build Status Publish Status DockerHub Version

Latest DockerHub Images: https://hub.docker.com/orgs/pangeo/repositories

Image Description Size Pulls
base-image Foundational Dockerfile for builds
base-notebook minimally functional image for pangeo hubs
pangeo-notebook above + core earth science analysis packages
ml-notebook above + GPU-enabled tensorflow2

Design:

  1. compatible with Pangeo BinderHubs and JupyterHubs
  2. compatible with Repo2Docker Python configuration files
  3. reproducible build process and explicit conda package lists
  4. small size, fast build
  5. easy to customize

Everything stems from the Dockerfile in the base-image folder. The base-image configures default settings for Conda and Dask with condarc.yml and dask_config.yml files. The base-image is not meant to run on its own, it is the common foundation for -notebook images that install Python packages including JupyerLab and lab extensions. Lists of Conda packages for each image are specified in an environment.yml in each -notebook folder, and compatible Dask and Jupyter packages are guaranteed by specifying the pangeo-notebook conda metapackage.

You can pre-solve for compatible environments locally with conda-lock to convert the environment.yml file to a conda-linux-64.lock file which is an explicit list of compatible packages solved by Conda. The major advantage of doing this is that if you rebuild at a later date the resulting Conda environment is identical, which improves reproducibility. For this reason, when building off of the base-image, any existing conda-linux-64.lock file takes precedence over the environment.yml file.

Image tagging and "continuous building"

All images are public on DockerHub: https://hub.docker.com/orgs/pangeo

  • Pull requests from forks trigger rebuilding all image but can't push to DockerHub because they don't have access to repo secrets for authentication.

  • pangeo/base-notebook:master corresponds to current "staging" image in sync with master branch. Built with every commit to master. Also tagged with short GitHub short SHA pangeo/base-notebook:2639bd3.

  • Tags pushed to GitHub represent "production" releases with corresponding tags on dockerhub pangeo/pangeo-notebook:2020.03.11. The latest tag also corresponds to the most recent GitHub tag.

To build locally (and make a PR)

Important!!! regenerating the conda-lockfile is not currently done through CI. You need to create this file locally and include it in your PR. You'll need at least Conda installed, and Docker if you want to build and test locally.

# create a fork of this repo and clone it locally
git clone https://github.com/mygithub/pangeo-stacks-dev
cd pangeo-stacks-dev
# Install conda-lock
conda env create -f environment-condalock.yml
git checkout -b change-pangeo-notebook

edit pangeo-notebook/environment.yml to change packages! Note that make pangeo-notebook is a convenient shortcut to build and test. See the Makefile for specific commands that are run. For example, you can just run conda-lock and don't have to build and test locally.

make pangeo-notebook
git commit -a -m "added x packages, changed x version"
git push
# go to github to create PR, or use github cli https://cli.github.com

Use base-image for BinderHub-compatible repos

https://github.com/scottyhq/pangeodev-binder

To run locally

docker run -it --rm -p 8888:8888 pangeo/base-notebook:latest jupyter lab --ip 0.0.0.0