diff --git a/Lesson-02.qmd b/Lesson-Contain.qmd
similarity index 74%
rename from Lesson-02.qmd
rename to Lesson-Contain.qmd
index 46c9b40..31e6ae0 100644
--- a/Lesson-02.qmd
+++ b/Lesson-Contain.qmd
@@ -2,7 +2,7 @@
title: "Biostat 823 - Containerization"
author: "Hilmar Lapp"
institute: "Duke University, Department of Biostatistics & Bioinformatics"
-date: "Sep 14, 2023"
+date: "Oct 10, 2024"
format:
revealjs:
slide-number: true
@@ -23,7 +23,7 @@ Reproducibility of computational research faces four major challenges^[Boettiger
## "Dependency Hell"
-* Software dependencies have themselves dependencies recursively
+* Software dependencies have themselves dependencies, recursively
* Dependencies can be be often difficult to install (require compilation, manual "tweaks" due local OS or other differences, etc)
* Required version may conflict with that required by other software, or may not work with the local OS version, making it impossible to install.
- The likelihood of conflicts is particularly high on shared computing environments.
@@ -42,7 +42,7 @@ Reproducibility of computational research faces four major challenges^[Boettiger
- This can happen anywhere in the dependency chain.
* Dependencies can also become unmaintained or end-of-life
- Can result in removal from package repositories.
- - Python 2.x example
+ - Python 2.x example; CRAN package removal by policy
## Virtual Machine as solution?
@@ -65,10 +65,10 @@ Reproducibility of computational research faces four major challenges^[Boettiger
- 2007 (v1) and 2013--16 (v2): Linux control groups ([cgroups](https://en.wikipedia.org/wiki/Cgroups))
- 2008: [Linux Containers (LXC)](https://en.wikipedia.org/wiki/LXC)
- 2013: [Docker](https://www.docker.com)
-- 2015: [Singularity](https://en.wikipedia.org/wiki/Singularity_(software))
+- 2015: [Singularity](https://en.wikipedia.org/wiki/Singularity_(software)) and ([since 2021](https://apptainer.org/news/community-announcement-20211130/)) [Apptainer](https://apptainer.org)
::: aside
-There are many [OS-level virtualization](https://en.wikipedia.org/wiki/OS-level_virtualization) systems. LXC, Docker, and Singularity are by far the most important ones.
+There are many [OS-level virtualization](https://en.wikipedia.org/wiki/OS-level_virtualization) systems. LXC, and especially *Docker*, and *Apptainer/Singularity* are by far the most important ones.
:::
## Properties of containerized processes {.smaller}
@@ -83,8 +83,8 @@ There are many [OS-level virtualization](https://en.wikipedia.org/wiki/OS-level_
![](images/docker-for-mac.png){fig-align="right" width="30%" style="float: right"}
* On Windows and macOS, requires a Linux VM
- - Part of the Docker installation (uses WSL on Windows; LinuxKit / Hypervisor Framework on macOS)
- - Unsupported by Singularity
+ - Part of the Docker installation (uses [WSL/WSL2](https://learn.microsoft.com/en-us/windows/wsl/about) on Windows; LinuxKit / Hypervisor Framework on macOS)
+ - Apptainer can [use WSL/WSL2 on Windows](https://apptainer.org/docs/admin/main/installation.html#windows), with access to GPUs; [on macOS](https://apptainer.org/docs/admin/main/installation.html#mac), requires [Lima](https://lima-vm.io) as VM host (no GPU)
::: aside
Figure modified from [Gianluca Quercini, Cloud computing -- Docker Primer](https://gquercini.github.io/courses/cloud-computing/references/docker-primer/)
@@ -98,18 +98,19 @@ Figure modified from [Gianluca Quercini, Cloud computing -- Docker Primer](https
From [ELIXIR containers nextflow: Docker](https://biocorecrg.github.io/ELIXIR_containers_nextflow/docker.html)
:::
-## Singularity: Containers for HPC
+## Apptainer / Singularity: Containers for HPC {.smaller}
* HPC systems are shared computing environments
- Docker daemon runs as root, processes within container can run as root
- Not permissible on a shared computing environment
-* Singularity does not require elevated privileges
+* Apptainer does not require elevated privileges
- Launcher run by user, not a daemon run by root
- Processes inside container run as same user as outside
-* Singularity containers can be built (bootstrapped) from (many) Docker container images
+* Apptainer containers can be built (bootstrapped) from (many) Docker container images
- Most scientific software containers are compatible
+ - Issues can occur for containers that run services under a privileged user (httpd, database server, etc)
-## Singularity architecture vs Docker
+## Apptainer / Singularity vs Docker
![](images/singularity_architecture.png)
@@ -162,15 +163,6 @@ CMD ["java", "-jar", "picard.jar"]
- [GitHub Packages](https://ghcr.io) Repository (includes container images)
- Gitlab container registry (gitlab-registry.oit.duke.edu for Duke OIT's Gitlab installation)
-## (Note) Container images are layered
-
-* Container file system is a [union mount](https://en.wikipedia.org/wiki/Union_mount)
- - [OverlayFS](https://en.wikipedia.org/wiki/OverlayFS) supported by Linux kernel since 2014
- - Allows layering image content
- - Each command in the definition creates a layer
- - Layers are cached for image builds and pulls
-* [Best practices for container definition](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) include controlling layer cache invalidation
-
## (Note) Multi-stage builds
* Build layers are read-only
@@ -179,39 +171,50 @@ CMD ["java", "-jar", "picard.jar"]
- Multiple container builds in one container definition
- Use to retain build products but not the software environment needed to create them (which can be large)
-## (Note) Build docker, run singularity
+## (Note) Build docker, run apptainer
-* Building Docker images typically more flexible
- - No Singularity Desktop version for Windows or macOS (requires Linux VM instead)
- - `singularity build` normally requires `sudo` privileges
-* Singularity can use (most) Docker images directly
+* Building Docker container images typically more flexible
+ - No Apptainer Desktop version for Windows or macOS (requires Linux VM instead)
+ - Container build instructions may cause problems with `apptainer build` in unprivileged environment (which uses `--fakeroot` by default)
+* Apptainer can use (most) Docker images directly
- Can download and run in one step:
```shell
- $ singularity run docker://
+ $ apptainer run docker://
```
-* Use `--fakeroot` for `singularity build` in a non-privileged environment
-## (Note) Mounting data into the container
+## (Note) Mounting data into container {.smaller}
Requires bind mount at container runtime (`docker run`):
-* `--volume :` (Docker)
-* `--bind :` (Singularity)
+* [Docker](https://docs.docker.com/engine/storage/bind-mounts/):
+
+ `--volume :`
+
+ or
+
+ `--mount type=bind,source=,target=`
+
+ Using `--mount` generates an error if target directory (or file) doesn't exist
+
+* [Apptainer](https://apptainer.org/docs/user/main/bind_paths_and_mounts.html#user-defined-bind-paths):
+
+ `--bind :`
+
+ Or use `--mount` (see above).
* Can be used for directories and files
-* Using `--mount` generates an error if target directory (or file) doesn't exist
## Resources (I)
* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)
* [Docker command line reference](https://docs.docker.com/engine/reference/commandline/cli/)
-* [Singularity file reference](https://docs.sylabs.io/guides/latest/user-guide/definition_files.html)
-* [Singularity command line reference](https://docs.sylabs.io/guides/latest/user-guide/cli.html)
-* [Open Containers Initiative (OCI) standard for annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md)
+* [Apptainer file reference](https://apptainer.org/docs/user/main/definition_files.html)
+* [Apptainer command line reference](https://apptainer.org/docs/user/main/cli.html)
+* [Open Containers Initiative (OCI) standard for annotations](https://specs.opencontainers.org/image-spec/annotations/)
## Resources (II)
-* [Introduction to Docker](https://carpentries-incubator.github.io/docker-introduction/) (Carpentries Incubator lesson)
+* [Intro to Docker Workshop](https://imageomics.github.io/docker-workshop/) (Based on Carpentries Incubator lesson)
+* [Into to Singularity Workshop](https://carpentries-incubator.github.io/singularity-introduction/) (Carpentries Incubator lesson)
* [DCC OnDemand](https://dcc-ondemand-01.oit.duke.edu/)
* [Jupyter Docker Stacks](https://jupyter-docker-stacks.readthedocs.io/)
- Customized [Biostat Jupyter Docker container](https://github.com/Duke-GCB/biostat-jupyter)
-* [Biostat-823 "everything" GPU container](https://gitlab.oit.duke.edu/owzar001/bios-823-container-gpu/-/blob/main/README.md) (Singularity)
\ No newline at end of file
diff --git a/pub/Lesson-02.html b/pub/Lesson-Contain.html
similarity index 87%
rename from pub/Lesson-02.html
rename to pub/Lesson-Contain.html
index 5fdfd1f..9df4aa3 100644
--- a/pub/Lesson-02.html
+++ b/pub/Lesson-Contain.html
@@ -1,23 +1,23 @@
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
-
+
Biostat 823 - Containerization
-
-
+
+
-
-
-
-
-
+
+
+
+
+