Skip to content

Commit

Permalink
Revise content
Browse files Browse the repository at this point in the history
  • Loading branch information
jason-seqera committed Jul 12, 2024
1 parent 29752d4 commit 5c68c47
Show file tree
Hide file tree
Showing 15 changed files with 122 additions and 125 deletions.
26 changes: 16 additions & 10 deletions docs/cli/install.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,24 @@ title: Install the Wave CLI

To install the `wave` CLI for your platform, complete the following steps:

1. Download the [latest version of the Wave CLI][download] for your platform.
1. Download latest version of the Wave CLI for your platform:

2. In a new terminal, complete the following steps:
- To install the latest release from GitHub:

1. Move the executable from your downloads folder to a location in your `PATH`, such as `~/bin`. For example: `mv wave-cli-0.8.0-macos-x86_64 ~/bin/wave`
2. Ensure that the executable bit is set. For example: `chmod u+x ~/bin/wave`
1. Download the [latest version of the Wave CLI][download] for your platform.

3. You can also use [Homebrew](https://brew.sh/) in macos and linux, you can install like this:
```bash
brew install seqeralabs/tap/wave-cli
```
1. In a new terminal, complete the following steps:

4. Verify that you can build containers with Wave:
1. Move the executable from your downloads folder to a location in your `PATH`, such as `~/bin`. For example: `mv wave-cli-0.8.0-macos-x86_64 ~/bin/wave`
1. Ensure that the executable bit is set. For example: `chmod u+x ~/bin/wave`

- To install the latest version with [Homebrew]:

```bash
brew install seqeralabs/tap/wave-cli
```

1. Verify that you can build containers with Wave:

1. Create a basic `Dockerfile`:

Expand All @@ -26,7 +31,7 @@ To install the `wave` CLI for your platform, complete the following steps:
EOF
```
2. Use the CLI to build the container:
1. Use the CLI to build the container:
```
wave -f Dockerfile
Expand All @@ -39,3 +44,4 @@ To install the `wave` CLI for your platform, complete the following steps:
```
[download]: https://github.com/seqeralabs/wave-cli/releases
[Homebrew]: https://brew.sh/
53 changes: 0 additions & 53 deletions docs/guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,61 +4,8 @@ title: User guide

## Getting started

Wave containers can be used with any container runtime supporting the [Docker Registry API v2](https://docs.docker.com/registry/spec/api/) and it's smoothly integrated with [Nextflow](https://www.nextflow.io/) and [Seqera Platform](https://cloud.tower.nf/).

This feature requires Nextflow `22.10.0` or later.

### Nextflow installation

If you've already installed Nextflow, update to the latest version using this command::

```bash
nextflow -self-update
```

If you don't have Nextflow already installed, install it with the command below:

```bash
curl get.nextflow.io | bash
```

### Wave configuration

Wave can be used in any Nextflow pipeline by adding the following snippet to your `nextflow.config` file:

```groovy
wave {
enabled = true
}
tower {
accessToken = '<your access token>'
}
```

:::tip
The use of the Seqera access token is not mandatory, however, it's required to enable access to private repositories and it allows higher service rate limits compared to anonymous users.
:::

## Known limitation

### Use of sha256 digest in the image name

The Wave does not support the use of sha256 digest in the image name, e.g. `ubuntu@sha256:3235...ce8f`, when using
the augmentation process to extend container images.

In order to reference a container via sha256 digest in the image name with Wave you will need to *freeze* image mode
that will force the creation of a new container image using the container you have specified as base image.

In your pipeline configuration, ensure that you specify the following settings:

```groovy
wave.enabled = true
wave.freeze = true
wave.strategy = ['dockerfile']
wave.build.repository = 'docker.io/<user>/<repository>'
```

## More examples

Check out the [Wave showcase repository](https://github.com/seqeralabs/wave-showcase) for more examples how to use Wave containers.
4 changes: 1 addition & 3 deletions docs/guides/fusion-file-system.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Use Wave with Fusion file system
---

Wave containers allows you to run your containerized workflow with the fusion-page.
Wave containers allows you to run your containerized workflow with the Fusion file system.

This enables the use of an object storage bucket such as AWS S3 or Google Cloud Storage as your pipeline work directory, simplifying and speeding up many operations on local, AWS Batch, Google Batch or Kubernetes executions.

Expand All @@ -11,7 +11,5 @@ For more information, refer to the following documentation pages:
- [Fusion documentation][fusion]
- [Nextflow Fusion integration documentation][nextflow-fusion]

See the `Fusion documentation <fusion-page>` for more details.

[fusion]: https://docs.seqera.io/fusion
[nextflow-fusion]: https://www.nextflow.io/docs/latest/fusion.html
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,6 @@ tower {
}
```

That's it. When launching the pipeline execution, Wave will allow Nextflow to access the private container repositories defined in your pipeline configuration, using the credentials stored in the Seqera Platform credentials manager.

====

Containers built by Wave are uploaded to the Wave default repository hosted on AWS ECR with name `195996028523.dkr.ecr.eu-west-1.amazonaws.com/wave/build`. The images in this repository are automatically deleted 1 week from the date of their push.

If you want to store Wave containers in your own container repository use the following settings in the Nextflow configuration file:
Expand All @@ -28,7 +24,8 @@ wave.build.cacheRepository = 'example.com/your/cache-repo'

The first repository is used to store the built container images. The second one is used to store the individual image layers for caching purposes.

The repository access keys need to be specified using the Seqera Platform credentials manager as specified in the [Authenticate private repositories](#Authenticate private repositories) section.
When launching the pipeline execution, Wave allows Nextflow to access the private container repositories defined in your pipeline configuration, by using the credentials stored in the Seqera Platform credentials manager.


[credentials]: /platform_versioned_docs/version-23.4.0/credentials/overview
[pat]: /platform_versioned_docs/version-23.4.0/api/overview#authentication
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ Moreover the access to the repository must be granted in the compute nodes by us
Please see Singularity documentation for further details.
:::

:::{note}
In order to build Singularity native images, both `singularity.ociAutoPull` and `singularity.ociMode` need to be disabled in the configuration. For more information, see the Nextflow [configuration][[config]] documentation.
:::note
In order to build Singularity native images, both `singularity.ociAutoPull` and `singularity.ociMode` need to be disabled in the configuration. For more information, see the Nextflow [configuration][config] documentation.
:::

[private]: /platform_versioned_docs/version-23.4.0/credentials/overview
Expand Down
File renamed without changes.
39 changes: 39 additions & 0 deletions docs/nextflow/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,45 @@
title: Nextflow configuration for Wave
---

This feature requires Nextflow `22.10.0` or later.

Wave can be used in any Nextflow pipeline by adding the following snippet to your `nextflow.config` file:

```groovy
wave {
enabled = true
}
tower {
accessToken = '<your access token>'
}
```

:::tip
The use of the Seqera access token is not mandatory, however, it's required to enable access to private repositories and it allows higher service rate limits compared to anonymous users.
:::

## Limitations

### Use of sha256 digest in the image name

The Wave does not support the use of sha256 digest in the image name, e.g. `ubuntu@sha256:3235...ce8f`, when using
the augmentation process to extend container images.

In order to reference a container via sha256 digest in the image name with Wave you will need to *freeze* image mode
that will force the creation of a new container image using the container you have specified as base image.

In your pipeline configuration, ensure that you specify the following settings:

```groovy
wave.enabled = true
wave.freeze = true
wave.strategy = ['dockerfile']
wave.build.repository = 'docker.io/<user>/<repository>'
```

## Nextflow configuration

The following configuration options are available:

| Method | Description |
Expand Down
4 changes: 3 additions & 1 deletion docs/service/architecture.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@
title: Architecture
---

Currently this is TBD.
Wave containers can be used with any container runtime supporting the [Docker Registry API v2](https://docs.docker.com/registry/spec/api/) and it's smoothly integrated with [Nextflow](https://www.nextflow.io/) and [Seqera Platform](https://cloud.tower.nf/).

TBD.
33 changes: 16 additions & 17 deletions docs/service/augmentation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,25 @@
title: Container augmentation
---

The container augmentation provisioning mode allows "extending" the content of a container image without rebuilding it. Instead, this mechanism allows modifying a container image during the pull phase made by a Docker client.
The container augmentation provisioning mode allows _extending_ the content of a container image without rebuilding it. Instead, this mechanism modifies a container image during the pull phase made by a Docker-compatible client. Augmented containers are ephemeral: they are not stored in a container repository, and they can only be accessed for a short period of time. The extended content added by Wave is served from the Cloudflare CDN. The augmentation process does perform any _build_ operation behind the scenes.

Container augmentation works as follows:
This approach supports use cases such as the following:

1. The client, either Nextflow or Wave client, submits a container request specifying: i) the (Platform) user identity; ii) the container image to be augmented; iii) the container extension configuration, which can be either a custom payload, one or more extension layers or container images.
1. The Wave service validates the request and authorizes the user submitting a request to the Platform service
1. Finally, the Wave service responds with an ephemeral container image name e.g. wave.seqera.io/wt/<ID TOKEN>/library/alpine:latest
The ID TOKEN is uniquely assigned and it's used to identify and authorize the following container request.
1. The Docker client uses the return image name to pull the container binary content of the upstream image directly from the target registry, finally, the content added by Wave as one or more layer extensions is shipped by the Wave service
- Authenticate access to your private repositories with Seqera Platform credentials
- Extend existing containers by adding infrastructure and pipeline dependencies on the fly, without rebuilding and maintaining additional container images

*Key points*
Container augmentation works as follows:

- Wave acts as a proxy between the Docker client and the target registry that hosts the container image.
- During this process, Wave modifies, if needed, the container manifest to add the new content as specified by the request, but it does not (and cannot) alter the container layer blob files that have a unique checksum that is preserved.
- The image blobs are downloaded directly from the target registry (not from Wave) **[there's an exception to be discussed]**
- The extended content added by Wave is served via Cloudflare CDN.
- This process does not carry out any "build" operation behind the scenes.
- Augmented containers are ephemeral: they are not stored in a container repository, and they can only be accessed for a short period of time.
1. The client, either Nextflow or Wave client, submits a container request specifying:
1. The (Platform) user identity
1. The container image to be augmented
1. The container extension configuration, which can be either a custom payload, one or more extension layers or container images.
1. The Wave service validates the request and authorizes the user submitting a request to the Platform service.
1. The Wave service responds with an ephemeral container image name e.g. `wave.seqera.io/wt/<id_token>/library/alpine:latest`. The `<id token>` is uniquely assigned and is used to identify and authorize the following container request.
1. The Docker client uses the return image name to pull the container binary content of the upstream image directly from the target registry, finally, the content added by Wave as one or more layer extensions is shipped by the Wave service.

*Use cases*
Notable parts of this workflow include:

- Authenticate access to the private repositories via Platform credentials
- Extending existing containers by adding infrastructure and pipeline dependencies on the fly without rebuilding and maintaining additional container images
- Wave acts as a proxy between the Docker client and the target registry that hosts the container image.
- Wave modifies, if needed, the container manifest to add the new content as specified by the request, but it does not (and cannot) alter the container layer blob files that have a unique checksum, which is preserved.
- Image blobs are downloaded directly from the target registry, not from Wave. ( **[there's an exception to be discussed]**)
7 changes: 4 additions & 3 deletions docs/service/features.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
title: Features
---

||||||||
|--- |--- |--- |--- |--- |--- |--- |
Wave provides the following container builds

||Provisioning mode|Source|Freeze|Build repo|Accessibility|Format|
|--- |--- |--- |--- |--- |--- |--- |
|Ephemeral|Augmentation|Container image|No|n/a|Temporary token|Docker|
|Ephemeral|Build|Container file|No|Default|Temporary token|Docker|
|Ephemeral|Build|Conda package|No|Default|Temporary token|Docker|
Expand Down Expand Up @@ -36,7 +37,7 @@ Imagine you have a base Ubuntu image in a container registry. Wave acts as a pro
2. Custom layer injection: Wave injects your custom layer, denoted by "ω", which could represent application code, libraries, configurations etc.
3. New image creation: Wave combines the downloaded Ubuntu layers with your custom layer, effectively creating a new image on the fly.

![](_images/wave_container_augmentation.png)
![](../_images/wave_container_augmentation.png)

### Benefits of Wave augmentation

Expand Down
45 changes: 26 additions & 19 deletions docs/service/freeze.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,34 @@
title: Container freeze
---

The container freeze mode allows the provisioning of non-ephemeral containers that can be stored permanently in a container registry of your choice. When using the freeze mode, the Wave service transparently carries out a regular build.
The container _freeze mode_ allows the provisioning of non-ephemeral containers that are stored permanently in a container registry of your choice. When using the freeze mode, the Wave service transparently carries out a regular container build.

Freeze Mode Description:
This approach supports use cases such as the following:

1. The client, either Nextflow or Wave client, submits a container request specifying: i) the (Platform) user identity; ii) the container image to augment; iii) the container extension configuration, which can be either a custom payload, one or more extension layers or container images; iv) the target repository where the built container should be uploaded.
1. The Wave service validates the request and authorizes the user via a request to the Platform service.
1. The Wave service checks if the container image already exists in the target registry
1. If the image does not exist, Wave launches a container build job and pushes the resulting image to the target registry
1. The Wave service responds with the container image name e.g. your.registry.com/some/image/build:1234567
- Create container images on-demand from Conda packages
- Deliver multi-architecture (AMD64 and ARM64) and multi-format (Docker and Singularity) container collections
- Deliver container images in the same region where compute is performed

*Key points*
Wave freeze mode works as follows:

- Container images provisioned via the freeze mode are regular container builds.
- Each container image is associated with a unique ID that is obtained by hashing i) the Container file, ii) any package dependencies, iii) the target platform i.e. amd64 or arm64, iv) the target repository name.
1. The client, either Nextflow or Wave client, submits a container request specifying:
1. The (Seqera Platform) user identity
1. The container image to augment
1. The container extension configuration, which can be either a custom payload, one or more extension layers or container images
1. The target repository where the built container should be uploaded
1. The Wave service validates the request and authorizes the user via a request to the Platform service.
1. The Wave service checks if the container image already exists in the target registry.
1. If the image does not exist, Wave launches a container build job and pushes the resulting image to the target registry.
1. The Wave service responds with the container image name e.g. `example.com/some/image/build:1234567`.

Notable parts of this workflow include:

- Container images provisioned with freeze mode are regular container builds.
- Each container image is associated with a unique ID that is obtained by hashing the following elements:
- The Container file
- Any package dependencies
- The target platform, which is either AMD64 or ARM64
- The target repository name
- When a request for the same container is made, the same ID is assigned to it and therefore, the build is skipped.
- The resulting images are hosted in the customer repository and not cached locally (provided that a cache repository is specified)
- The container images are stored permanently as long as the repository owner does not delete it

*Use cases*

- Create container images on-demand via Conda packages
- Deliver multi-architecture (amd64, arm64) and multi-format (Docker, Singularity) container collections.
- Deliver container images in the same region where compute is performed

- The resulting images are hosted in your selected repository and not cached locally, unless a cache repository is specified.
- The container images are stored permanently unless the repository owner deletes them.
8 changes: 4 additions & 4 deletions docs/service/singularity-containers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
title: Singularity containers
---

Singularity and Apptainer use a proprietary format called Singularity Image Format (SIF).
Wave does can provision containers based on the Singularity image format either using a `Singularityfile` or Conda package(s). The resulting Singularity image file is stored as an ORAS artefact in an OCI-compliant container registry of your choice or the Wave Community registry.
Singularity and Apptainer use a proprietary format called _Singularity Image Format_ (SIF). The Wave service can provision containers based on the Singularity image format either by using a `Singularityfile` file or Conda packages. The resulting Singularity image file is stored as an ORAS artefact in an OCI-compliant container registry of your choice or the Wave Community registry.

The advantage of this approach is that Singularity and Apptainer engines can pull and execute those container images natively without requiring extra conversion steps, as needed when using Docker images with those two engines.

Note: considering the Singularity image format's peculiarities, Wave's freeze mode is mandatory when provisioning Singulairy images.

:::note
Considering the Singularity image format's peculiarities, Wave's freeze mode is mandatory when provisioning Singularity images.
:::
Loading

0 comments on commit 5c68c47

Please sign in to comment.