Merge branch 'master' into 3595-nf-cli

Signed-off-by: Ben Sherman <[email protected]>

.github/workflows/build.yml

@@ -92,7 +92,7 @@ jobs:
       any_changed: ${{ steps.changed-files.outputs.any_changed }}
 
   test:
-    if: ${{ !contains(github.event.head_commit.message, '[ci fast]') }}
+    if: ${{ !contains(github.event.head_commit.message, '[ci fast]') && needs.build.outputs.any_changed == 'true' }}
     needs: build
     runs-on: ubuntu-latest
     strategy:

VERSION

@@ -1 +1 @@
-23.06.0-edge
+23.09.1-edge

build.gradle

@@ -82,6 +82,8 @@ allprojects {
         mavenCentral()
         maven { url 'https://repo.eclipse.org/content/groups/releases' }
         maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
+        maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
+        maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }
     }
 
     configurations {
@@ -100,8 +102,8 @@ allprojects {
 
         // Documentation required libraries
         groovyDoc 'org.fusesource.jansi:jansi:1.11'
-        groovyDoc "org.codehaus.groovy:groovy-groovydoc:3.0.18"
-        groovyDoc "org.codehaus.groovy:groovy-ant:3.0.18"
+        groovyDoc "org.codehaus.groovy:groovy-groovydoc:3.0.19"
+        groovyDoc "org.codehaus.groovy:groovy-ant:3.0.19"
     }
 
     test {

changelog.txt

@@ -1,5 +1,164 @@
 NEXTFLOW CHANGE-LOG
 ===================
+23.09.1-edge - 11 Sep 2023
+- Revert "Allow setting shell directive when using the trace file (#4210)" [9f9edcdc]
+
+23.09.0-edge - 10 Sep 2023
+- Add check for latest version (#4194) [3e8cd488]
+- Add inspect command (#4069) [090c31ce]
+- Add maxSubmitAwait (#3736) [5686bf1d]
+- Add scripts logging for troubleshooting [c056a74e]
+- Add support for Spack to Singularity builds [23c4ec1d]
+- Add support for Wave native build for Singularity [8a434893]
+- Add support for inputs and outputs arity [42504d3c]
+- Add support for remote debug (#4266) [87e0648a]
+- Add warning about using clusterOptions with process directives (#4248) [a68c1a3e]
+- Add which to dockerfile build [817aa05b]
+- Align Singularity experience with Docker (#4230) [a1e1d3ca] [c1cc60dd]
+- Allow setting shell directive when using the trace file (#4210) [7d6ad624]
+- Always emit publish event for cached task outputs (#4227) [62686ce8]
+- Deprecated Wave report feature [80c5cb27]
+- Disable staging script for remote work dir (#4282) [80f7cd46]
+- Disable version check on CI tests [db79e13f]
+- Docs improvement to clarify the usage of the bin dir (#4121) [2daa4172]
+- Document API differences of process path inputs (#4189) [c37e00bc]
+- Document use of local variables in closures (#4251) [05ff784a]
+- Fix IOException should be thrown when failing to creare Azure directory [b0bdfd79]
+- Fix Parallel execution of Conda corrupts data and packages (#4253) [976c282c]
+- Fix Parse negative CLI params as numbers (#4238) [1ae70d5d]
+- Fix Too long Http connection pool timeout [fa961e7f]
+- Fix Wave build for Singularity files [a60ef72b]
+- Fix Wave build when Conda package name is quoted [d19cb0b7]
+- Fix failing test [2785ffe9]
+- Fix fs cp command with remote file [366eedec]
+- Fix printf command with negative exit code (#4213) [465468b0]
+- Fix security deps in nf-azure plugin [c30d5211]
+- Fix setting `executor.queueSize = 0` is ignored (#4228) [6664b578]
+- Improve Wave handing of Conda envs [736ab9bb]
+- Improve pod options documentation (#4274) [c3aa26e1]
+- Make TraceRecord@store field public accessible [07582f0b]
+- Remove -dsl1 and -dsl1 rub cli options [b6721b71]
+- Remove experimental -dockerize option /2 [7def5542]
+- Remove experimental -dockerize option [937c8fb7]
+- Report an error on duplicate workflow name definitions (#4088) [fce9702e]
+- Undocument internal NXF_DEBUG variable [7955db8d]
+- Update AZ File share doc (#4235) [69d317b6]
+- Update docs about splitCsv() operator (#4163) [1dfb621c]
+- Update documentation of NXF_DEBUG (#4187) [a88a4245]
+- Use sipHash24 in place of deprecated murmur32 for script aliasing [bb96763f]
+- Bump groovy 3.0.19 [cb411208]
+- Bump [email protected] [7e5d414e]
+- Bump [email protected] [83410f39]
+- Bump [email protected] [a9f6dd65]
+
+23.08.1-edge - 17 Aug 2023
+- Add 429 http status code to Wave retriable errors [8eb5f305]
+- Add resource labels support for Azure Batch (#4178) [7b5e50a1]
+- Apply K8s Pod metadata to Job (#4057) [4d918627]
+- Document error about trailing backslash with space (#4180) [245afa5d]
+- Enable cloud cache based on environment variable (#4160) [a66b0e63]
+- Escape semicolons in paths (#4193) [552501ce]
+- FIx nested InvocationTargetException (#4192) [67980f19]
+- Fix Execution should fail if report or timeline file already exists [b238d7e2]
+- Fix Process hangs when using flatten and finish errorStrategy [d99b3432]
+- Fix `workflow.container` map resolution (#4190) [96ab8a69]
+- Fix checkpoint thread termination (#4166) [2b449daa]
+- Fix env output when changing task workdir [8e4d7fed]
+- Fix if-guard on log.trace in trask processor inner class [50f6f6d5]
+- Fix typos in source code comments (#4173) [e78bc37e]
+- Improve Conda build error report [7b19fb03]
+- Improve handling of name-only container env variables [3051cd13]
+- Minor changes [7e58c945]
+- Remove dockerize launcher classpath file (#4191) [2bae5198]
+- Remove lock file from cloudcache (#4167) [6e6ea579]
+- Update AWS instructions for creating a custom AMI (#4174) [563bff13]
+- Update changelog [98f88a50]
+- Update tip about modifying maps (#4153) [30036dbf]
+- Use root user in Wave container based on micromamba (#4038) [a3a75ea2]
+- Bump [email protected] [6670bb06]
+- Bump [email protected] [fb8f6681]
+- Bump [email protected] [d06b8365]
+- Bump [email protected] [7555b17d]
+
+23.04.3 - 11 Aug 2023
+- Increase Wave client max attempts [8c67610a]
+- Fix log typo [03e19ea2]
+- Add 429 http status code to Wave retriable errors [a8b8c6c5]
+- Improve handling Wave server errors [621c9665]
+- Bump [email protected] [d7fa3f26]
+- Bump corretto 17.0.8 [7a73a78f]
+
+23.08.0-edge - 5 Aug 2023 
+- Add `-value` option to `config` command (#4142) [57e3100b]
+- Add `deleteTasksOnCompletion` to Azure Batch configuration (#4114) [b14674dc]
+- Add Tower logs checkpoint (#4132) [71dfecc2]
+- Allow use virtual threads in Wave client [dd32f80a]
+- Allow workflow entry from module import (#4128) [51f5c842]
+- Disable cache backup/restore if cloudcache is used (#4125) [46e828e1]
+- Document behavior of withName selector with included aliases (#4129) [8b7e3d48]
+- Fix Option fixOwnership traverse parent directories [f2a2ea35]
+- Fix Redirection http redirection across different hosts [fcdeec02]
+- Fix Wave disable flag [8579e7a4]
+- Fix bug with K8s resource labels (#4147) eu-west-1[3f4b8557]
+- Fix glob resolution for remove files [19a72c40]
+- Fix incorrect error message on missing comma (#4085) eu-west-1[a59af39f]
+- Fix missing changelog for version 23.07.0-edge eu-west-1[9a33e936]
+- Fix strict mode docs (#4150) [6b46b507]
+- Improve plugin docs (#3957) [22638d46]
+- Improve Wave config logging [547fad62]
+- Improve TaskPollingMonitor logging [077ed5dd]
+- Improve Wave and Fusion docs (#4149) [d2229bde]
+- Increase Wave client max attempts [fe5dd497]
+- Remove module all components import [a6d08c04]
+- Restore Tower CacheManager for backward compatibility [6d269070]
+- Bump amazoncorretto:17.0.8 [00eb145c]
+- Bump [email protected] [78e4b278]
+- Bump [email protected] [41c8c164]
+- Bump [email protected] [5f33ac17]
+- Bump [email protected] [981315ad]
+
+23.07.0-edge - 23 Jul 2023
+- Add CPU model name to trace files and traceRecord (#3946) [e0d91bf7]
+- Add ability to disable CLI params type detection [9a1c584d]
+- Add cloudcache plugin (#4097) [ac90cc26]
+- Add missing header to Wave container await [d39866e6]
+- Add remote bin support for TES in a workdir (#3990) [8a22168a]
+- Add retry logic to wave image await [9fc1d3bd]
+- Add rule to build Nextflow docker image for ARM (#4020) [705d55f5]
+- Add support for AWS SSO credentials provider (#4045) [53e33cde]
+- Add support for Wave container freeze [9a5903e6]
+- Add support legacy Wave retry [73a1e7d4]
+- Allow SLURM executor option `--mem-per-cpu` (#4023) [96c04e3b]
+- Allow disabling the Wave requirement when Fusion is enabled [9180d633]
+- Disable Singularity and Apptainer home mount by default (#4056) [a0ee4657]
+- Document `NXF_WRAPPER_STAGE_FILE_THRESHOLD` environment variable (#4113) [bda47567]
+- Fix AzFileSystem retry policy [ba9b6d18] [c2f3cc96]
+- Fix Improve error message for invalid Azure URI [0f4d8867]
+- Fix Treat HTTP headers as case insensitive (#4116) [97fd3358]
+- Fix invalid detection of hierarchical namespace stub blobs as files (#4046) [ce06c877]
+- Fix stage script in Fusion script launcher (#4109) [0933f47e]
+- Ignore accelerator type for AWS Batch (#4043) [263ecca8]
+- Implement Weblog feature as an external plugin [f9f2c338]
+- Improve "Unexpected input: '{'" error message (#4122) [ef9d3cf0]
+- Improve Azure retry logging [de58697a]
+- Improve description of channels and channel types (#4120) [8975734d]
+- Improve handling Wave server errors [84f7a61a]
+- Increase Azure min retry delay to 250ms [2e77e5e4]
+- Remove default arch from wave request [f0e5c0c1]
+- Remove logging of report JSON data (#4098) [099e5039]
+- Wait for all child processes in nxf_parallel (#4050) [60a5f1a7]
+- Bump Groovy 3.0.18 [207eb535]
+- Bump micromamba 1.4.9 [6307f9b5]
+- Bump [email protected] [57464746]
+- Bump [email protected] [e1512f6a]
+- Bump [email protected] [27e7f663]
+- Bump [email protected] [1895efc4]
+- Bump [email protected] [cb6242c4]
+- Bump [email protected] [f98feb77]
+- Bump [email protected] [6aede7c0]
+- Bump [email protected] [3278d798]
+- Bump [email protected] [2998db5d]
+
 23.06.0-edge - 14 Jun 2023
 - Add AWS Kms integration test [19449bf4]
 - Add Wave containers reports (preview) [9d9e2758]
@@ -26,18 +185,28 @@ NEXTFLOW CHANGE-LOG
 - Fix packing all including Wave [735b8d9c]
 - Fix static compiler errors [f48a473c]
 - Improve documentation on Nextflow scripts (#3953) [80050c03]
+- Improve description of channels and channel types (#4120) [8975734d]
+- Improve "Unexpected input: '{'" error message (#4122) [ef9d3cf0]
 - Minor change in Wave config [4da0442a]
 - Prevent null exit code when Google batch is unable to acces exit status [f68a39ec]
 - Refactor Conda and Spack support for Wave to Java [36b9e226]
 - Remove `--no-home` default option from charliecloud builder (#3956) [915074a4]
 - Update workflow.revision max length to match the one in Tower (#4010) [1433a903]
 - Bump amazocorretto:17.0.7 [c8aa1214]
 - Bump azure-storage-blob:12.22.1 [2a36fa77]
-- Bump [email protected] [0872fba5]
-- Bump [email protected] [64bf3115]
-- Bump [email protected] [7ca7d808]
-- Bump [email protected] [afd368c4]
-- Bump [email protected] [58c332a1]
+- Bump [email protected] [2998db5d]
+- Bump [email protected] [3278d798]
+- Bump [email protected] [6aede7c0]
+- Bump [email protected] [f98feb77]
+- Bump [email protected] [1895efc4]
+- Bump [email protected] [57464746]
+- Bump [email protected] [cb6242c4]
+
+23.04.2 - 8 Jun 2023
+- Fix non-deterministic null container engine error [f93221ab]
+- Add retry policy to Wave client [2f1532f6]
+- Fix wave build docs [34a73022]
+- Bump [email protected] [350201b5]
 
 23.05.0-edge - 15 May 2023
 - Add support for custom custom root directory to resolve relative paths (#3942) [f06bb1f7]
@@ -1286,7 +1455,7 @@ NEXTFLOW CHANGE-LOG
 - see 20.07.0 and 20.07.0-RC1
 
 20.07.0 - (skipped)
-- Allow unqualified stdin/stdout defintions with DSL2 [bcdcaab6]
+- Allow unqualified stdin/stdout definitions with DSL2 [bcdcaab6]
 
 20.07.0-RC1 - 21 Jul 2020 
 - Add Dsl2 enable flag [08238109]

docker/Dockerfile

@@ -1,5 +1,5 @@
-FROM amazoncorretto:17.0.7
-RUN yum install -y procps-ng shadow-utils
+FROM amazoncorretto:17.0.8
+RUN yum install -y procps-ng shadow-utils which
 
 ENV NXF_HOME=/.nextflow
 ARG TARGETPLATFORM=linux/amd64
@@ -11,7 +11,6 @@ COPY nextflow /usr/local/bin/nextflow
 
 # download runtime
 RUN mkdir /.nextflow \
- && touch /.nextflow/dockerized \
  && chmod 755 /usr/local/bin/nextflow \
  && chmod 755 /usr/local/bin/entry.sh \
  && nextflow info

docs/aws.md

@@ -260,10 +260,11 @@ There are several reasons why you might need to create your own [AMI (Amazon Mac
 
 ### Create your custom AMI
 
-From the EC2 Dashboard, select **Launch Instance**, then select **AWS Marketplace** in the left-hand pane and search for "ECS". In the result list, select **Amazon ECS-Optimized Amazon Linux 2 AMI**, then continue as usual to configure and launch the instance.
+From the EC2 Dashboard, select **Launch Instance**, then select **Browse more AMIs**. In the new page, select 
+**AWS Marketplace AMIs**, and then search for **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI**. Select the AMI and continue as usual to configure and launch the instance.
 
 :::{note}
-The selected instance has a bootstrap volume of 8GB and a second EBS volume of 30GB for scratch storage, which is not enough for real genomic workloads. Make sure to specify an additional volume with enough storage for your pipeline execution.
+The selected instance has a root volume of 30GB. Make sure to increase its size or add a second EBS volume with enough storage for real genomic workloads.
 :::
 
 When the instance is running, SSH into it (or connect with the Session Manager service), install the AWS CLI, and install any other tool that may be required (see following sections).
@@ -303,7 +304,7 @@ Afterwards, verify that the AWS CLI package works correctly:
 
 ```console
 $ ./miniconda/bin/aws --version
-aws-cli/1.19.79 Python/3.8.5 Linux/4.14.231-173.361.amzn2.x86_64 botocore/1.20.79
+aws-cli/1.29.20 Python/3.11.4 Linux/4.14.318-241.531.amzn2.x86_64 botocore/1.31.20
 ```
 
 :::{note}
@@ -328,7 +329,7 @@ The grandparent directory of the `aws` tool will be mounted into the container a
 
 ### Docker installation
 
-Docker is required by Nextflow to execute tasks on AWS Batch. The **Amazon ECS-Optimized Amazon Linux 2** AMI has Docker installed, however, if you create your AMI from a different AMI that does not have Docker installed, you will need to install it manually.
+Docker is required by Nextflow to execute tasks on AWS Batch. The **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI** has Docker installed, however, if you create your AMI from a different AMI that does not have Docker installed, you will need to install it manually.
 
 The following snippet shows how to install Docker on an Amazon EC2 instance:
 
@@ -353,7 +354,7 @@ These steps must be done *before* creating the AMI from the current EC2 instance
 
 The [ECS container agent](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_agent.html) is a component of Amazon Elastic Container Service (Amazon ECS) and is responsible for managing containers on behalf of ECS. AWS Batch uses ECS to execute containerized jobs, therefore it requires the agent to be installed on EC2 instances within your Compute Environments.
 
-The ECS agent is included in the **Amazon ECS-Optimized Amazon Linux 2** AMI. If you use a different AMI, you can also install the agent on any EC2 instance that supports the Amazon ECS specification.
+The ECS agent is included in the **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI** . If you use a different base AMI, you can also install the agent on any EC2 instance that supports the Amazon ECS specification.
 
 To install the agent, follow these steps:
 
@@ -373,6 +374,10 @@ curl -s http://localhost:51678/v1/metadata | python -mjson.tool (test)
 The `AmazonEC2ContainerServiceforEC2Role` policy must be attached to the instance role in order to be able to connect the EC2 instance created by the Compute Environment to the ECS container.
 :::
 
+:::{note}
+The `AmazonEC2ContainerRegistryReadOnly` policy should be attached to the instance role in order to get read-only access to Amazon EC2 Container Registry repositories.
+:::
+
 ## Jobs & Execution
 
 ### Custom job definition

docs/azure.md

@@ -63,7 +63,7 @@ azure {
 }
 ```
 
-The files in the File share are available to the task in the directory: `<YOUR MOUNT DESTINATION>/<YOUR SOURCE FILE SHARE NAME>`.
+The files in the File share are available to the task in the directory: `<YOUR MOUNT DESTINATION>`.
 
 For instance, given the following configuration:
 
@@ -73,15 +73,20 @@ azure {
         // ...
 
         fileShares {
-            dir1 {
-                mountPath = "/mnt/mydata/"
+          rnaseqResources {
+                mountPath = "/mnt/mydata/myresources"
             }
         }
     }
 }
 ```
 
-The task can access the File share in `/mnt/mydata/dir1`.
+The task can access the File share in `/mnt/mydata/myresources`. Note: The string `rnaseqResources` in the above config can be any name of your choice, and it does not affect the underlying mount. 
+
+:::{warning}
+Azure File shares do not support authentication and management with Active Directory. The storage account key must be 
+set in the configuration if a share is mounted.
+:::
 
 (azure-batch)=
 

docs/channel.md

@@ -2,12 +2,12 @@
 
 # Channels
 
-Nextflow is based on the Dataflow programming model in which processes communicate through channels.
+Nextflow is based on the dataflow programming model in which processes communicate through channels.
 
 A channel has two major properties:
 
-1. Sending a message is an *asynchronous* operation which completes immediately, without having to wait for the receiving process.
-2. Receiving data is a blocking operation which stops the receiving process until the message has arrived.
+1. Sending a message is an *asynchronous* (i.e. non-blocking) operation, which means the sender doesn't have to wait for the receiving process.
+2. Receiving a message is a *synchronous* (i.e. blocking) operation, which means the receiving process must wait until a message has arrived.
 
 (channel-types)=
 
@@ -27,12 +27,10 @@ A queue channel can be created by factory methods ([of](#of), [fromPath](#frompa
 
 ### Value channel
 
-A *value channel* a.k.a. *singleton channel* is bound to a single value and can be read any number of times without being consumed.
+A *value channel* contains a single value and can be consumed any number of times by a process or operator.
 
 A value channel can be created with the [value](#value) factory method or by any operator that produces a single value ({ref}`operator-first`, {ref}`operator-collect`, {ref}`operator-reduce`, etc). Additionally, a process will emit value channels if it is invoked with all value channels, including simple values which are implicitly wrapped in a value channel.
 
-A value channel is implicitly created by a process when it is invoked with a simple value. Furthermore, a value channel is also implicitly created as output for a process whose inputs are all value channels.
-
 For example:
 
 ```groovy
@@ -54,7 +52,7 @@ workflow {
 }
 ```
 
-In the above example, since the `foo` process is invoked with a simple value instead of a channel, the input is implicitly converted to a value channel, and the output is also emitted as a value channel.
+In the above example, since the `foo` process is invoked with a simple value instead of a channel, the input is implicitly wrapped in a value channel, and the output is also emitted as a value channel.
 
 See also: {ref}`process-multiple-input-channels`.