Merge branch 'master' into docs-standard-library

Signed-off-by: Ben Sherman <[email protected]>

VERSION

@@ -1 +1 @@
-24.09.2-edge
+24.10.0

changelog.txt

@@ -1,5 +1,27 @@
 NEXTFLOW CHANGE-LOG
 ===================
+24.10.0 - 27 Oct 2024
+- Add `manifest.contributors` config option (#5322) [cf0f9690]
+- Add wave mirror and scan config [92e69776]
+- Add wave mirror vs module bundles conflicst warning [b37a8a5b]
+- Add Workflow output definition (second preview) (#5185) [ea128463]
+- Demote azure batch task status log level to trace (#5416) [d6c684bb]
+- Document formal script syntax (#5336) [a48ac589]
+- Add Expose trace record meta info in the task context (#5402) [ffdd3819]
+- Fix NextflowMeta compilation error [500f49c0]
+- Fix XFileSystem serialization issue (#5411) [711864fd]
+- Fix closure rendering in yaml and json config (#5408) [d3a85ceb]
+- Fix failing docs snippet (#5430) [a818f4b8]
+- Fix http files stream  (#5405) [e2e stage] [718dcbe6]
+- Fix nf-tower plugin to upload logs when early failures  (#5434) [4222442a]
+- Fix support for micromamba (#4302) [12431302]
+- Harmonise documentation for hybrid cloud execution (#5362) [a69407d2]
+- Prevent NPE when containerInfo is null [368a266a]
+- Remove invalid equals char (#5414) [7f39e6ed]
+- Bump [email protected] [f67e157a]
+- Bump [email protected] [7a572a12]
+- Bump [email protected] [6524d8dc]
+
 24.09.2-edge - 14 Oct 2024
 - Add Google LS deprecation notice (#5400) [0ee1d9bc]
 - Add hybrid workflow documentation for Azure (#5361) [f81f27de]

docs/amazons3.md

@@ -16,7 +16,7 @@ s3://my-bucket/data/sequences.fa
 
 The usual file operations can be applied to a path handle with the above notation. For example, the content of an S3 file can be printed as follows:
 
-```groovy
+```nextflow
 println file('s3://my-bucket/data/sequences.fa').text
 ```
 

docs/aws.md

@@ -252,7 +252,7 @@ Container options may be passed in long form (e.g `--option value`) or short for
 
 Few examples:
 
-```groovy
+```nextflow
 containerOptions '--tmpfs /run:rw,noexec,nosuid,size=128 --tmpfs /app:ro,size=64'
 
 containerOptions '-e MYVAR1 --env MYVAR2=foo2 --env MYVAR3=foo3 --memory-swap 3240000 --memory-swappiness 20 --shm-size 16000000'

docs/azure.md

@@ -163,7 +163,7 @@ By default, the `cpus` and `memory` directives are used to find the smallest mac
 
 To specify multiple Azure machine families, use a comma separated list with glob (`*`) values in the `machineType` directive. For example, the following will select any machine size from D or E v5 machines, with additional data disk, denoted by the `d` suffix:
 
-```config
+```groovy
 process.machineType = "Standard_D*d_v5,Standard_E*d_v5"
 ```
 

docs/cache-and-resume.md

@@ -112,9 +112,9 @@ While Nextflow tries to make it easy to write safe concurrent code, it is still
 
 Consider the following example:
 
-```groovy
-Channel.of(1,2,3) | map { it -> X=it; X+=2 } | view { "ch1 = $it" }
-Channel.of(1,2,3) | map { it -> X=it; X*=2 } | view { "ch2 = $it" }
+```nextflow
+Channel.of(1,2,3) | map { v -> X=v; X+=2 } | view { v -> "ch1 = $v" }
+Channel.of(1,2,3) | map { v -> X=v; X*=2 } | view { v -> "ch2 = $v" }
 ```
 
 The problem here is that `X` is declared in each `map` closure without the `def` keyword (or other type qualifier). Using the `def` keyword makes the variable local to the enclosing scope; omitting the `def` keyword makes the variable global to the entire script.
@@ -123,12 +123,12 @@ Because `X` is global, and operators are executed concurrently, there is a *race
 
 The solution is to not use a global variable where a local variable is enough (or in this simple example, avoid the variable altogether):
 
-```groovy
+```nextflow
 // local variable
-Channel.of(1,2,3) | map { it -> def X=it; X+=2 } | view { "ch1 = $it" }
+Channel.of(1,2,3) | map { v -> def X=v; X+=2 } | view { v -> "ch1 = $v" }
 
 // no variable
-Channel.of(1,2,3) | map { it -> it * 2 } | view { "ch2 = $it" }
+Channel.of(1,2,3) | map { v -> v * 2 } | view { v -> "ch2 = $v" }
 ```
 
 (cache-nondeterministic-inputs)=
@@ -137,7 +137,7 @@ Channel.of(1,2,3) | map { it -> it * 2 } | view { "ch2 = $it" }
 
 Sometimes a process needs to merge inputs from different sources. Consider the following example:
 
-```groovy
+```nextflow
 workflow {
     ch_foo = Channel.of( ['1', '1.foo'], ['2', '2.foo'] )
     ch_bar = Channel.of( ['2', '2.bar'], ['1', '1.bar'] )
@@ -158,7 +158,7 @@ It is tempting to assume that the process inputs will be matched by `id` like th
 
 The solution is to explicitly join the two channels before the process invocation:
 
-```groovy
+```nextflow
 workflow {
     ch_foo = Channel.of( ['1', '1.foo'], ['2', '2.foo'] )
     ch_bar = Channel.of( ['2', '2.bar'], ['1', '1.bar'] )

docs/channel.md

@@ -37,7 +37,7 @@ channels if it is invoked with all value channels, including simple values which
 
 For example:
 
-```groovy
+```nextflow
 process foo {
   input:
   val x
@@ -67,7 +67,7 @@ Channel factories are functions that can create channels.
 
 For example, the `Channel.of()` factory can be used to create a channel from an arbitrary list of arguments:
 
-```groovy
+```nextflow
 Channel.of(1, 2, 3).view()
 ```
 

docs/conda.md

@@ -45,7 +45,7 @@ Alternatively, it can be specified by setting the variable `NXF_CONDA_ENABLED=tr
 
 Conda package names can specified using the `conda` directive. Multiple package names can be specified by separating them with a blank space. For example:
 
-```groovy
+```nextflow
 process foo {
   conda 'bwa samtools multiqc'
 
@@ -94,7 +94,7 @@ Read the Conda documentation for more details about how to create [environment f
 
 The path of an environment file can be specified using the `conda` directive:
 
-```groovy
+```nextflow
 process foo {
   conda '/some/path/my-env.yaml'
 
@@ -124,7 +124,7 @@ Like before, the extension matters. Make sure the dependencies file has a `.txt`
 
 If you already have a local Conda environment, you can use it in your workflow specifying the installation directory of such environment by using the `conda` directive:
 
-```groovy
+```nextflow
 process foo {
   conda '/path/to/an/existing/env/directory'
 

docs/conf.py

@@ -336,3 +336,77 @@
 
 # Allow duplicate toc entries.
 #epub_tocdup = True
+
+# see also: https://github.com/pygments/pygments/blob/master/pygments/lexers/jvm.py
+from pygments.lexer import RegexLexer
+from sphinx.highlighting import lexers
+
+class NextflowLexer(RegexLexer):
+    """
+    For Nextflow source code.
+    """
+
+    import re
+    from pygments.lexer import bygroups, using, this, default
+    from pygments.token import Comment, Operator, Keyword, Name, String, Number, Whitespace
+    from pygments.util import shebang_matches
+
+    name = 'Nextflow'
+    url = 'https://nextflow.io/'
+    aliases = ['nextflow', 'nf']
+    filenames = ['*.nf']
+    mimetypes = ['text/x-nextflow']
+    # version_added = '1.5'
+
+    flags = re.MULTILINE | re.DOTALL
+
+    tokens = {
+        'root': [
+            # Nextflow allows a file to start with a shebang
+            (r'#!(.*?)$', Comment.Preproc, 'base'),
+            default('base'),
+        ],
+        'base': [
+            (r'[^\S\n]+', Whitespace),
+            (r'(//.*?)(\n)', bygroups(Comment.Single, Whitespace)),
+            (r'/\*.*?\*/', Comment.Multiline),
+            # keywords: go before method names to avoid lexing "throw new XYZ"
+            # as a method signature
+            (r'(assert|catch|else|'
+             r'if|instanceof|new|return|throw|try|in|as)\b',
+             Keyword),
+            # method names
+            (r'^(\s*(?:[a-zA-Z_][\w.\[\]]*\s+)+?)'  # return arguments
+             r'('
+             r'[a-zA-Z_]\w*'                        # method name
+             r'|"(?:\\\\|\\[^\\]|[^"\\])*"'         # or double-quoted method name
+             r"|'(?:\\\\|\\[^\\]|[^'\\])*'"         # or single-quoted method name
+             r')'
+             r'(\s*)(\()',                          # signature start
+             bygroups(using(this), Name.Function, Whitespace, Operator)),
+            (r'@[a-zA-Z_][\w.]*', Name.Decorator),
+            (r'(def|enum|include|from|output|process|workflow)\b', Keyword.Declaration),
+            (r'(boolean|byte|char|double|float|int|long|short|void)\b',
+             Keyword.Type),
+            (r'(true|false|null)\b', Keyword.Constant),
+            (r'""".*?"""', String.Double),
+            (r"'''.*?'''", String.Single),
+            (r'"(\\\\|\\[^\\]|[^"\\])*"', String.Double),
+            (r"'(\\\\|\\[^\\]|[^'\\])*'", String.Single),
+            (r'/(\\\\|\\[^\\]|[^/\\])*/', String),
+            (r"'\\.'|'[^\\]'|'\\u[0-9a-fA-F]{4}'", String.Char),
+            (r'(\.)([a-zA-Z_]\w*)', bygroups(Operator, Name.Attribute)),
+            (r'[a-zA-Z_]\w*:', Name.Label),
+            (r'[a-zA-Z_$]\w*', Name),
+            (r'[~^*!%&\[\](){}<>|+=:;,./?-]', Operator),
+            (r'[0-9][0-9]*\.[0-9]+([eE][0-9]+)?[fd]?', Number.Float),
+            (r'0x[0-9a-fA-F]+', Number.Hex),
+            (r'[0-9]+L?', Number.Integer),
+            (r'\n', Whitespace)
+        ],
+    }
+
+    def analyse_text(text):
+        return shebang_matches(text, r'nextflow')
+
+lexers['nextflow'] = NextflowLexer()

docs/config.md

@@ -20,6 +20,8 @@ When more than one of these options for specifying configurations are used, they
 You can use the `-C <config-file>` option to use a single configuration file and ignore all other files.
 :::
 
+(config-syntax)=
+
 ## Syntax
 
 The Nextflow configuration syntax is based on the Nextflow script syntax. It is designed for setting configuration options in a declarative manner while also allowing for dynamic expressions where appropriate.

docs/developer/nextflow.ast.md

@@ -25,7 +25,7 @@ You can see the effect of Nextflow's AST transforms by using the Nextflow consol
 
 Here is the example from {ref}`your-first-script`:
 
-```groovy
+```nextflow
 params.str = 'Hello world!'
 
 process splitLetters {

docs/developer/plugins.md

@@ -138,7 +138,7 @@ class MyExecutor extends Executor implements ExtensionPoint {
 
 You can then use this executor in your pipeline:
 
-```groovy
+```nextflow
 process foo {
     executor 'my-executor'
 }
@@ -177,15 +177,15 @@ class MyExtension extends PluginExtensionPoint {
 
 You can then use this function in your pipeline:
 
-```groovy
+```nextflow
 include { reverseString } from 'plugin/my-plugin'
 
 channel.of( reverseString('hi') )
 ```
 
 You can also use an alias:
 
-```groovy
+```nextflow
 include { reverseString as anotherReverseMethod } from 'plugin/my-plugin'
 ```
 
@@ -226,7 +226,7 @@ class MyExtension extends PluginExtensionPoint {
 
 You can then use them in your pipeline:
 
-```groovy
+```nextflow
 include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb'
 
 def sql = 'select * from FOO'

docs/dsl1.md

@@ -6,7 +6,7 @@ In Nextflow version `22.03.0-edge`, DSL2 became the default DSL version. In vers
 
 In Nextflow versions prior to `22.03.0-edge`, you must enable DSL2 explicitly in order to use it. You can either set the feature flag in your pipeline script:
 
-```groovy
+```nextflow
 nextflow.enable.dsl=2
 ```
 
@@ -20,7 +20,7 @@ export NXF_DEFAULT_DSL=2
 
 In DSL1, a process definition is also the process invocation. Process inputs and outputs are connected to channels using `from` and `into`. Here is the {ref}`your-first-script` example written in DSL1:
 
-```groovy
+```nextflow
 nextflow.enable.dsl=1
 
 params.str = 'Hello world!'
@@ -54,7 +54,7 @@ To migrate this code to DSL2, you need to move all of your channel logic through
 Refer to the {ref}`workflow-page` page to learn how to define a workflow. The DSL2 version of the above script is duplicated here for your convenience:
 
 ```{literalinclude} snippets/your-first-script.nf
-:language: groovy
+:language: nextflow
 ```
 
 ## Channel forking
@@ -65,7 +65,7 @@ In DSL2, channels are automatically forked when connecting two or more consumers
 
 For example, this would not work in DSL1 but is not a problem in DSL2:
 
-```groovy
+```nextflow
 Channel
     .from('Hello','Hola','Ciao')
     .set{ cheers }
@@ -105,7 +105,7 @@ DSL2 scripts cannot exceed 64 KB in size. Large DSL1 scripts may need to be spli
 
   For example:
 
-  ```groovy
+  ```nextflow
   process foo {
       input:
       tuple X, 'some-file.sam'
@@ -121,7 +121,7 @@ DSL2 scripts cannot exceed 64 KB in size. Large DSL1 scripts may need to be spli
 
   Use:
 
-  ```groovy
+  ```nextflow
   process foo {
       input:
       tuple val(X), path('some-file.sam')
@@ -164,7 +164,7 @@ An early preview of DSL2 was available in 2020. Note that some of that early DSL
 
   For example:
 
-  ```groovy
+  ```nextflow
   include './some/library'
   include bar from './other/library'
 
@@ -176,7 +176,7 @@ An early preview of DSL2 was available in 2020. Note that some of that early DSL
 
   Should be replaced with:
 
-  ```groovy
+  ```nextflow
   include { foo } from './some/library'
   include { bar } from './other/library'
 

docs/flux.md

@@ -36,9 +36,9 @@ For additional Flux settings, see the {ref}`flux-executor` section.
 
 Here is an example pipeline that we will use:
 
-```groovy
+```nextflow
 workflow {
-    breakfast = Channel.from '🥞️', '🥑️', '🥧️', '🍵️', '🍞️'
+    breakfast = Channel.of '🥞️', '🥑️', '🥧️', '🍵️', '🍞️'
     haveMeal(breakfast)
 }