Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document formal script syntax #5336

Merged
merged 29 commits into from
Oct 23, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9d6c826
Update docs and tests with strict syntax
bentsherman Sep 25, 2024
189b560
Remove some references to Groovy
bentsherman Sep 26, 2024
24bb403
Rename "implicit workflow" -> "entry workflow"
bentsherman Sep 26, 2024
5c0ec1d
Initial syntax reference page
bentsherman Sep 28, 2024
39ee1c9
Revert non-docs changes
bentsherman Sep 28, 2024
00ba5d1
Cleanup
bentsherman Sep 28, 2024
69772bf
Add section on script definitions
bentsherman Sep 28, 2024
24e65e0
Update config syntax docs
bentsherman Sep 28, 2024
e9faf58
Apply suggestions from code review
bentsherman Oct 1, 2024
23e05b7
Update docs/module.md
bentsherman Oct 1, 2024
1a173a0
Apply suggestions from code review
bentsherman Oct 1, 2024
74480fc
Add remaining sections to syntax page
bentsherman Sep 30, 2024
4aee05a
Apply suggestions from code review
bentsherman Oct 1, 2024
e073700
minor edits
bentsherman Oct 1, 2024
6a66f71
Apply suggestions from review
bentsherman Oct 3, 2024
0d91594
Apply suggestions from code review
bentsherman Oct 3, 2024
3d29edb
Update description of expressions
bentsherman Oct 8, 2024
71236c9
Suggestions for syntax page
christopher-hakkaart Oct 9, 2024
505c98e
Revert "Suggestions for syntax page"
christopher-hakkaart Oct 9, 2024
0645746
Apply suggestions from review
bentsherman Oct 17, 2024
9444ed1
Apply suggestions from review
bentsherman Oct 18, 2024
110ede7
Apply suggestions from review
bentsherman Oct 18, 2024
48ac8b5
Update docs/dsl1.md
bentsherman Oct 21, 2024
30ea996
Revert "Update docs/dsl1.md"
bentsherman Oct 21, 2024
93154ee
Update docs/config.md
bentsherman Oct 21, 2024
04a311d
Remove redundant subscriber code snippet
bentsherman Oct 21, 2024
63633d1
Add note about implicit closure parameter
bentsherman Oct 23, 2024
17b7ab5
Merge branch 'master' into docs-strict-syntax
pditommaso Oct 23, 2024
3f6f377
Update docs [ci skip]
pditommaso Oct 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/channel.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ process foo {

workflow {
result = foo(1)
result.view { txt -> "Result: ${txt}" }
result.view { file -> "Result: ${file}" }
}
```

Expand Down
2 changes: 2 additions & 0 deletions docs/developer/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ This page describes how to create, test, and publish third-party plugins.

The best way to get started with your own plugin is to refer to the [nf-hello](https://github.com/nextflow-io/nf-hello) repository. This repository provides a minimal plugin implementation with several examples of different extension points and instructions for building, testing, and publishing.

Plugins can be written in Java or Groovy.

The minimal dependencies are as follows:

```groovy
Expand Down
30 changes: 15 additions & 15 deletions docs/module.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

# Modules

In Nextflow, a **module** is a script that may contain functions, processes, and workflows (collectively referred to as *components*). A module can be included in other modules or pipeline scripts and even shared across workflows.
Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts. When a script is included in this way, it is referred to as a **module**. Modules can be included by other modules or pipeline scripts and can even be shared across workflows.
christopher-hakkaart marked this conversation as resolved.
Show resolved Hide resolved

:::{note}
Modules were introduced in DSL2. If you are still using DSL1, see the {ref}`dsl1-page` page to learn how to migrate your Nextflow pipelines to DSL2.
:::

## Module inclusion

A component defined in a module script can be imported into another Nextflow script using the `include` keyword.
Any definition in a module can be included into another Nextflow script using the `include` keyword.
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

For example:

Expand All @@ -23,7 +23,7 @@ workflow {
}
```

The above snippet imports a process named `foo`, defined in the module script, into the main execution context. This way, `foo` can be invoked in the `workflow` scope.
The above snippet imports a process named `foo`, defined in the module, into the main execution context. This way, `foo` can be invoked in the `workflow` scope.

Nextflow implicitly looks for the script file `./some/module.nf`, resolving the path against the *including* script location.

Expand Down Expand Up @@ -57,7 +57,7 @@ Module directories allow the use of module scoped binaries scripts. See [Module

## Multiple inclusions

A Nextflow script can include any number of modules, and an `include` statement can import any number of components from a module. Multiple components can be included from the same module by using the syntax shown below:
A Nextflow script can include any number of modules, and an `include` statement can import any number of definitions from a module. Multiple definitions can be included from the same module by using the syntax shown below:

```groovy
include { foo; bar } from './some/module'
Expand All @@ -73,7 +73,7 @@ workflow {

## Module aliases

When including a module component, it's possible to specify an *alias* with the `as` keyword. Aliasing allows you to avoid module name clashes, by assigning them different names in the including context. For example:
When including definition from a module, it's possible to specify an *alias* with the `as` keyword. Aliasing allows you to avoid module name clashes, by assigning them different names in the including context. For example:

```groovy
include { foo } from './some/module'
Expand All @@ -85,7 +85,7 @@ workflow {
}
```

You can even include the same component multiple times under different names:
You can even include the same definition multiple times under different names:
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

```groovy
include { foo; foo as bar } from './some/module'
Expand All @@ -101,10 +101,10 @@ workflow {
## Module parameters

:::{deprecated} 24.07.0-edge
As a best practice, parameters should be used in the entry workflow and passed to functions / processes / workflows as explicit inputs.
As a best practice, parameters should be used in the entry workflow and passed to workflows, processes, and functions as explicit inputs.
:::

A module script can define parameters using the same syntax as a Nextflow workflow script:
A module can define parameters using the same syntax as a Nextflow workflow script:

```groovy
params.foo = 'Hello'
Expand Down Expand Up @@ -184,9 +184,9 @@ Ciao world!

## Module templates

The module script can be defined in an external {ref}`template <process-template>` file. The template file can be placed in the `templates` directory where the module script is located.
Process script {ref}`templates <process-template>` can be included alongside a module in the `templates` directory.

For example, suppose we have a project L with a module script that defines two processes, P1 and P2, both of which use templates. The template files can be made available in the local `templates` directory:
For example, suppose we have a project L with a module that defines two processes, P1 and P2, both of which use templates. The template files can be made available in the local `templates` directory:
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

```
Project L
Expand All @@ -210,15 +210,15 @@ Pipeline B
└── main.nf
```

With the possibility to keep the template files inside the project L, A and B can use the modules defined in L without any changes. A future project C would do the same, just cloning L (if not available on the system) and including its module script.
With the possibility to keep the template files inside the project L, A and B can use the modules defined in L without any changes. A future project C would do the same, just cloning L (if not available on the system) and including its module.
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

Beside promoting the sharing of modules across pipelines, there are several advantages to keeping the module template under the script path:

1. module components are *self-contained*,
2. module components can be tested independently from the pipeline(s) that import them,
3. it is possible to create libraries of module components.
1. modules are *self-contained*,
2. modules can be tested independently from the pipeline(s) that import them,
3. it is possible to create libraries of modules.
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

Ultimately, having multiple template locations allows a more structured organization within the same project. If a project has several module components, and all of them use templates, the project could group module scripts and their templates as needed. For example:
Ultimately, having multiple template locations allows a more structured organization within the same project. If a project has several modules, and all of them use templates, the project could group module scripts and their templates as needed. For example:
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

```
baseDir
Expand Down
4 changes: 2 additions & 2 deletions docs/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ In the above example, `$USER` is treated as a Bash variable, while `!{str}` is t

### Native execution

Whereas the `script` block defines a script that is executed as a separate job, the `exec` block simply executes the code that it is given.
Whereas the `script` block defines a script that is executed as a separate job, the `exec` block simply executes the code that it is given, without launching a job.
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

For example:

Expand Down Expand Up @@ -876,7 +876,7 @@ process randomNum {

workflow {
numbers = randomNum()
numbers.view { v -> "Received: ${v.text}" }
numbers.view { file -> "Received: ${file.text}" }
}
```

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/feature-flags.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
(feature-flags)=
(config-feature-flags)=

# Feature flags

Expand Down
6 changes: 3 additions & 3 deletions docs/reference/operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ Channel
.collectFile( name: 'result.fa', sort: { v -> v.size() } ) {
v -> v.sequence
}
.view { v -> v.text }
.view { fa -> fa.text }
```

:::{warning}
Expand Down Expand Up @@ -748,10 +748,10 @@ Available options:
: The zero-based index of each item to use as the matching key. Can also be a list of indices, e.g. `by: [0, 2]` (default: `[0]`).

`failOnDuplicate`
: When `true`, an error is reported when the operator receives multiple items from the same channel with the same key (default: `false`). Value is set to `true` if {ref}`strict mode <feature-flags>` is enabled.
: When `true`, an error is reported when the operator receives multiple items from the same channel with the same key (default: `false`). Value is set to `true` if {ref}`strict mode <config-feature-flags>` is enabled.

`failOnMismatch`
: When `true`, an error is reported when the operator receives an item from one channel for which there no matching item from the other channel (default: `false`). Value is set to `true` if {ref}`strict mode <feature-flags>` is enabled. This option cannot be used with `remainder`.
: When `true`, an error is reported when the operator receives an item from one channel for which there no matching item from the other channel (default: `false`). Value is set to `true` if {ref}`strict mode <config-feature-flags>` is enabled. This option cannot be used with `remainder`.

`remainder`
: When `true`, unmatched items are emitted at the end, otherwise they are discarded (default: `false`).
Expand Down
27 changes: 0 additions & 27 deletions docs/reference/stdlib.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,6 @@ The following constants are globally available in a Nextflow script:

`params`
: Map of workflow parameters specified in the config file or as command line options.
: :::{note}
As a best practice, parameters should only be used in the entry workflow.
:::

`projectDir`
: Alias of `workflow.projectDir`.
Expand Down Expand Up @@ -330,18 +327,6 @@ The following methods are available for a `Duration` object:
`getSeconds()`, `toSeconds()`
: Get the duration value in seconds (rounded down).

(stdlib-list)=

## List

TODO

(stdlib-map)=

## Map

TODO

(stdlib-types-memoryunit)=

## MemoryUnit
Expand Down Expand Up @@ -700,15 +685,3 @@ The following methods are available for splitting and counting the records in fi

`splitText()`
: Splits a text file into a list of lines. See the {ref}`operator-splittext` operator for available options.

(stdlib-set)=

## Set

TODO

(stdlib-string)=

## String

TODO
8 changes: 1 addition & 7 deletions docs/reference/syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ The first line of a script can be a [shebang](https://en.wikipedia.org/wiki/Sheb

### Feature flag

A feature flag declaration is an assignment, where the target should be a valid {ref}`feature flag <feature-flags>` and the source should be a literal (e.g. number, string, boolean):
A feature flag declaration is an assignment, where the target should be a valid {ref}`feature flag <config-feature-flags>` and the source should be a literal (e.g. number, string, boolean):
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

```groovy
nextflow.preview.topic = true
Expand Down Expand Up @@ -629,8 +629,6 @@ Logic unconfined.

Note that a slashy string cannot be empty because it would become a line comment.

Refer to {ref}`stdlib-string` for the set of available string operations.

### Dynamic string

Double-quoted strings can be interpolated using the `${}` placeholder, which can contain any expression:
Expand Down Expand Up @@ -669,8 +667,6 @@ A list literal consists of square brackets with a comma-separated list of zero o
[1, 2, 3]
```

Refer to {ref}`stdlib-list` for the set of available list operations.

### Map

A map literal consists of square brackets with a comma-separated list of one or more key-value pairs, with the key and value separated by a colon:
Expand All @@ -693,8 +689,6 @@ def x = 'foo'
// -> ['foo': 1]
```

Refer to {ref}`stdlib-map` for the set of available map operations.

### Closure

A closure, also known as an anonymous function, consists of a parameter list followed by zero or more statements, wrapped in curly braces:
Expand Down
Loading
Loading