How to use Nextflow profiles

This tutorial explains how to define Nextflow profiles and configuration files, and how to use them with YAMP.

Profiles

Nextflow (and therefore YAMP) can take advantage of profiles to define multiple sets of parameters to be used when launching a pipeline. These parameters can describe the Nextflow behaviour (e.g, whether to use Docker), the YAMP behaviour (e.g, whether to run the de-duplication step), or specific analysis parameters (e.g, where the adapters sequences are saved, or which Phred score use)

In YAMP, profiles are defined and collected in the nexflow.config file, where you can find some ready-to-use examples. For instance, YAMP provides a profile to tell Nextflow to use Docker:

docker {
  docker.enabled = true
  docker.runOptions = '-u \$(id -u):\$(id -g)'
}

and one to tell Nextflow to use Singularity:

singularity {
  singularity.enabled = true
}

We also defined a profile to be used on our HPC (Rosalind):

rosalind {
  singularity.enabled = true
  process.executor = 'slurm'
  process.queue = 'brc'
}

Configuration files and profiles

While the profiles above describe all the parameters within their specification, users can also specify these parameters within Nextflow config (configuration) files. A config file is a simple text file containing a set of parameters defined using the syntax

name = value

This is particularly useful when you have a long list of parameters or many profiles. Describing configuration files is outside the scope of this tutorial, but you can find more information in the Nextflow documentation.

In YAMP, configuration files are saved in the ./conf folder. Let's suppose that we have a config file for testing, called test.config (we have it, check it out!). You can tell Nextflow to use it by defining the following test profile:

test {
  includeConfig 'conf/test.config'
}

which tell Nextflow (and therefore YAMP) to include all the parameters specified in the conf/test.config file when the test profile is used. You can include multiple configuration files within the same profile, as we do, for instance, in our test_paired_end_complete profile:

test_paired_end_complete {
    includeConfig 'conf/test.config'
	includeConfig 'conf/test_paired_end_complete.config'
}

Use profiles during YAMP execution

To tell YAMP that you want to use a specific profile, you need to specify it on the command line, for instance:

nextflow run YAMP.nf -profile test

tells Nextflow/YAMP to use the test profile.

You can also combine multiple profiles by separating them with a comma:

nextflow run YAMP.nf -profile test,docker

More about configuration files and profiles

If you want to know more about both configuration files and profiles, please refer to the Nextflow documentation here, here, and here.