Merge pull request #629 from MaxUlysse/Docs

Update docs

.github/CONTRIBUTING.md

@@ -15,10 +15,16 @@ is as follows:
 1. Check that there isn't already an issue about your idea in the
    [Sarek issues](https://github.com/SciLifeLab/Sarek/issues) to avoid
    duplicating work.
-    * Feel free to add a new issue here for the same reason.
+    * Feel free to add a [new issue here](https://github.com/SciLifeLab/Sarek/issues/new/choose) for the same reason.
 2. Fork the Sarek repository to your GitHub account
-3. Make the necessary changes / additions within your forked repository
-4. Submit a Pull Request against the master branch and wait for the code to be reviewed and merged.
+3. [Configure a remote for your fork](https://help.github.com/articles/configuring-a-remote-for-a-fork/)
+```
+git remote add upstream https://github.com/SciLifeLab/Sarek.git
+```
+
+4. [Sync your fork](https://help.github.com/articles/syncing-a-fork/)
+5. Make the necessary changes / additions within your forked repository
+6. Submit a [Pull Request](https://github.com/SciLifeLab/Sarek/compare) against the `dev` branch and wait for the code to be reviewed and merged.
 
 If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
 

PULL_REQUEST_TEMPLATE.md → .github/PULL_REQUEST_TEMPLATE.md

@@ -4,6 +4,8 @@ Please fill in the appropriate checklist below (delete whatever is not relevant)
 These are the most common things requested on pull requests (PRs).
 
 ## PR checklist
+ - [ ] PR is made against `dev` branch
+ - [ ] PR is a hotfix against `master` branch
  - [ ] This comment contains a description of changes (with reason)
  - [ ] If you've fixed a bug or added code that should be tested, add tests!
  - [ ] Ensure the test suite passes (`./scripts/test.sh -p docker -t ALL`).

.github/RELEASE_CHECKLIST.md

@@ -2,21 +2,22 @@
 This checklist is for our own reference
 
 1. Check that everything is up to date and ready to go
-  - Travis test is passing
-  - Manual testing on Bianca is passing
-2. Increase version numbers.
+    - Travis tests are passing
+    - Manual tests on Bianca are passing
+2. Increase version numbers
 3. Update version numbers in code: `configuration/base.config`
 4. Build, and get the containers.
-  - `./scripts/do_all.sh --push --tag <VERSION>`
-  - `./scripts/do_all.sh --pull --tag <VERSION>`
+    - `./scripts/do_all.sh --push --tag <VERSION>`
+    - `./scripts/do_all.sh --pull --tag <VERSION>`
 5. Test against sample data.
-  - Check for any command line errors
-  - Check version numbers are printed correctly
-  - `./scripts/test.sh -p docker --tag <VERSION>`
-  - `./scripts/test.sh -p singularity --tag <VERSION>`
-  - `./scripts/test.sh -p singularityPath --tag <VERSION>`
+    - Check for any command line errors
+    - Check version numbers are printed correctly
+    - `./scripts/test.sh -p docker --tag <VERSION>`
+    - `./scripts/test.sh -p singularity --tag <VERSION>`
+    - `./scripts/test.sh -p singularityPath --tag <VERSION>`
 6. Commit and push version updates
 7. Make a [release](https://github.com/SciLifeLab/Sarek/releases) on GitHub
-8. Tweet that new version is released
-9. Commit and push. Continue making more awesome :metal:
-10. Have fika :cake:
+8. Choose an appropriate codename for the release
+9. Tweet that new version is released
+10. Commit and push. Continue making more awesome :metal:
+11. Have fika :cake:

CHANGELOG.md

@@ -19,9 +19,13 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - [#615](https://github.com/SciLifeLab/Sarek/pull/615) - Use `splitCsv` instead of `readlines`
 - [#621](https://github.com/SciLifeLab/Sarek/pull/621) - Improve install script
 - [#621](https://github.com/SciLifeLab/Sarek/pull/621) - Simplify tests
+- [#627](https://github.com/SciLifeLab/Sarek/pull/627), [#629](https://github.com/SciLifeLab/Sarek/pull/629) - Refactor docs
+- [#629](https://github.com/SciLifeLab/Sarek/pull/629) - Refactor config
+- [#632](https://github.com/SciLifeLab/Sarek/pull/632) - Use 2 threads and 2 cpus FastQC processes
 
 ### `Removed`
 - [#616](https://github.com/SciLifeLab/Sarek/pull/616) - Remove old Issue Template
+- [#629](https://github.com/SciLifeLab/Sarek/pull/629) - Remove old Dockerfiles
 
 ### `Fixed`
 - [#621](https://github.com/SciLifeLab/Sarek/pull/621) - Fix VEP tests

Singularity

@@ -0,0 +1,18 @@
+From:nfcore/base
+Bootstrap:docker
+
+%labels
+    MAINTAINER Maxime Garcia <[email protected]>
+    DESCRIPTION Singularity image containing all requirements for the Sarek pipeline
+    VERSION 2.1.0
+
+%environment
+    PATH=/opt/conda/envs/sarek-2.1.0/bin:$PATH
+    export PATH
+
+%files
+    environment.yml /
+
+%post
+    /opt/conda/bin/conda env create -f /environment.yml
+    /opt/conda/bin/conda clean -a

annotate.nf

@@ -34,20 +34,6 @@ kate: syntax groovy; space-indent on; indent-width 2;
 ================================================================================
 */
 
-// Check that Nextflow version is up to date enough
-// try / throw / catch works for NF versions < 0.25 when this was implemented
-try {
-    if( ! nextflow.version.matches(">= ${params.nfRequiredVersion}") ){
-        throw GroovyException('Nextflow version too old')
-    }
-} catch (all) {
-    log.error "====================================================\n" +
-              "  Nextflow version ${params.nfRequiredVersion} required! You are running v${workflow.nextflow.version}.\n" +
-              "  Pipeline execution will continue, but things may break.\n" +
-              "  Please update Nextflow.\n" +
-              "============================================================"
-}
-
 if (params.help) exit 0, helpMessage()
 if (!SarekUtils.isAllowedParams(params)) exit 1, "params unknown, see --help for more information"
 if (!checkUppmaxProject()) exit 1, "No UPPMAX project ID found! Use --project <UPPMAX Project ID>"
@@ -279,27 +265,13 @@ if (params.verbose) vcfCompressedoutput = vcfCompressedoutput.view {
   "Index : ${it[3].fileName}"
 }
 
-process GetVersionBCFtools {
-  publishDir directoryMap.version, mode: 'link'
-  output: file("v_*.txt")
-  when: !params.noReports
-  script: QC.getVersionBCFtools()
-}
-
 process GetVersionSnpEFF {
   publishDir directoryMap.version, mode: 'link'
   output: file("v_*.txt")
   when: 'snpeff' in tools || 'merge' in tools
   script: QC.getVersionSnpEFF()
 }
 
-process GetVersionVCFtools {
-  publishDir directoryMap.version, mode: 'link'
-  output: file("v_*.txt")
-  when: !params.noReports
-  script: QC.getVersionVCFtools()
-}
-
 process GetVersionVEP {
   publishDir directoryMap.version, mode: 'link'
   output: file("v_*.txt")

bin/scrape_tool_versions.py

@@ -9,7 +9,7 @@
     'bcftools': ['v_bcftools.txt', r"bcftools (\S+)"],
     'BWA': ['v_bwa.txt', r"Version: (\S+)"],
     'FastQC': ['v_fastqc.txt', r"FastQC v(\S+)"],
-    'GATK': ['v_gatk.txt', r"GATK version(\S+)"],
+    'GATK': ['v_gatk.txt', r"Version:(\S+)"],
     'htslib': ['v_samtools.txt', r"htslib (\S+)"],
     'Manta': ['v_manta.txt', r"([0-9.]+)"],
     'MultiQC': ['v_multiqc.txt', r"multiqc, version (\S+)"],
@@ -28,24 +28,24 @@
 results = OrderedDict()
 results['Sarek'] = '<span style="color:#999999;\">N/A</span>'
 results['Nextflow'] = '<span style="color:#999999;\">N/A</span>'
+results['AlleleCount'] = '<span style="color:#999999;\">N/A</span>'
+results['ASCAT'] = '<span style="color:#999999;\">N/A</span>'
+results['bcftools'] = '<span style="color:#999999;\">N/A</span>'
 results['BWA'] = '<span style="color:#999999;\">N/A</span>'
-results['samtools'] = '<span style="color:#999999;\">N/A</span>'
-results['htslib'] = '<span style="color:#999999;\">N/A</span>'
+results['FastQC'] = '<span style="color:#999999;\">N/A</span>'
+results['FreeBayes'] = '<span style="color:#999999;\">N/A</span>'
 results['GATK'] = '<span style="color:#999999;\">N/A</span>'
-results['Picard'] = '<span style="color:#999999;\">N/A</span>'
+results['htslib'] = '<span style="color:#999999;\">N/A</span>'
 results['Manta'] = '<span style="color:#999999;\">N/A</span>'
-results['Strelka'] = '<span style="color:#999999;\">N/A</span>'
-results['FreeBayes'] = '<span style="color:#999999;\">N/A</span>'
-results['AlleleCount'] = '<span style="color:#999999;\">N/A</span>'
+results['MultiQC'] = '<span style="color:#999999;\">N/A</span>'
+results['Picard'] = '<span style="color:#999999;\">N/A</span>'
+results['Qualimap'] = '<span style="color:#999999;\">N/A</span>'
 results['R'] = '<span style="color:#999999;\">N/A</span>'
-results['ASCAT'] = '<span style="color:#999999;\">N/A</span>'
+results['samtools'] = '<span style="color:#999999;\">N/A</span>'
 results['SnpEff'] = '<span style="color:#999999;\">N/A</span>'
-results['VEP'] = '<span style="color:#999999;\">N/A</span>'
-results['FastQC'] = '<span style="color:#999999;\">N/A</span>'
-results['Qualimap'] = '<span style="color:#999999;\">N/A</span>'
-results['bcftools'] = '<span style="color:#999999;\">N/A</span>'
+results['Strelka'] = '<span style="color:#999999;\">N/A</span>'
 results['vcftools'] = '<span style="color:#999999;\">N/A</span>'
-results['MultiQC'] = '<span style="color:#999999;\">N/A</span>'
+results['VEP'] = '<span style="color:#999999;\">N/A</span>'
 
 # Search each file using its regex
 for k, v in regexes.items():

buildContainers.nf

@@ -34,20 +34,6 @@ kate: syntax groovy; space-indent on; indent-width 2;
 ================================================================================
 */
 
-// Check that Nextflow version is up to date enough
-// try / throw / catch works for NF versions < 0.25 when this was implemented
-try {
-    if( ! nextflow.version.matches(">= ${params.nfRequiredVersion}") ){
-        throw GroovyException('Nextflow version too old')
-    }
-} catch (all) {
-    log.error "====================================================\n" +
-              "  Nextflow version ${params.nfRequiredVersion} required! You are running v${workflow.nextflow.version}.\n" +
-              "  Pipeline execution will continue, but things may break.\n" +
-              "  Please update Nextflow.\n" +
-              "============================================================"
-}
-
 if (params.help) exit 0, helpMessage()
 if (!SarekUtils.isAllowedParams(params)) exit 1, "params unknown, see --help for more information"
 if (!checkUppmaxProject()) exit 1, "No UPPMAX project ID found! Use --project <UPPMAX Project ID>"
@@ -87,8 +73,9 @@ process BuildDockerContainers {
   when: params.docker
 
   script:
+  path = container == "sarek" ? "${baseDir}" : "${baseDir}/containers/${container}/."
   """
-  docker build -t ${params.repository}/${container}:${params.tag} ${baseDir}/containers/${container}/.
+  docker build -t ${params.repository}/${container}:${params.tag} ${path}
   """
 }
 

buildReferences.nf

@@ -37,20 +37,6 @@ kate: syntax groovy; space-indent on; indent-width 2;
 ================================================================================
 */
 
-// Check that Nextflow version is up to date enough
-// try / throw / catch works for NF versions < 0.25 when this was implemented
-try {
-    if( ! nextflow.version.matches(">= ${params.nfRequiredVersion}") ){
-        throw GroovyException('Nextflow version too old')
-    }
-} catch (all) {
-    log.error "====================================================\n" +
-              "  Nextflow version ${params.nfRequiredVersion} required! You are running v${workflow.nextflow.version}.\n" +
-              "  Pipeline execution will continue, but things may break.\n" +
-              "  Please update Nextflow.\n" +
-              "============================================================"
-}
-
 if (params.help) exit 0, helpMessage()
 if (!SarekUtils.isAllowedParams(params)) exit 1, "params unknown, see --help for more information"
 if (!checkUppmaxProject()) exit 1, "No UPPMAX project ID found! Use --project <UPPMAX Project ID>"

configuration/base.config → conf/base.config

@@ -11,46 +11,35 @@ wf_repository = 'maxulysse'
 
 params {
   // set up default params
-  annotateTools = '' // Tools to annotate by annotate.nf
-  annotateVCF = '' // Files to annotate by annotate.nf
   containerPath = '' // Path to Singularity images
-  containers = '' // List of containers to build in buildContainers.nf
   docker = false // Don't use Docker to build buildContainers.nf
   download = false // Don't download reference files in buildReferences.nf
   explicitBqsrNeeded = true // Enable recalibration in main.nf
   genome = 'GRCh38' // Default reference genome is GRCh38
-  genome_base = '' // Path to the reference files
   help = false // Don't give help information
   max_cpus = 16 // Base specifications
   max_memory = 128.GB // Base specifications
   max_time = 240.h // Base specifications
   more = false // Don't give version information
-  nfRequiredVersion = '0.25.0' // Minimum version of nextflow required
   noBAMQC = false // Use BAMQC
   noGVCF = false // HaplotypeCaller will output gVCF as well
   noReports = false // Reports are made by default
   nucleotidesPerSecond = 1000.0 // To estimate interval size by default
   onlyQC = false // All process will be run and not only the QC tools
   outDir = "${PWD}" // Path to output directory
-  project = '' // UPPMAX project number
   push = false // Don't push container to DockerHub
-  refDir = '' // Path to the references to build
   repository = wf_repository // DockerHub containers repository
-  sample = '' // sample files in tsv format
-  sampleDir = '' // samples directory (for Germline only)
-  sequencing_center = '' // CN field in BAM files
   singularity = false // Don't use singularity to build buildContainers.nf
   step = 'mapping' // Default step is mapping
   strelkaBP = false // Don't use Manta's candidate indels as input to Strelka
   tag = 'latest' // Default tag is latest, to be overwritten by --tag <version>
   test = false // Not testing by default
-  tools = '' // List of tools to use
   verbose = false // Enable for more verbose information
-  version = '2.0.0' // Workflow version
+  version = '2.1.0' // Workflow version
 }
 
 process {
-  $ConcatVCF {
+  withName:ConcatVCF {
     // For unknown reasons, ConcatVCF sometimes fails with SIGPIPE
     // (exit code 141). Rerunning the process will usually work.
     errorStrategy = {task.exitStatus == 141 ? 'retry' : 'terminate'}
@@ -69,6 +58,11 @@ timeline { // Turning on timeline tracking by default
   file = "${params.outDir}/Reports/Sarek_timeline.html"
 }
 
+dag { // Turning on dag by default
+  enabled = true
+  file = "${params.outDir}/Reports/Sarek_DAG.svg"
+}
+
 trace { // Turning on trace tracking by default
   enabled = true
   fields = 'process,task_id,hash,name,attempt,status,exit,realtime,%cpu,vmem,rss,submit,start,complete,duration,realtime,rchar,wchar'