Merge pull request #16 from CCBR/cli

Update CLI

.github/CONTRIBUTING.md

@@ -12,14 +12,27 @@ Pre-commit will run the hooks every time you commit.
 
 ## Versions
 
-Increment the version number following semantic versioning[^3] in the `VERSION` file.
+Increment the version number following semantic versioning[^1] in the `VERSION` file.
 
-[^3]: semantic versioning guidelines https://semver.org/
+[^1]: semantic versioning guidelines https://semver.org/
 
 ## Changelog
 
-Keep the changelog up to date with all changes in `docs/CHANGELOG.md`.
+Keep the changelog up to date with all notable changes in `CHANGELOG.md`[^2].
+
+[^2]: changelog guidelines: https://keepachangelog.com/en/1.1.0/
 
 ## VS code extensions
 
 If you use VS code, installing [nf-core extension pack](https://marketplace.visualstudio.com/items?itemName=nf-core.nf-core-extensionpack) is recommended.
+
+## Installation
+
+For testing and debugging, We recommend installing the dev version of cruise to a user-specific location.
+
+```
+git clone https://github.com/CCBR/CRUISE
+mkdir -p ~/bin/cruise
+pip install ./CRUISE -t ~/bin/cruise
+export PATH="$HOME/bin/cruise/bin:$PATH"
+```

CHANGELOG.md

@@ -12,6 +12,11 @@
   - fc (fold change)
   - bf (bayes factor)
   - pr (precision recall)
+- CLI (#16)
+  - `cruise init` to initialize a project directory.
+  - `cruise run`
+    - `--mode slurm` to submit a slurm job to biowulf or frce.
+    - `--main path/to/main.nf` to select a different local install of cruise, or specify the repo (`CCBR/CRUISE`) to get it from GitHub.
 
 <!--
 ## CRUISE v0.1.0

README.md

@@ -6,18 +6,48 @@
 
 [![build](https://github.com/CCBR/CRUISE/actions/workflows/build.yml/badge.svg)](https://github.com/CCBR/CRUISE/actions/workflows/build.yml)
 
+## Set up
+
+Cruise is installed on the [Biowulf HPC](#biowulf).
+For installation in other execution environments,
+refer to the [docs](https://ccbr.github.io/cruise).
+
+### Biowulf
+
+Cruise is available on [Biowulf](https://hpc.nih.gov/) in the `ccbrpipeliner` module.
+You'll first need to start an interactive session and create a directory from where you'll run cruise.
+
+```sh
+# start an interactive node
+sinteractive --mem=2g --cpus-per-task=2 --gres=lscratch:200
+# make a working directory for your project and go to it
+mkdir -p /data/$USER/chipseq
+cd /data/$USER/chipseq
+# load the ccbrpipeliener module
+module load ccbrpipeliner
+```
+
 ## Usage
 
-Run the test profile with stubs to see which processes will run
+Initialize and run cruise with test data:
 
 ```sh
-cruise run -profile test,singularity -stub
+# copy the cruise config files to your current directory
+cruise init
+# preview the cruise jobs that will run with the test dataset
+cruise run --mode local -profile test -preview
+# launch a cruise run on slurm with the test dataset
+cruise run --mode slurm -profile test,biowulf
 ```
 
-Run the test profile
+To run cruise on your own data, you'll need to create a sample sheet.
+Take a look at the example: [assets/samplesheet_test.csv](assets/samplesheet_test.csv).
+
+Once you've created a samplesheet with paths to your fastq files,
+run cruise with the `--input` option to specify the path to your sample sheet:
 
 ```sh
-cruise run -profile test,singularity
+cruise run --mode slurm -profile biowulf --input samplesheet.csv
 ```
 
 ## Help & Contributing

bin/install.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+version=$1
+
+repo_path=/data/CCBR_Pipeliner/Pipelines/CRUISE/cruise-dev/
+install_path=/data/CCBR_Pipeliner/Pipelines/CRUISE/${version}
+bin_path=${install_path}/bin/
+
+. "/data/CCBR_Pipeliner/db/PipeDB/Conda/etc/profile.d/conda.sh"
+conda activate py311
+
+echo "Installing CRUISE to ${install_path}"
+pip install ${repo_path} --target ${install_path}
+chmod +x ${install_path}/cruise/bin/*.*
+
+if [[ ":$PATH:" != *":${bin_path}:"* ]];then
+    export PATH="${PATH}:${bin_path}"
+fi
+
+if [[ ":$PYTHONPATH:" != *":${install_path}:"* ]];then
+    export PYTHONPATH="${PYTHONPATH}:${install_path}"
+fi

conf/biowulf.config

@@ -1,24 +1,19 @@
 
 params {
-  config_profile_description = 'Biowulf nf-core config'
-  config_profile_contact = '[email protected]'
-  config_profile_url = 'https://hpc.nih.gov/apps/nextflow.html'
-  max_memory = '224 GB'
-  max_cpus = 32
-  max_time = '72 h'
-
-  igenomes_base = '/fdb/igenomes/'
-}
-
-executor {
-
-    $slurm {
-        queue = 'norm'
-        queueSize = 200
-        pollInterval = '2 min'
-        queueStatInterval = '5 min'
-        submitRateLimit = '6/1min'
-        retry.maxAttempts = 1
+    config_profile_description = 'Biowulf nf-core config'
+    config_profile_contact = '[email protected]'
+    config_profile_url = 'https://hpc.nih.gov/apps/nextflow.html'
+    max_memory = '224 GB'
+    max_cpus = 32
+    max_time = '72 h'
+
+    igenomes_base = '/fdb/igenomes/'
+
+    // CCBR shared resource paths
+    index_dir = '/data/CCBR_Pipeliner/db/PipeDB/Indices'
+    fastq_screen {
+        conf = "assets/fastq_screen_biowulf.conf"
+        db_dir = '/data/CCBR_Pipeliner/db/PipeDB/lib/fastq_screen_db/'
     }
 }
 
@@ -30,20 +25,12 @@ singularity {
 }
 
 env {
-    SINGULARITY_CACHEDIR="/data/CCBR_Pipeliner/SIFS"
+    SINGULARITY_CACHEDIR = "/data/CCBR_Pipeliner/SIFS"
 }
 
-process {
-            executor = 'slurm'
-            maxRetries = 1
-
-            clusterOptions = ' --gres=lscratch:200 '
-
-            scratch = '/lscratch/$SLURM_JOBID'
-
-            stageInMode = 'symlink'
-            stageOutMode = 'rsync'
-
-            // for running pipeline on group sharing data directory, this can avoid inconsistent files timestamps
-            cache = 'lenient'
-}
+process.clusterOptions = ' --gres=lscratch:200 '
+process.scratch = '/lscratch/$SLURM_JOBID'
+process.stageInMode = 'symlink'
+process.stageOutMode = 'rsync'
+// for running pipeline on group sharing data directory, this can avoid inconsistent files timestamps
+process.cache = 'lenient'

conf/frce.config

@@ -0,0 +1,29 @@
+params {
+    config_profile_description = 'FRCE config'
+    max_memory = '224 GB'
+    max_cpus = 32
+    max_time = '72 h'
+
+    // CCBR shared resource paths
+    index_dir = null // TODO
+    fastq_screen {
+        conf = "assets/fastq_screen_frce.conf" // TODO
+        db_dir = null // TODO
+    }
+}
+
+singularity {
+    enabled = true
+    autoMounts = true
+    cacheDir = "/mnt/projects/CCBR-Pipelines/SIFs"
+    envWhitelist='https_proxy,http_proxy,ftp_proxy,DISPLAY,SLURM_JOBID,SINGULARITY_BINDPATH'
+}
+
+process.scratch = null // TODO
+
+process.stageInMode = 'symlink'
+process.stageOutMode = 'rsync'
+
+// for running pipeline on group sharing data directory, this can avoid inconsistent files timestamps
+process.cache = 'lenient'
+}

conf/interactive.config

@@ -0,0 +1,6 @@
+params {
+    config_profile_name = 'Run tasks locally on a slurm interactive node'
+    max_memory = '220 GB'
+    max_cpus = 56
+    max_time = '12 h'
+}

conf/slurm.config

@@ -0,0 +1,27 @@
+
+executor {
+
+    $slurm {
+        queue = 'norm'
+        queueSize = 200
+        pollInterval = '2 min'
+        queueStatInterval = '5 min'
+        submitRateLimit = '6/1min'
+        retry.maxAttempts = 1
+    }
+}
+
+process {
+    executor = 'slurm'
+    maxRetries = 1
+
+    clusterOptions = ' --gres=lscratch:200 '
+
+    scratch = '/lscratch/$SLURM_JOBID'
+
+    stageInMode = 'symlink'
+    stageOutMode = 'rsync'
+
+    // for running pipeline on group sharing data directory, this can avoid inconsistent files timestamps
+    cache = 'lenient'
+}

docs/getting-started.md

@@ -0,0 +1,63 @@
+# Getting Started with CRUISE
+
+TODO intro paragraph
+
+## Installation
+
+CRUISE is installed on the Biowulf and FRCE clusters as part of the
+`ccbrpipeliner` module.
+If you'd like to run the pipeline in a different execution environment,
+take a look at [how to run the nextflow pipeline directly](nextflow.md).
+
+## Prepare a sample sheet
+
+TODO
+
+## Initialize
+
+Copy the configuration files to your current working directory
+
+```sh
+cruise init
+```
+
+## Run
+
+TODO preview, stub, mode=slurm
+
+TODO required params
+
+Run preview to view processes that will run:
+
+```sh
+cruise run -profile test -preview
+```
+
+Launch a stub run to view processes that will run and download containers:
+
+```sh
+cruise run -profile test,singularity -stub
+```
+
+Run the test dataset using the test profile:
+
+```sh
+cruise run -profile test,singularity
+```
+
+or explicitly specify the output directory and input:
+
+```sh
+cruise run -profile singularity --outdir results/test --input assets/samplesheet_test.csv
+```
+
+### Custom reference genome
+
+TODO different required params
+
+Create and use a custom reference genome:
+
+```sh
+cruise run -profile test -entry MAKE_REFERENCE
+cruise run -profile test -c results/test/genome/custom_genome.config
+```

docs/index.md

@@ -1,6 +1,7 @@
+<!-- the README is inserted here automatically-->
+
 --8<-- "README.md"
 
-<!-- the README is inserted here automatically-->
 <!--- write additional info that you only want to appear in the docs index here -->
 
 Information on who the pipeline was developed for, and a statement if it's only been tested on Biowulf. For example: