diff --git a/README.md b/README.md index 8b15ec5..e6a7220 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,9 @@ This is the repository for the Snakemake BYOC workshop to host lecture slides. -The workshop website can be found here: https://uppsala.instructure.com/courses/70024 +The workshop website can be found here: https://uppsala.instructure.com/courses/93486 Previous workshop websites: +August 2022: https://uppsala.instructure.com/courses/70024 September 2021: https://uppsala.instructure.com/courses/52153 October 2020: https://nbisweden.github.io/workshop-archive/workshop-snakemake-byoc/2020-10-07/ diff --git a/lectures/README.md b/lectures/README.md index 876d08e..030f2e7 100644 --- a/lectures/README.md +++ b/lectures/README.md @@ -11,6 +11,8 @@ conda activate lectures ## Rendering +### Rmarkdown + Lectures in **Rmarkdown** format can be rendered using the following from the command line: @@ -18,16 +20,25 @@ the command line: Rscript -e 'rmarkdown::render(, "xaringan::moon_reader")' ``` -The HTML files can be converted to PDF with this command: +### revealjs lectures + +Lectures in subfolders `smkprofile` and `wc-schema` are created with +[revealjs](https://revealjs.com) and can be rendered as follows: ```bash -Rscript -e 'library(webshot); webshot(, )' +Rscript -e 'rmarkdown::render()' ``` -### revealjs lectures +### Quarto lectures -Lectures in subfolders `smkprofile` and `wc-schema` are created with [revealjs](https://revealjs.com) and can be rendered as follows: +``` +quarto render +``` -```bash -Rscript -e 'rmarkdown::render()' -``` \ No newline at end of file +## HTML to PDF conversion + +For both Rmarkdown- and quarto-based lectures in HTML format, the tool +`https://github.com/astefanutti/decktape` was tested. + +Note that for Rmarkdown-based lecture slides, the `template.css` has to +be placed in the same directory as the Rmd- and HTML-file. diff --git a/lectures/anatomy-of-a-rule/anatomy.html b/lectures/anatomy-of-a-rule/anatomy.html index 4bdadcc..89ef961 100644 --- a/lectures/anatomy-of-a-rule/anatomy.html +++ b/lectures/anatomy-of-a-rule/anatomy.html @@ -1,652 +1,2910 @@ - - - Anatomy of a Snakefile - - - - - - - - - - + + + + + + + + + + + Anatomy of a Snakefile + + + + + + + + + + + + + + + +
+
+ +
+

Anatomy of a Snakefile

+

Snakemake BYOC NBIS course

+ +
+
+ +

2024-05-27

+
+
+

Basic structure of a rule

+
rule:
+    output: "results/sample1.stats.txt"
+    shell:
+        """
+        echo -e "sample1\t50%" > {output}
+        """
+
+
+

Basic structure of a rule

+
$ snakemake -c 1
+Assuming unrestricted shared filesystem usage.
+Building DAG of jobs...
+Using shell: /bin/bash
+Provided cores: 1 (use --cores to define parallelism)
+Rules claiming more threads will be scaled down.
+Job stats:
+job      count
+-----  -------
+1            1
+total        1
+
+Select jobs to execute...
+Execute 1 jobs...
+
+[Fri May 17 23:47:24 2024]
+localrule 1:
+    output: results/sample1.stats.txt
+    jobid: 0
+    reason: Missing output files: results/sample1.stats.txt
+    resources: tmpdir=/var/folders/wb/jf9h8kw11b734gd98s6174rm0000gp/T
+
+[Fri May 17 23:47:24 2024]
+Finished job 0.
+1 of 1 steps (100%) done
+Complete log: .snakemake/log/2024-05-17T234724.252920.snakemake.log
+
+
+

Basic structure of a rule

+

More commonly, rules are named and have both input and output:

+
rule generate_stats:
+    output: "results/sample1.stats.txt"
+    input: "results/sample1.bam"
+    shell:
+        """
+        samtools flagstat {input} > {output}
+        """
+
+
+

Basic structure of a rule

+

Rules are linked by their input and output files:

+
rule a:
+    output: "a.txt"
+    shell:
+        "echo 'a' > a.txt"
+rule b:
+    input: "a.txt"
+    output: "b.txt"
+    shell:
+        "cat a.txt > b.txt"
+
+
+

Basic structure of a rule

+

You can also link rules explicitly:

+
rule a:
+    output: "a.txt"
+    shell:
+        "echo 'a' > a.txt"
+
+rule b:
+    input: rules.a.output
+    output: "b.txt"
+    shell:
+        "cat {input} > {output}"
+

but then the rule that supplies the file must be define before the rule that uses it.

+
+
+

Wildcards

+

Wildcards generalize a workflow. Imagine you have not just sample1 but samples 1..100.

+

Instead of writing 100 rules…

+
rule generate_stats_sample1:
+    output: "results/sample1.stats.txt"
+    input: "results/sample1.bam"
+...
+rule generate_stats_sample100:
+    output: "results/sample100.stats.txt"
+    input: "results/sample100.bam"
+
+
+

Wildcards

+

…we can introduce one or more wildcards which Snakemake can match to several text strings using regular expressions.

+

In our example, we replace the actual sample ids with the wildcard sample:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    shell:
+        """
+        samtools flagstat {input} > {output}
+        """
+
+
+

Wildcards

+

Rules can have multiple wildcards…

+
rule generate_stats:
+    output: "results/{sample}_{lane}.stats.txt"
+    input: "results/{sample}_{lane}.bam"
+    shell:
+      """
+      samtools flagstats {input} > {output}
+      """
+
+
+

Wildcards

+

…but all the wildcards must be present in the output section.

+
+

Will work:

+
rule generate_stats:
+    output: "results/{sample}_{lane}.stats.txt"
+    input: "results/{sample}.bam"
+
+
+

Won’t work.

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}_{lane}.bam"
+
Wildcards in input files cannot be determined from output files: 'lane'
+
+
+
+

Rule ambiguities

+

Ambiguities can arise when two rules produce the same output:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    shell:
+        """
+        samtools flagstat {input} > {output}
+        """
+
+rule print_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.log"
+    shell:
+        """
+        grep "% alignment" {input} > {output}
+        """
+      
+rule make_report:
+    output: "results/{sample}.report.pdf"
+    input: "results/{sample}.stats.txt"
+
+
+

Rule ambiguities

+
$ snakemake -c 1 -n results/sample1.report.pdf
+Building DAG of jobs...
+AmbiguousRuleException:
+Rules generate_stats and print_stats are ambiguous for the file 
+results/sample1.stats.txt.
+
+
+

Rule ambiguities

+

This can be handled in a number of ways:

+
+
    +
  • by changing the output file name of one of the rules
    • +
+
+
+
    +
  • or via the ruleorder directive:
    • +
+
ruleorder: generate_stats > print_stats
+
+
+
    +
  • or by specifically referring to the output of a certain rule:
    • +
+
rule make_report:
+    output: "results/{sample}.report.pdf"
+    input: rules.generate_stats.output
+
+
+
+

Logging

+

Logfiles and messages add descriptions and help with debugging:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    shell:
+        """
+        samtools flagstat {input} > {output} 2>{log}
+        """
+
+
+
+
+ +
+

Tip

+
+
+

Log files are not deleted by snakemake if there’s an error.

+
+
+
+
+
+

Resources

+

Compute resources can be set with threads and resources:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: 4
+    resources:
+        mem_mb=100
+    shell:
+        """
+        samtools flagstat --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+

Resources

+

It’s also possible to set threads based on the cores given to snakemake (e.g. --cores 8 or -c 8).

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5 
+    resources:
+        mem_mb=100
+    shell:
+        """
+        samtools flagstat --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+

Resources

+

Resources can also be callables, allowing them to be set dynamically:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5 
+    resources:
+        mem_mb=lambda wildcards: 1000 if wildcards.sample == "sample1-large" else 100
+    shell:
+        """
+        samtools flagstat --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+

Parameters

+

Non-file rule parameters can be set with the params directive:

+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5
+    resources:
+        mem_mb=100
+    params:
+        verbosity = 2
+    shell:
+        """
+        samtools flagstat --verbosity {params.verbosity} \
+          --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+

Software environments

+

Software environments can be set for each rule using the conda: directive:

+
+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5
+    resources:
+        mem_mb=100
+    params:
+        verbosity = 2
+    conda: "envs/samtools.yml"
+    shell:
+        """
+        samtools flagstat --verbosity {params.verbosity} \
+          --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+
+

Software environments

+
+
rule generate_stats:
+    ...
+    conda: "envs/samtools.yml"
+    ...
+
+

Contents of envs/samtools.yml

+
name: samtools
+channels:
+  - bioconda
+dependencies:
+  - samtools=1.15.1
+
+

To make Snakemake use the conda environment, specify --software-deployment-method conda (or --sdm conda) on the command line. For Snakemake versions before 8.0, use --use-conda.

+
+
+
+

Software environments

+

On compute clusters, you can also specify packages to load with envmodules:

+
+
rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5
+    resources:
+        mem_mb=100
+    params:
+        verbosity = 2
+    conda: "envs/samtools.yml"
+    envmodules: 
+        "bioinfo-tools",
+        "samtools"
+    shell:
+        """
+        samtools flagstat --verbosity {params.verbosity} \
+          --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+
+

Software environments

+

On compute clusters, you can also specify packages to load with envmodules:

+
+
rule generate_stats:
+    ...
+    envmodules: 
+        "bioinfo-tools",
+        "samtools"
+    ...
+
+

To make Snakemake use envmodules, specify --use-envmodules on the command line.

+
+
+

Config files

+

Config files allow you to configure workflows without having to change the underlying code.

+
+

Config files should be in YAML or JSON format

+
+ +
+
+
samples: ["sample1", "sample2", "sample3"]
+verbosity: 2
+
+
+
{
+  "samples": [
+    "sample1",
+    "sample2",
+    "sample3"
+  ],
+  "verbosity": 2
+}
+
+
+
+
+
+
+

Config files

+

Specify one or more config files on the command line with:

+
snakemake --configfile config.yml -j 1
+
+

or directly in a snakefile, e.g.:

+
configfile: "config.yml""
+
+
+
+

Config files

+

The config parameters are available as a dictionary inside your snakefiles and can be accessed from within rules:

+
rule all:
+    input:
+        expand("results/{sample}.stats.txt", sample = config["samples"])
+
+rule generate_stats:
+    output: "results/{sample}.stats.txt"
+    input: "results/{sample}.bam"
+    log: "results/{sample}.flagstat.log"
+    message: "Generating stats for sample {wildcards.sample}"
+    threads: workflow.cores * 0.5
+    resources:
+        mem_mb=100
+    params:
+        verbosity = config["verbosity"]
+    conda: "envs/samtools.yml"
+    envmodules: 
+        "bioinfo-tools",
+        "samtools"
+    shell:
+        """
+        samtools flagstat --verbosity {params.verbosity} \
+          --threads {threads} {input} > {output} 2>{log}
+        """
+
+
+

What else?

+

Snakemake is constantly being updated with new features. Check out the documentation, and specifically the section about writing rules.

+
+
+

Questions?

+
+

+
+

Snakemake BYOC 2024 - Reproducible Research

+
+
+
+
+
+ + + + + + + + + - - - - + - - + + + + + + + + + \ No newline at end of file diff --git a/lectures/anatomy-of-a-rule/anatomy.pdf b/lectures/anatomy-of-a-rule/anatomy.pdf deleted file mode 100644 index 2cbd064..0000000 Binary files a/lectures/anatomy-of-a-rule/anatomy.pdf and /dev/null differ diff --git a/lectures/anatomy-of-a-rule/anatomy.Rmd b/lectures/anatomy-of-a-rule/anatomy.qmd similarity index 56% rename from lectures/anatomy-of-a-rule/anatomy.Rmd rename to lectures/anatomy-of-a-rule/anatomy.qmd index 4e98900..97f74ff 100644 --- a/lectures/anatomy-of-a-rule/anatomy.Rmd +++ b/lectures/anatomy-of-a-rule/anatomy.qmd @@ -1,33 +1,30 @@ --- title: "Anatomy of a Snakefile" subtitle: "Snakemake BYOC NBIS course" -date: "`r format(Sys.time(), '%d %B, %Y')`" -output: - xaringan::moon_reader: - self-contained: true - seal: false - css: ["default", "../template.css"] - nature: - slideNumberFormat: "" +date: 2024-05-27 +format: + revealjs: + theme: + - white + - ../custom.scss + # - ../revealjs.css + embed-resources: true + toc: false + toc-depth: 1 + slide-level: 2 + slide-number: true + #preview-links: true + #chalkboard: true + # Multiple logos not possible; would need to make custom logo combining both logos + footer: Snakemake BYOC 2024 - Reproducible Research + logo: https://nbis.se/nbislogo-green.svg + smaller: true + highlight-style: gruvbox --- -layout: true - - +## Basic structure of a rule ---- - -class: center, middle - -.HUGE[Anatomy of] -
-.HUGE[a Snakefile] -
- ---- -# Basic structure of a rule - -```python +```{.python} rule: output: "results/sample1.stats.txt" shell: @@ -35,39 +32,42 @@ rule: echo -e "sample1\t50%" > {output} """ ``` --- +## Basic structure of a rule -```bash -$ snakemake -c 1 results/sample1.stats.txt +```{.bash code-line-numbers="false"} +$ snakemake -c 1 +Assuming unrestricted shared filesystem usage. Building DAG of jobs... Using shell: /bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. -Job counts: - count jobs - 1 1 - 1 +Job stats: +job count +----- ------- +1 1 +total 1 + Select jobs to execute... +Execute 1 jobs... -[Tue Sep 28 11:50:45 2021] -rule 1: +[Fri May 17 23:47:24 2024] +localrule 1: output: results/sample1.stats.txt jobid: 0 + reason: Missing output files: results/sample1.stats.txt + resources: tmpdir=/var/folders/wb/jf9h8kw11b734gd98s6174rm0000gp/T -[Tue Sep 28 11:50:45 2021] +[Fri May 17 23:47:24 2024] Finished job 0. 1 of 1 steps (100%) done - -$ cat results/sample1.stats.txt -sample1 50% +Complete log: .snakemake/log/2024-05-17T234724.252920.snakemake.log ``` ---- -# Basic structure of a rule +## Basic structure of a rule More commonly, rules are named and have both input and output: -```python +```{.python} rule generate_stats: output: "results/sample1.stats.txt" input: "results/sample1.bam" @@ -76,17 +76,49 @@ rule generate_stats: samtools flagstat {input} > {output} """ ``` ---- -# Wildcards +## Basic structure of a rule + +Rules are linked by their input and output files: + +```{.python} +rule a: + output: "a.txt" + shell: + "echo 'a' > a.txt" +rule b: + input: "a.txt" + output: "b.txt" + shell: + "cat a.txt > b.txt" +``` + +## Basic structure of a rule + +You can also link rules explicitly: + +```{.python code-line-numbers="2,7"} +rule a: + output: "a.txt" + shell: + "echo 'a' > a.txt" + +rule b: + input: rules.a.output + output: "b.txt" + shell: + "cat {input} > {output}" +``` + +but then the rule that supplies the file must be define before the rule that uses it. -.green[Wildcards] generalize a workflow. Imagine you have not just sample1 but samples 1..100. +## Wildcards {auto-animate=true} --- +Wildcards generalize a workflow. Imagine you have not just sample1 but samples 1..100. Instead of writing 100 rules... -```python +```{.python code-line-numbers="false"} rule generate_stats_sample1: output: "results/sample1.stats.txt" input: "results/sample1.bam" @@ -96,9 +128,9 @@ rule generate_stats_sample100: input: "results/sample100.bam" ``` --- +## Wildcards {auto-animate=true} -...we can introduce one or more .green[wildcards] which Snakemake can match to several text strings using regular expressions. +...we can introduce one or more `wildcards` which Snakemake can match to several text strings using regular expressions. In our example, we replace the actual sample ids with the wildcard `sample`: @@ -112,13 +144,12 @@ rule generate_stats: samtools flagstat {input} > {output} """ ``` ---- -# Wildcards +## Wildcards Rules can have multiple wildcards... -```python +```{.python} rule generate_stats: output: "results/{sample}_{lane}.stats.txt" input: "results/{sample}_{lane}.bam" @@ -128,35 +159,37 @@ rule generate_stats: """ ``` --- +## Wildcards -...but wildcards **must** be present in the output section. +...but all the wildcards **must** be present in the output section. +::: {.fragment} **Will work:** -```python +```{.python} rule generate_stats: output: "results/{sample}_{lane}.stats.txt" input: "results/{sample}.bam" ``` +::: --- - -**Won't work:** -```python +::: {.fragment} +Won't work. +```{.python} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}_{lane}.bam" ``` -```bash + +```{.bash code-line-numbers="false"} Wildcards in input files cannot be determined from output files: 'lane' ``` +::: ---- -# Rule ambiguities +## Rule ambiguities {auto-animate=true} Ambiguities can arise when two rules produce the same output: -```python +```{.python} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -177,44 +210,46 @@ rule make_report: output: "results/{sample}.report.pdf" input: "results/{sample}.stats.txt" ``` --- -```bash + +## Rule ambiguities {auto-animate=true} + +```{.bash code-line-numbers="false"} $ snakemake -c 1 -n results/sample1.report.pdf Building DAG of jobs... AmbiguousRuleException: -Rules generate_stats and print_stats are ambiguous for the file results/sample1.stats.txt. +Rules generate_stats and print_stats are ambiguous for the file +results/sample1.stats.txt. ``` ---- -# Rule ambiguities - -Ambiguities can arise when two rules produce the same output: +## Rule ambiguities This can be handled in a number of ways: +::: {.fragment} - by changing the output file name of one of the rules +::: --- - +::: {.fragment} - or via the `ruleorder` directive: -```python +```{.python} ruleorder: generate_stats > print_stats ``` +::: --- - +::: {.fragment} - or by specifically referring to the output of a certain rule: -```python +```{.python} rule make_report: output: "results/{sample}.report.pdf" input: rules.generate_stats.output ``` +::: ---- -# Logging +## Logging Logfiles and messages add descriptions and help with debugging: -```python + +```{.python} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -225,15 +260,17 @@ rule generate_stats: samtools flagstat {input} > {output} 2>{log} """ ``` +:::{.callout-tip} Log files are not deleted by snakemake if there's an error. ---- -# Resources +::: -Compute resources can be set with .green[threads] and .green[resources]: +## Resources {auto-animate=true} -```python +Compute resources can be set with **threads** and **resources**: + +```{.python} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -247,18 +284,18 @@ rule generate_stats: samtools flagstat --threads {threads} {input} > {output} 2>{log} """ ``` ---- -# Resources -Compute resources can be set with .green[threads] and .green[resources]: +## Resources {auto-animate=true} -```python +It's also possible to set threads based on the cores given to snakemake (_e.g._ `--cores 8` or `-c 8`). + +```{.python code-line-numbers="6"} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" log: "results/{sample}.flagstat.log" message: "Generating stats for sample {wildcards.sample}" - threads: workflow.cores * 0.5 # <---- threads as a function of workflow cores + threads: workflow.cores * 0.5 resources: mem_mb=100 shell: @@ -267,17 +304,30 @@ rule generate_stats: """ ``` -It's also possible to set threads based on the cores given to snakemake (_e.g._ `--cores 8` or `-c 8`). +## Resources {auto-animate=true} --- +Resources can also be callables, allowing them to be set dynamically: -More on resources in a lecture tomorrow. +```{.python code-line-numbers="8"} +rule generate_stats: + output: "results/{sample}.stats.txt" + input: "results/{sample}.bam" + log: "results/{sample}.flagstat.log" + message: "Generating stats for sample {wildcards.sample}" + threads: workflow.cores * 0.5 + resources: + mem_mb=lambda wildcards: 1000 if wildcards.sample == "sample1-large" else 100 + shell: + """ + samtools flagstat --threads {threads} {input} > {output} 2>{log} + """ +``` ---- -# Parameters +## Parameters {auto-animate=true auto-animate-restart=true} -Rule parameters can be set with the .green[params] directive: -```python +Non-file rule parameters can be set with the **params** directive: + +```{.python} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -294,12 +344,13 @@ rule generate_stats: --threads {threads} {input} > {output} 2>{log} """ ``` ---- -# Software environments -Software environments can be set for each rule using `conda:`: +## Software environments {auto-animate=true} -```python +Software environments can be set for each rule using the `conda:` directive: + +::: {data-id="code1"} +```{.python code-line-numbers="11"} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -317,9 +368,22 @@ rule generate_stats: --threads {threads} {input} > {output} 2>{log} """ ``` +::: + +## Software environments {auto-animate=true auto-animate-easing=None} + +::: {data-id="code1"} +```{.python code-line-numbers="3"} +rule generate_stats: + ... + conda: "envs/samtools.yml" + ... +``` +::: + +Contents of `envs/samtools.yml` ```yaml -### Contents of envs/samtools.yml ### name: samtools channels: - bioconda @@ -327,14 +391,16 @@ dependencies: - samtools=1.15.1 ``` -To make Snakemake use the conda environment, specify `--use-conda` on the command line. +:::{.fragment} +To make Snakemake use the conda environment, specify `--software-deployment-method conda` (or `--sdm conda`) on the command line. For Snakemake versions before 8.0, use `--use-conda`. +::: ---- -# Software environments +## Software environments {auto-animate=true auto-animate-restart=true} -Or by using `envmodules`, _e.g._ in compute clusters: +On compute clusters, you can also specify packages to load with `envmodules:` -```python +::: {data-id="code2"} +```{.python code-line-numbers="12-14"} rule generate_stats: output: "results/{sample}.stats.txt" input: "results/{sample}.bam" @@ -355,30 +421,45 @@ rule generate_stats: --threads {threads} {input} > {output} 2>{log} """ ``` +::: + +## Software environments {auto-animate=true auto-animate-easing=None auto-animate-delay=0} + +On compute clusters, you can also specify packages to load with `envmodules:` + +::: {data-id="code2"} +```{.python code-line-numbers="3-5"} +rule generate_stats: + ... + envmodules: + "bioinfo-tools", + "samtools" + ... +``` +::: To make Snakemake use envmodules, specify `--use-envmodules` on the command line. --- +## Config files -More on conda and envmodules tomorrow. +**Config files** allow you to configure workflows without having to change the underlying code. ---- -# Config files +:::{.fragment} -.green[Config files] allow you to configure workflows without having to change the underlying code. +Config files should be in `YAML` or `JSON` format --- +::: {.panel-tabset} -.small[Config files should be in `YAML` or `JSON` format:] +### YAML -```yaml -### Contents of config.yml ### +```{.yaml code-line-numbers="false"} samples: ["sample1", "sample2", "sample3"] verbosity: 2 ``` -```json -### Contents of config.json ### +### JSON + +```{.json code-line-numbers="false"} { "samples": [ "sample1", @@ -388,59 +469,32 @@ verbosity: 2 "verbosity": 2 } ``` +::: +::: --- +## Config files -.small[Specify one or more config files on the command line with:] -```bash +Specify one or more config files on the command line with: + +```{.bash code-line-numbers="false"} snakemake --configfile config.yml -j 1 ``` --- - -.small[Or directly in a snakefile, _e.g._:] -```python +:::{.fragment} +or directly in a snakefile, _e.g._: +```{.python code-line-numbers="false"} configfile: "config.yml"" ``` +::: ---- -# Config files - -The config parameters are available as a dictionary inside your snakefiles: - -```yaml -### Contents of config.yml ### -samples: ["sample1", "sample2", "sample3"] -verbosity: 2 -``` - -```python -configfile: "config.yml" -print(config) -{'samples': ['sample1', 'sample2', 'sample3'], 'verbosity': 2} -``` - --- +## Config files -This allows you to manipulate the config variable inside your snakemake files and python code. - -```python -config["samples"].append("sample4") -print(config) -{'samples': ['sample1', 'sample2', 'sample3', 'sample4'], 'verbosity': 2} -``` - ---- -# Config files - -In our example rule, verbosity level can be controlled via a config file like this: - -```python -configfile: "config.yml" +The config parameters are available as a dictionary inside your snakefiles and can be accessed from within rules: +```{.python code-line-numbers="3,14"} rule all: - input: - expand("results/{sample}.stats.txt", sample = config["samples"]) + input: + expand("results/{sample}.stats.txt", sample = config["samples"]) rule generate_stats: output: "results/{sample}.stats.txt" @@ -451,7 +505,7 @@ rule generate_stats: resources: mem_mb=100 params: - verbosity = config["verbosity"] # <----- + verbosity = config["verbosity"] conda: "envs/samtools.yml" envmodules: "bioinfo-tools", @@ -462,87 +516,9 @@ rule generate_stats: --threads {threads} {input} > {output} 2>{log} """ ``` ---- -# Config files - -Config files are also convenient for defining what the workflow will do. - --- - -If no targets are given on the command line, Snakemake will run the first rule specified. - -By convention this rule is named `all` and is used as a 'pseudo-rule' to define what the workflow will generate. - -```python -rule all: - input: - "results/sample1.stats.txt", - "results/sample2.stats.txt", - "results/sample3.stats.txt" - -rule generate_stats: - output: "results/{sample}.stats.txt" - input: "results/{sample}.bam" - log: "results/{sample}.flagstat.log" -... -``` ---- -# Config files - -Config files are also convenient for defining what the workflow will do. - -If no targets are given on the command line, Snakemake will run the first rule specified. - -By convention this rule is named `all` and is used as a 'pseudo-rule' to define what the workflow will generate. - -```python -samples = ["sample1", "sample2", "sample3"] -rule all: - input: - expand("results/{sample}.stats.txt", sample = config["samples"]) - -rule generate_stats: - output: "results/{sample}.stats.txt" - input: "results/{sample}.bam" - log: "results/{sample}.flagstat.log" -... -``` - -If we define a list of samples we can condense the input section of the `all` rule using the .green[expand] function. +## What else? ---- -# Config files - -By defining samples in the config file (either directly or via a sample list that is read and stored in the config dictionary), your workflow becomes way more flexible. - -```yaml -### Contents of config.yml ### -samples: ["sample1", "sample2", "sample3"] -verbosity: 2 -``` - -```python -configfile: "config.yml" -rule all: - input: - expand("results/{sample}.stats.txt", sample = config["samples"]) - -rule generate_stats: - output: "results/{sample}.stats.txt" - input: "results/{sample}.bam" - log: "results/{sample}.flagstat.log" -... -``` - ---- - -# What else? - -Snakemake is constantly being updated with new features. Check out the documentation (https://snakemake.readthedocs.io/), and specifically the section about [writing rules](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html). - - ---- +Snakemake is constantly being updated with new features. Check out the [documentation](https://snakemake.readthedocs.io/), and specifically the section about [writing rules](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html). -class: center, middle -# Questions? \ No newline at end of file +## Questions? \ No newline at end of file diff --git a/lectures/example-workflow/example-workflow.Rmd b/lectures/example-workflow/example-workflow.Rmd index eff6a20..8dcde02 100644 --- a/lectures/example-workflow/example-workflow.Rmd +++ b/lectures/example-workflow/example-workflow.Rmd @@ -6,7 +6,7 @@ output: xaringan::moon_reader: self-contained: true seal: false - css: ["default", "../template.css"] + css: ["default", "template.css"] nature: slideNumberFormat: "" --- @@ -25,7 +25,8 @@ class: center, middle ```{r Setup, echo = FALSE, message = FALSE} # Knitr setup knitr::opts_chunk$set(message = FALSE, - warning = FALSE) + warning = FALSE, + python.reticulate = FALSE) # Load packages library("dplyr") @@ -60,7 +61,7 @@ library("kableExtra") # The GenErode pipeline -* Started at Snakemake version 3.10 (!), current pipeline runs with Snakemake version 6.12.1 (latest version: 7.13.0) +* Started at Snakemake version 3.10 (!), current pipeline runs with Snakemake version 7.20.0 (latest version: 8.11.6) -- @@ -106,8 +107,10 @@ library("kableExtra") -- -* Cluster execution (e.g. UPPMAX) with slurm: - * `config/cluster.yaml` file to set up slurm profile +* Cluster execution with slurm (e.g. UPPMAX): + * `config/slurm/` folder with configuration files for the slurm profile and for + the deprecated cluster configuration (`--cluster` still exists in Snakemake v7 + but not in v8 anymore) --- diff --git a/lectures/example-workflow/example-workflow.pdf b/lectures/example-workflow/example-workflow.pdf index 0813293..2656940 100644 Binary files a/lectures/example-workflow/example-workflow.pdf and b/lectures/example-workflow/example-workflow.pdf differ diff --git a/lectures/example-workflow/template.css b/lectures/example-workflow/template.css new file mode 100644 index 0000000..a197c05 --- /dev/null +++ b/lectures/example-workflow/template.css @@ -0,0 +1,124 @@ +/* CSS style general NBIS presentations using Xaringen and RMarkdown */ + +/* Import Google font */ +@import url('https://fonts.googleapis.com/css?family=Open+Sans&display=swap'); + +/* Fonts */ +body,p,h1,h2,h3,h4,h5,h6 { + font-family: 'Open Sans', sans-serif; + font-weight: 400; + font-size: 20px; + color: #333333; +} + +/* Header 1 font size */ +h1 { + font-size: 1.7em !important; +} + +/* Header 2 font size */ +h2 { + font-size: 1.2em !important; +} + +/* Header 3 font size */ +h3 { + font-size: 0.9em !important; +} + +/* Custom font sizes */ +.micro { font-size: 25% } +.tiny { font-size: 50% } +.small { font-size: 75% } +.large { font-size: 125% } +.huge { font-size: 150% } +.Huge { font-size: 200% } +.HUGE { font-size: 250% } + +.remark-code { font-size: 50% } + +/* Custom colours */ +.green { color: #85be42; font-weight: bold } +.dark-green { color: #487a2d; font-weight: bold } +.blue { color: #0099c2; font-weight: bold } +.dark-blue { color: #015491; font-weight: bold } +.light-grey { color: #999999 } +.grey { color: #666666 } +.dark-grey { color: #333333 } + +/* Set the maximum figure size to 100% of the slide */ +img, video, iframe { + max-width: 100%; + max-height: 100%; +} + +table_rot { + color: black; + text-transform: uppercase; + @include transform(rotate(270deg)) + @include transform-origin(0 0); +} + +/* SciLifeLab logo in bottom left corner */ +div.scilife-logo { + content: ""; + background-image: url("https://www.scilifelab.se/wp-content/uploads/2020/09/SciLifeLab_Logotype_Green_POS_large-copy-1536x334.png"); + background-repeat: no-repeat; + background-size: contain; + position: absolute; + bottom: 10px; + left: 15px; + height: 50px; + width: 200px; +} + +/* NBIS logo in bottom right corner */ +div.nbis-logo { + content: ""; + background-image: url("https://nbis.se/assets/img/logos/nbislogo-green.svg"); + background-repeat: no-repeat; + background-size: contain; + position: absolute; + bottom: 23px; + right: 27px; + height: 35px; + width: 100px; +} + +/* Blockquotes */ +blockquote { + border-left: 5px solid #85be42; + background: #f9f9f9; + font-style: italic; + padding: 0.01em; + padding-left: 12px; + padding-right: 12px; + margin-left: 12px; + margin-right: 12px; + border-radius: 0 5px 5px 0; +} + +/* Link colour */ +a { + color: #85be42; + text-decoration: none; +} + +/* Underline links when hovering */ +a:hover { + text-decoration: underline; +} + +/* Inline code color and border */ +.remark-inline-code { + background: #f2f2f2; + border-radius: 3px; + margin-right: 2px; + padding: 2px 3px 1px; + font-size: 80% !important; +} + +.image-with-text { + display: inline-block; + vertical-align: middle; +} diff --git a/lectures/scatter-gather/Snakefile b/lectures/scatter-gather/Snakefile index c1b6ec4..b7d871e 100644 --- a/lectures/scatter-gather/Snakefile +++ b/lectures/scatter-gather/Snakefile @@ -3,11 +3,11 @@ import os samples = ["sample1", "sample2"] splits = 5 -scatteritems = range(1, splits+1) +scatteritems = [f"{split:03d}" for split in list(range(1, splits+1))] wildcard_constraints: - scatteritems = "\d+", - sample = "[\w\d\-\.]+" + scatteritems = "\\d+", + sample = "\\w+" rule all: input: @@ -23,24 +23,25 @@ rule scatter: conda: "envs/seqkit.yml" params: - parts = splits, + splits = splits, outdir = lambda wildcards, output: os.path.dirname(output[0]) shell: """ - seqkit split -p {params.parts} -O {params.outdir} {input} > {log} 2>&1 - rename 's/part_0*//' {params.outdir}/{wildcards.sample}.*.fastq + seqkit split --by-part-prefix {wildcards.sample}. -p {params.splits} -O {params.outdir} {input} > {log} 2>&1 """ -rule reversecomplement: +rule rc: output: "rc/{sample}/{sample}.{scatteritem}.rc.fastq" input: "splits/{sample}/{sample}.{scatteritem}.fastq" + log: + "logs/{sample}.{scatteritem}.rc.log" conda: "envs/seqkit.yml" shell: """ - seqkit seq --reverse --complement {input} > {output} + seqkit seq --seq-type DNA --reverse --complement {input} > {output} 2> {log} """ rule gather: @@ -49,4 +50,6 @@ rule gather: input: expand("rc/{{sample}}/{{sample}}.{scatteritem}.rc.fastq", scatteritem = scatteritems) shell: - "cat {input} > {output}" \ No newline at end of file + """ + cat {input} > {output} + """ diff --git a/lectures/scatter-gather/Snakefile_checkpoints b/lectures/scatter-gather/Snakefile_checkpoints new file mode 100644 index 0000000..6943f2b --- /dev/null +++ b/lectures/scatter-gather/Snakefile_checkpoints @@ -0,0 +1,61 @@ +import os +import random + +samples = ["sample1", "sample2"] + +wildcard_constraints: + scatteritems = "\\d+", + sample = "\\w+" + + +rule all: + input: + expand("{sample}.rc.fastq", sample = samples) + +checkpoint scatter: + output: + directory("splits/{sample}") + input: + "data/{sample}.fastq" + log: + "logs/{sample}.scatter.log" + conda: + "envs/seqkit.yml" + params: + splits = random.randint(1,10) + shell: + """ + seqkit split --by-part-prefix {wildcards.sample}. -p {params.splits} -O {output} {input} > {log} 2>&1 + """ + +rule rc: + output: + "rc/{sample}/{sample}.{scatteritem}.rc.fastq" + input: + "splits/{sample}/{sample}.{scatteritem}.fastq" + log: + "logs/{sample}.{scatteritem}.rc.log" + conda: + "envs/seqkit.yml" + shell: + """ + seqkit seq --seq-type DNA --reverse --complement {input} > {output} 2> {log} + """ + +def aggregate_input(wildcards): + checkpoint_output = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_output,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input + +rule gather: + output: + "{sample}.rc.fastq" + input: + aggregate_input + shell: + """ + cat {input} > {output} + """ diff --git a/lectures/scatter-gather/dag_scatter.png b/lectures/scatter-gather/dag_scatter.png deleted file mode 100644 index 13cb16f..0000000 Binary files a/lectures/scatter-gather/dag_scatter.png and /dev/null differ diff --git a/lectures/scatter-gather/envs/seqkit.yml b/lectures/scatter-gather/envs/seqkit.yml index 374bb8b..935d978 100644 --- a/lectures/scatter-gather/envs/seqkit.yml +++ b/lectures/scatter-gather/envs/seqkit.yml @@ -3,6 +3,6 @@ channels: - bioconda - conda-forge - defaults + - nanoporetech dependencies: - - seqkit - - rename \ No newline at end of file + - seqkit \ No newline at end of file diff --git a/lectures/scatter-gather/filegraph.png b/lectures/scatter-gather/filegraph.png deleted file mode 100644 index f3cea06..0000000 Binary files a/lectures/scatter-gather/filegraph.png and /dev/null differ diff --git a/lectures/scatter-gather/scatter-gather.Rmd b/lectures/scatter-gather/scatter-gather.Rmd deleted file mode 100644 index cdb8042..0000000 --- a/lectures/scatter-gather/scatter-gather.Rmd +++ /dev/null @@ -1,329 +0,0 @@ ---- -title: "Scatter/gather-operations in Snakemake" -subtitle: "Snakemake BYOC NBIS course" -date: "`r format(Sys.time(), '%d %B, %Y')`" -output: - xaringan::moon_reader: - self-contained: true - seal: false - css: ["default", "../template.css"] - nature: - slideNumberFormat: "" ---- - -layout: true - - - ---- - -class: center, middle - -.HUGE[Scatter/gather-operations] -
-.HUGE[in Snakemake] - -```{r Setup, echo = FALSE, message = FALSE} -# Knitr setup -knitr::opts_chunk$set(message = FALSE, - warning = FALSE) - -# Load packages -library("dplyr") -library("kableExtra") -``` - ---- - -# What does scatter/gather mean? - --- - -* .green[Scatter]: turn input into several pieces of output - --- - -* .green[Gather]: bring together (aggregate) results from the different pieces - - --- - -Snakemake now has built-in support for scatter/gather processes via the `scattergather` directive. Described further in the documentation: [Defining scatter-gather processes](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#defining-scatter-gather-processes). Currently not very flexible though. - ---- - -# When are scatter-gather processes handy? - --- - -- demultiplexing sequencing runs - - - multiple samples per plate - - split plates into separate files per sample - --- - -- extract reads from bam files - - - reads mapped to several genomes - - split sequences per genome - --- - -- parallelize analyses - - - _e.g._ multiple sequences per sample - - split input into smaller chunks and run analyses in parallell - -_etc_... - --- - -Between scattering and gathering there's some type of analyses performed. - ---- - -# The basics - -```python -DATASETS = ["a", "b", "c"] - -rule scatter: - output: - expand('{dataset}.txt', dataset=DATASETS) - input: - data = 'data.tar.gz' - shell: - """ - tar xvf {input} - """ - -rule uppercase: - input: - "{dataset}.txt" - output: - "{dataset}.uppercase.txt" - shell: - """ - tr [a-z] [A-Z] < {input} > {output} - """ - -rule gather: - output: - "aggregated.txt" - input: - expand("{dataset}.uppercase.txt", dataset=DATASETS) - shell: - """ - cat {input} > {output} - """ -``` --- - -```bash -snakemake -c 1 - -Job stats: -job count min threads max threads ---------- ------- ------------- ------------- -gather 1 1 1 -scatter 1 1 1 -uppercase 3 1 1 -total 5 1 1 -``` - ---- - -# The basics - -.center[] - ---- - -# Example: split files for parallelization - --- -- one fastq file per sample -``` -data -├── sample1.fastq -└── sample2.fastq -``` --- -- split into several files (scatter) -``` -splits -├── sample1 -│ ├── sample1.1.fastq -│ ├── sample1.2.fastq -│ ├── sample1.3.fastq -| ├── sample1.4.fastq -│ └── sample1.5.fastq -├── sample2 -| ├── sample2.1.fastq -| ├── sample2.2.fastq -| ├── sample2.3.fastq -| ├── sample2.4.fastq -└ └── sample2.5.fastq -``` ---- -# Example: split files for parallelization - -- process individual files (parallelization) -``` -rc -├── sample1 -│ ├── sample1.1.rc.fastq -│ ├── sample1.2.rc.fastq -│ ├── sample1.3.rc.fastq -| ├── sample1.4.rc.fastq -│ └── sample1.5.rc.fastq -├── sample2 -| ├── sample2.1.rc.fastq -| ├── sample2.2.rc.fastq -| ├── sample2.3.rc.fastq -| ├── sample2.4.rc.fastq -└ └── sample2.5.rc.fastq -``` --- -- aggregate results (gather) -``` -sample1.rc.fastq -sample2.rc.fastq -``` - ---- -# Example: split files for parallelization - -We start with defining the number of splits -```python -splits = 5 -scatteritems = range(1, splits+1] -``` - --- -Then define a rule to scatter each sample fastq -```python -rule scatter: - output: - expand("splits/{{sample}}/{{sample}}.{scatteritem}.fastq", scatteritem = scatteritems) - input: - "data/{sample}.fastq" - log: - "logs/{sample}.scatter.log" - conda: - "envs/seqkit.yml" - params: - parts = splits, - outdir = lambda wildcards, output: os.path.dirname(output[0]) - shell: - """ - seqkit split -p {params.parts} -O {params.outdir} {input} > {log} 2>&1 - rename 's/part_0*//' {params.outdir}/{wildcards.sample}.*.fastq - """ -``` - -Here `scatteritem` is not a wildcard because it is expanded using the `scatteritems` list. - ---- -# Example: split files for parallelization - -Next, a rule to do something with the split files per sample - -```python -rule reversecomplement: - output: - "rc/{sample}/{sample}.{scatteritem}.rc.fastq" - input: - "splits/{sample}/{sample}.{scatteritem}.fastq" - conda: - "envs/seqkit.yml" - shell: - """ - seqkit seq --reverse --complement {input} > {output} - """ -``` - -Here both `scatteritem` and `sample` are wildcards. The rule is generalized to work on any value for these wildcards. - ---- -# Example: split files for parallelization - -Then a rule to gather the results per sample - -```python -rule gather: - output: - "{sample}.rc.fastq" - input: - expand("rc/{{sample}}/{{sample}}.{scatteritem}.rc.fastq", scatteritem = scatteritems) - shell: - "cat {input} > {output}" -``` - -Here `scatteritem` is not a wildcard, but `sample` is. The rule can gather split files for any sample. - ---- -# Example: split files for parallelization - -Finally we put everything together, and define a pseudo rule 'all' that takes as input the gathered results for -all samples. - -```python -samples = ["sample1", "sample2"] - -splits = 5 -scatteritems = range(1, splits+1) - -rule all: - input: - expand("{sample}.rc.fastq", sample = samples) - -rule scatter: - output: - expand("splits/{{sample}}/{{sample}}.{scatteritem}.fastq", scatteritem = scatteritems) - input: - "data/{sample}.fastq" - -rule reversecomplement: - output: - "rc/{sample}/{sample}.{scatteritem}.rc.fastq" - input: - "splits/{sample}/{sample}.{scatteritem}.fastq" - -rule gather: - output: - "{sample}.rc.fastq" - input: - expand("rc/{{sample}}/{{sample}}.{scatteritem}.rc.fastq", scatteritem = scatteritems) -``` ---- -# Example: split files for parallelization - -```bash -snakemake -c 1 --use-conda - -Building DAG of jobs... -Job stats: -job count min threads max threads ------------------ ------- ------------- ------------- -all 1 1 1 -gather 2 1 1 -reversecomplement 10 1 1 -scatter 2 1 1 -total 15 1 1 -``` --- - -.center[] - --- - -This example workflow is available at the course GitHub repository: [workshop-snakemake-byoc/tree/main/lectures/scatter-gather/](https://github.com/NBISweden/workshop-snakemake-byoc/tree/master/lectures/example-workflow) - ---- - - -class: center, middle - -.HUGE[Questions?] diff --git a/lectures/scatter-gather/scatter-gather.html b/lectures/scatter-gather/scatter-gather.html index 42b3efe..d732291 100644 --- a/lectures/scatter-gather/scatter-gather.html +++ b/lectures/scatter-gather/scatter-gather.html @@ -1,425 +1,3387 @@ - - - Scatter/gather-operations in Snakemake - - - - - - - - - - - - - - + + - - + + + + + + + + + + + \ No newline at end of file diff --git a/lectures/scatter-gather/scatter-gather.pdf b/lectures/scatter-gather/scatter-gather.pdf deleted file mode 100644 index 68a923c..0000000 Binary files a/lectures/scatter-gather/scatter-gather.pdf and /dev/null differ diff --git a/lectures/scatter-gather/scatter-gather.qmd b/lectures/scatter-gather/scatter-gather.qmd new file mode 100644 index 0000000..3f8d5aa --- /dev/null +++ b/lectures/scatter-gather/scatter-gather.qmd @@ -0,0 +1,579 @@ +--- +title: "Scatter/gather-operations in Snakemake" +subtitle: "Snakemake BYOC NBIS course" +date: 2024-05-27 +format: + revealjs: + theme: + - white + - ../custom.scss + embed-resources: true + toc: false + toc-depth: 1 + slide-level: 2 + slide-number: true + #preview-links: true + #chalkboard: true + # Multiple logos not possible; would need to make custom logo combining both logos + footer: Snakemake BYOC 2024 - Reproducible Research + logo: https://nbis.se/nbislogo-green.svg + smaller: true + highlight-style: gruvbox +--- + + +```{r Setup, echo = FALSE, message = FALSE} +# Knitr setup +knitr::opts_chunk$set(message = FALSE, + warning = FALSE) + +# Load packages +library("dplyr") +library("kableExtra") +``` + +## What does scatter/gather mean? + +:::{.fragment} +**Scatter**: turn input into several pieces of output +::: + +:::{.fragment} +**Gather**: bring together (aggregate) results from the different pieces +::: + +:::{.fragment} + +Snakemake now has built-in support for scatter/gather processes via the `scattergather` directive. Described further in the documentation: [Defining scatter-gather processes](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#defining-scatter-gather-processes). Currently not very flexible though. +::: + +## When are scatter-gather processes handy? + +:::{.incremental} +- demultiplexing sequencing runs + + - multiple samples per plate + - split plates into separate files per sample + +- extract reads from bam files + + - reads mapped to several genomes + - split sequences per genome + +- parallelize analyses + + - split input into smaller chunks and run analyses in parallell + +::: + + +## A basic example + +```{.python code-line-numbers="|1|3-11|13-21|23-31"} +DATASETS = ["a", "b", "c"] + +rule scatter: + output: + expand('{dataset}.txt', dataset=DATASETS) + input: + data = 'data.tar.gz' + shell: + """ + tar xvf {input} + """ + +rule uppercase: + input: + "{dataset}.txt" + output: + "{dataset}.uppercase.txt" + shell: + """ + tr [a-z] [A-Z] < {input} > {output} + """ + +rule gather: + output: + "aggregated.txt" + input: + expand("{dataset}.uppercase.txt", dataset=DATASETS) + shell: + """ + cat {input} > {output} + """ +``` + +## Filegraph + +```{dot} +digraph snakemake_dag { + graph[bgcolor=white, margin=0]; + node[shape=box, style=rounded, fontname=sans, fontsize=10, penwidth=2]; + edge[penwidth=2, color=grey]; +0 [ shape=none, margin=0, label=< + +
+ + + + + + + + + + +
+ + + +
+gather +
↪ input
a.uppercase.txt
b.uppercase.txt
c.uppercase.txt
output →
aggregated.txt
>] +1 [ shape=none, margin=0, label=< + +
+ + + + +
+ + + +
+uppercase +
↪ input
{dataset}.txt
output →
{dataset}.uppercase.txt
>] +2 [ shape=none, margin=0, label=< + +
+ + + + +
+ + + + + + + +
+scatter +
↪ input
data.tar.gz
output →
a.txt
b.txt
c.txt
>] + 1 -> 0 + 2 -> 1 +} +``` + +## Another example: splitting files for parallelization + +## Splitting files for parallelization {auto-animate="true" auto-animate-easing=None} + +::: {data-id="files"} +- one fastq file per sample +``` +data +├── sample1.fastq +└── sample2.fastq +``` +::: + +## Splitting files for parallelization {auto-animate="true" auto-animate-easing=None} + +- split into several files (scatter) + +::: {data-id="files"} +``` +splits +├── sample1 +│ ├── sample1.001.fastq +│ ├── sample1.002.fastq +│ ├── sample1.003.fastq +| ├── sample1.004.fastq +│ └── sample1.005.fastq +├── sample2 +| ├── sample2.001.fastq +| ├── sample2.002.fastq +| ├── sample2.003.fastq +| ├── sample2.004.fastq +└ └── sample2.005.fastq +``` +::: + +## Splitting files for parallelization {auto-animate="true" auto-animate-easing=None} + +- process individual files (parallelization) + +::: {data-id="files"} +``` +rc +├── sample1 +│ ├── sample1.001.rc.fastq +│ ├── sample1.002.rc.fastq +│ ├── sample1.003.rc.fastq +| ├── sample1.004.rc.fastq +│ └── sample1.005.rc.fastq +├── sample2 +| ├── sample2.001.rc.fastq +| ├── sample2.002.rc.fastq +| ├── sample2.003.rc.fastq +| ├── sample2.004.rc.fastq +└ └── sample2.005.rc.fastq +``` +::: + +## Splitting files for parallelization {auto-animate="true" auto-animate-easing=None} + +- aggregate results (gather) + +::: {data-id="files"} +``` +├── sample1.rc.fastq +└── sample2.rc.fastq +``` +::: + +## Splitting files for parallelization + +We start with defining the number of splits + +```{python, echo=TRUE} +splits = 5 +scatteritems = [f"{split:03d}" for split in list(range(1, splits+1))] +scatteritems +``` + +## Splitting files for parallelization + +We also impose some constraints on the wildcards: + +```{.python} +wildcard_constraints: + scatteritems = "\\d+", + sample = "\\w+" +``` + +Here, scatteritems can be any number of digits, and sample can be any number of word characters (`[a-zA-Z0-9_]`). + +## Splitting files for parallelization + +Then define a rule to scatter each sample fastq + +```{python code=readLines("Snakefile")[15:31]} +#| echo: true +#| eval: false +``` + +Here `scatteritem` is not a wildcard because it is expanded using the `scatteritems` list. + +## Splitting files for parallelization + +Next, a rule to do something with the split files per sample + +```{python code=readLines("Snakefile")[32:45]} +#| echo: true +#| eval: false +``` + +Here both `scatteritem` and `sample` are wildcards. The rule is generalized to work on any value for these wildcards. + +## Splitting files for parallelization + +Then a rule to gather the results per sample + +```{python code=readLines("Snakefile")[46:55]} +#| echo: true +#| eval: false +``` + +Here `scatteritem` is not a wildcard, but `sample` is. The rule can gather split files for any sample. + +## Splitting files for parallelization + +Finally we put everything together, and define a pseudo rule 'all' that takes as input the gathered results for +all samples. + +```{python code=readLines("Snakefile")[11:15]} +#| echo: true +#| eval: false +``` + +## + +```{dot} +digraph snakemake_dag { + graph[bgcolor=white, margin=0]; + node[shape=box, style=rounded, fontname=sans, fontsize=24, penwidth=4]; + edge[penwidth=2, color=grey]; + 0[label = "all", color = "0.00 0.6 0.85", style="rounded"]; + 1[label = "gather", color = "0.50 0.6 0.85", style="rounded"]; + 2[label = "rc\nscatteritem: 001", color = "0.33 0.6 0.85", style="rounded"]; + 3[label = "scatter\nsample: sample1", color = "0.17 0.6 0.85", style="rounded"]; + 4[label = "rc\nscatteritem: 002", color = "0.33 0.6 0.85", style="rounded"]; + 5[label = "rc\nscatteritem: 003", color = "0.33 0.6 0.85", style="rounded"]; + 6[label = "rc\nscatteritem: 004", color = "0.33 0.6 0.85", style="rounded"]; + 7[label = "rc\nscatteritem: 005", color = "0.33 0.6 0.85", style="rounded"]; + 8[label = "gather", color = "0.50 0.6 0.85", style="rounded"]; + 9[label = "rc\nscatteritem: 001", color = "0.33 0.6 0.85", style="rounded"]; + 10[label = "scatter\nsample: sample2", color = "0.17 0.6 0.85", style="rounded"]; + 11[label = "rc\nscatteritem: 002", color = "0.33 0.6 0.85", style="rounded"]; + 12[label = "rc\nscatteritem: 003", color = "0.33 0.6 0.85", style="rounded"]; + 13[label = "rc\nscatteritem: 004", color = "0.33 0.6 0.85", style="rounded"]; + 14[label = "rc\nscatteritem: 005", color = "0.33 0.6 0.85", style="rounded"]; + 1 -> 0 + 8 -> 0 + 2 -> 1 + 4 -> 1 + 5 -> 1 + 6 -> 1 + 7 -> 1 + 3 -> 2 + 3 -> 4 + 3 -> 5 + 3 -> 6 + 3 -> 7 + 9 -> 8 + 11 -> 8 + 12 -> 8 + 13 -> 8 + 14 -> 8 + 10 -> 9 + 10 -> 11 + 10 -> 12 + 10 -> 13 + 10 -> 14 +} +``` + +This example workflow is available at the course GitHub repository: [workshop-snakemake-byoc/tree/main/lectures/scatter-gather/Snakefile](https://github.com/NBISweden/workshop-snakemake-byoc/tree/main/lectures/scatter-gather/Snakefile) + +# Dynamic output + +# Data-dependent conditional execution {transition="slide" transition-speed="slow"} + +# Checkpoints {transition="slide" transition-speed="slow"} + +## Checkpoints + +If the output of a rule is not known in advance, Snakemake can re-evaluate the workflow using **checkpoints**. + +:::{.fragment} +Several use-cases, _e.g._ clustering into an unknown number of clusters. +::: + +:::{.fragment} +Let's try this with the previous example by implementing a random number of splits. +::: + +## Checkpoints {auto-animate="true"} + +Before: number of splits defined ahead of time + +```{python code=readLines("Snakefile")[0:15]} +#| echo: true +#| eval: false +``` + +## Checkpoints {auto-animate="true"} + +Now: Number of splits will be random + +```{python code=readLines("Snakefile_checkpoints")[0:14]} +#| echo: true +#| eval: false +``` + +## Checkpoints {auto-animate="true" auto-animate-restart="true"} + +Before: The scatter rule expanded the output files + +```{.python code-line-numbers="3,11"} +rule scatter: + output: + expand("splits/{{sample}}/{{sample}}.{scatteritem}.fastq", scatteritem = scatteritems) + input: + "data/{sample}.fastq" + log: + "logs/{sample}.scatter.log" + conda: + "envs/seqkit.yml" + params: + splits = splits, + outdir = lambda wildcards, output: os.path.dirname(output[0]) + shell: + """ + seqkit split --by-part-prefix {wildcards.sample}. -p {params.splits} -O {params.outdir} {input} > {log} 2>&1 + """ +``` + +## Checkpoints {auto-animate="true"} + +Now: The scatter rule becomes a checkpoint with unknown number of output files + +```{.python code-line-numbers="3,11"} +checkpoint scatter: + output: + directory("splits/{sample}") + input: + "data/{sample}.fastq" + log: + "logs/{sample}.scatter.log" + conda: + "envs/seqkit.yml" + params: + splits = random.randint(1,10) + shell: + """ + seqkit split --by-part-prefix {wildcards.sample}. -p {params.splits} -O {output} {input} > {log} 2>&1 + """ +``` + +## Checkpoints + +The `rc` rule is left unchanged + +```{python code=readLines("Snakefile")[32:45]} +#| echo: true +#| eval: false +``` + +## Checkpoints {auto-animate="true"} + +Before: The gather rule expanded the input files + +```{.python code-line-numbers="5"} +rule gather: + output: + "{sample}.rc.fastq" + input: + expand("rc/{{sample}}/{{sample}}.{scatteritem}.rc.fastq", scatteritem = scatteritems) + shell: + """ + cat {input} > {output} + """ +``` + +## Checkpoints {auto-animate="true"} + +Now: we use an input function and the built-in `glob_wildcards` + +```{.python code-line-numbers="1-7,13"} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input + +rule gather: + output: + "{sample}.rc.fastq" + input: + aggregate_input + shell: + """ + cat {input} > {output} + """ +``` + +## Checkpoints {auto-animate="true"} + +```{.python code-line-numbers="2"} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input +``` + +- Get the output directory of the scatter checkpoint for the sample (`checkpoint_outdir='splits/sample1'`) + +## Checkpoints {auto-animate="true"} + +```{.python code-line-numbers="3" code=} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input +``` + +- Use `glob_wildcards` to infer the scatteritem wildcard based on existing files +- If `splits=3`, `scatteritems=["001", "002", "003"]` + +## Checkpoints {auto-animate="true"} + +```{.python code-line-numbers="4"} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input +``` +- The sample wildcard is known (`sample='sample1'`) + +## Checkpoints {auto-animate="true"} + +```{.python code-line-numbers="6"} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input +``` + +- `scatteritem` is expanded using the inferred `scatteritems` list + +## Checkpoints {auto-animate="true" .code.sstree} + +```{.python code-line-numbers="7"} +def aggregate_input(wildcards): + checkpoint_outdir = checkpoints.scatter.get(sample=wildcards.sample).output[0] + scatteritems = glob_wildcards(os.path.join(checkpoint_outdir,"{sample}.{scatteritem}.fastq")).scatteritem + input = expand("rc/{sample}/{sample}.{scatteritem}.rc.fastq", + sample=wildcards.sample, + scatteritem=scatteritems) + return input +``` + +- returned input becomes: + +```{.python code-line-numbers="false"} +["rc/sample1/sample1.001.rc.fastq", +"rc/sample1/sample1.002.rc.fastq", +"rc/sample1/sample1.003.rc.fastq"] +``` + +## + +```{dot} +digraph snakemake_dag { + graph[bgcolor=white, margin=0]; + node[shape=box, style=rounded, fontname=sans, fontsize=10, penwidth=2]; + edge[penwidth=2, color=grey]; + 0[label = "all", color = "0.33 0.6 0.85", style="rounded"]; + 1[label = "gather", color = "0.50 0.6 0.85", style="rounded"]; + 2[label = "scatter\nsample: sample1", color = "0.00 0.6 0.85", style="rounded"]; + 3[label = "gather", color = "0.50 0.6 0.85", style="rounded"]; + 4[label = "scatter\nsample: sample2", color = "0.00 0.6 0.85", style="rounded"]; + 1 -> 0 + 3 -> 0 + 2 -> 1 + 4 -> 3 +} +``` + +This example workflow is available at the course GitHub repository: [workshop-snakemake-byoc/tree/main/lectures/scatter-gather/Snakefile_checkpoints](https://github.com/NBISweden/workshop-snakemake-byoc/tree/main/lectures/scatter-gather/Snakefile_checkpoints) + +## Questions? \ No newline at end of file diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/_extension.yml b/lectures/welcome/_extensions/percyfal/nbis-course-logo/_extension.yml new file mode 100644 index 0000000..41dcd9a --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/_extension.yml @@ -0,0 +1,18 @@ +title: Nbiscourselogo +author: Per Unneberg +version: 1.0.0 +quarto-required: ">=1.4.0" +contributes: + formats: + html: + toc: true + theme: + - html.scss + revealjs: + theme: + - revealjs.scss + logo: logo.svg + institute: NBIS + filters: + - filters/callout.lua + - filters/button.lua diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/button.lua b/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/button.lua new file mode 100644 index 0000000..ed05af4 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/button.lua @@ -0,0 +1,10 @@ +function Div(div) + -- process exercise + if div.classes:includes("nbisbtn") then + if div.c[1].t == "Para" then + local newcontent = div.c[1].c + div.content = newcontent + end + end + return div +end diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/callout.lua b/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/callout.lua new file mode 100644 index 0000000..124d183 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/filters/callout.lua @@ -0,0 +1,61 @@ +-- callout.lua +-- +-- Custom filter to convert divs with class "callout" to callout +-- blocks for special callout classes exercise, answer, and hint +-- +local callout_attrs = { + exercise = { + type = "exercise", + collapse = false, + }, + answer = { + type = "answer", + collapse = true + }, + hint = { + type = "hint", + collapse = true + } +} + +local function isCallout(class) + return class == 'callout' or class:match("^callout%-") +end + +local function calloutType(div) + for _, class in ipairs(div.attr.classes) do + if isCallout(class) then + local type = class:match("^callout%-(.*)") + if type == nil then + type = "none" + end + return type + end + end + return nil +end + + +-- function Div +-- +-- Format div callout blocks. Only for html classes +function Div(div) + local title + local callout_type = calloutType(div) + if callout_type == nil then + return div + end + if div.content[1] ~= nil and div.content[1].t == "Header" then + title = pandoc.utils.stringify(div.content[1]) + div.content:remove(1) + else + title = callout_type:gsub("^%l", string.upper) + end + + return quarto.Callout({ + type = callout_type, + content = { pandoc.Div(div) }, + title = title, + collapse = callout_attrs[callout_type].collapse + }) +end diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/html.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/html.scss new file mode 100644 index 0000000..b8b0659 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/html.scss @@ -0,0 +1,29 @@ +/*-- scss:uses --*/ +@use 'templates/colors' as cl; +@use 'templates/callout' as co; +@use 'templates/course'; +@import 'templates/nbis'; + +/*-- scss:variables --*/ + +/*-- scss:rules --*/ + +/* sizes */ +.largest { + font-size: 1.953em; +} +.larger { + font-size: 1.563em; +} +.large { + font-size: 1.25em; +} +.small { + font-size: 0.8em; +} +.smallr { + font-size: 0.64em; +} +.smallest { + font-size: 0.512em; +} diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/logo.svg b/lectures/welcome/_extensions/percyfal/nbis-course-logo/logo.svg new file mode 100644 index 0000000..9f86ab2 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/logo.svg @@ -0,0 +1 @@ + diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/revealjs.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/revealjs.scss new file mode 100644 index 0000000..9cd1d30 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/revealjs.scss @@ -0,0 +1,120 @@ +/*-- scss:uses --*/ +@use 'templates/colors' as cl; +@use 'templates/callout' as co; +@use 'templates/course'; +@import 'templates/nbis'; + +/*-- scss:defaults --*/ +$presentation-slide-text-align: left !default; + +/*-- scss:variables --*/ +/* presentation */ +$presentation-font-size-root: 28px; +$presentation-line-height: 1.3; +$presentation-h1-font-size: 1.802em; +$presentation-h2-font-size: 1.602em; +$presentation-h3-font-size: 1.424em; +$presentation-h4-font-size: 1.266em; +$presentation-h5-font-size: 1.125em; +$presentation-h6-font-size: 1em; +$code-block-font-size: 0.95em; + +/*-- scss:mixins --*/ +.reveal .slides section[data-vertical-align-top] { + top: 0 !important; +} + +.reveal strong { + color: cl.$lime; + font-weight: bold; +} + +.reveal blockquote { + color: cl.$grape75 !important; + border-left: 0.25rem solid cl.$grape25 !important; +} + +.reveal .compact > p { + margin-top: 0px; + margin-bottom: 0px; + padding-top: 0px; + padding-bottom: 0px; +} + +.reveal .compact img { + margin-top: 0px; + margin-bottom: 0px; + padding-top: 0px; + padding-bottom: 0px; +} + +.reveal .panel-tabset [role="tab"][aria-selected="true"] { + border: lightgray 2px solid; + border-radius: 6px 6px 0px 0px; + border-bottom: none; +} + +.reveal .panel-tabset .tab-content { + border-top: lightgray 2px solid; +} + +.reveal pre { + font-size: 0.92em; + line-height: 1.3; +} + +.reveal pre code { + background-color: #f5f5f5 !important; + border-radius: 6px; +} + +.reveal code { + font-size: 0.9em; + padding-left: 4px; + padding-right: 4px; +} + +.reveal div.sourceCode { + border: 0 !important; + border-radius: 6px; + margin-bottom: 0.6em !important; +} + +.reveal .sourceCode span.in { + background-color: $code-block-bg; +} + +// .reveal .callout.callout-captioned .callout-icon::before { +// margin-top: 0.5rem; +// scale: 0.8; +// } + +// .callout-answer > .callout-header::before { +// font-family: "Font Awesome 5 Free"; +// content: "\f002 "; +// font-weight: 900; +// margin-right: 10px; +// } + + +/* sizes */ +.largest { + font-size: 1.953em; +} +.larger { + font-size: 1.563em; +} +.large { + font-size: 1.25em; +} +.small { + font-size: 0.8em; +} +.smallr { + font-size: 0.64em; +} +.smallest { + font-size: 0.512em; +} + +/*-- scss:end --*/ diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_callout.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_callout.scss new file mode 100644 index 0000000..2daa36b --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_callout.scss @@ -0,0 +1,66 @@ +/*-- scss:defaults --*/ +// Custom callout styles for exercises, answers, and hints +/*-- scss:uses --*/ +@use 'colors'; +@import url("https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.2/css/all.min.css"); + + +/*-- scss:variables --*/ +$callout-color-exercise: colors.$lime; +$callout-color-exercise-bg: colors.$lime25; +$callout-color-answer: colors.$grape; +$callout-color-answer-bg: colors.$grape25; +$callout-color-hint: colors.$aqua; +$callout-color-hint-bg: colors.$aqua25; + + +/*-- scss:rules --*/ + +// Exercise callout styling +div.callout-exercise { + border-left-color: $callout-color-exercise !important; +} + +div.callout-exercise.callout-style-default > .callout-header { + background-color: $callout-color-exercise-bg !important; +} + +.callout-exercise > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f303"; + font-weight: 900; + margin-right: 10px; +} + +// Answer callout styling +div.callout-answer { + border-left-color: $callout-color-answer !important; +} + +div.callout-answer.callout-style-default > .callout-header { + background-color: $callout-color-answer-bg !important; +} + +.callout-answer > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f002 "; + font-weight: 900; + margin-right: 10px; +} + + +// Hint callout styling +div.callout-hint { + border-left-color: $callout-color-hint !important; +} + +div.callout-hint.callout-style-default > .callout-header { + background-color: $callout-color-hint-bg !important; +} + +.callout-hint > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f12e"; + font-weight: 900; + margin-right: 10px; +} diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_colors.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_colors.scss new file mode 100644 index 0000000..29c2ef2 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_colors.scss @@ -0,0 +1,25 @@ +// SciLife color scheme: see https://www.scilifelab.se/community-pages/resources/visual-identity/ +$lime: #a7c947 !default; +$lime25: #e9f2d1 !default; +$lime50: #d3e4a3 !default; +$lime75: #bdd775 !default; + +$teal: #045c64 !default; +$teal25: #c0d6d8 !default; +$teal50: #82aeb2 !default; +$teal75: #43858b !default; + +$aqua: #4c979f !default; +$aqua25: #d2e5e7 !default; +$aqua50: #a6cbcf !default; +$aqua75: #79b1b7 !default; + +$grape: #491f53 !default; +$grape25: #d2c7d4 !default; +$grape50: #a48fa9 !default; +$grape75: #77577e !default; + +$lightgray: #e5e5e5 !default; +$mediumgray: #a6a6a6 !default; +$darkgray: #3f3f3f !default; +$black: #202020 !default; diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_course.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_course.scss new file mode 100644 index 0000000..ee68835 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_course.scss @@ -0,0 +1,114 @@ +/*-- scss:use --*/ +@use 'colors' as cl; +@use 'nbis' as nb; + +/*-- scss:rules --*/ + +.organisers-parent { + display: grid; + gap: 1em; + grid-template-columns: repeat(5, 1fr); +} + +.organisers-child { + text-align: center; +} + +.organisers-child img { + height: 150px; + width: 150px; + object-fit: cover; + border-radius: 50%; +} + +.circular-image img { + height: 150px; + width: 150px; + object-fit: cover; + border-radius: 90%; +} +/* link colour */ +a, table a{ + color: nb.$link-color; + text-decoration: underline; +} + +a:hover, table a:hover { + color: var(--color-primary); + text-decoration: underline; +} + +a:focus, table a:focus{ + color: var(--color-primary-dark); + text-decoration: underline; +} + +/* table even shading colour */ +.even { + background-color: var(--color-code-bg); +} + +/* schedule table */ +.table-schedule { + display: inline-block; + padding: 25px; + border-radius: 4px; + -webkit-box-shadow: 1px 1px 6px var(--color-shadow); + -moz-box-shadow: 1px 1px 6px var(--color-shadow); + box-shadow: 1px 1px 6px var(--color-shadow); + margin-bottom: 20px; +} + +.table-schedule td, .table-schedule th { + border: none !important; +} + +// Table entries +.topic { + margin-right: 5px; +} +.topic-highlight { + color:white; + font-weight:bold; + border-radius:4px; + padding:4px; + background: cl.$grape25; +} +.day-highlight { + color:white; + border-radius:4px; + padding-right:4px; + padding-left:4px; + background: cl.$lime; + margin-left:9px; + margin-right:7px; +} +.table-date { + font-size:110%; + font-weight: bold; + padding-top:10px; + padding-bottom:5px; + margin-top:10px; +} + +@media only screen and (max-width: 575px) { + .landing { + grid-template-columns: 100%; + text-align: center; + } + + .landing-update { + text-align: center; + } + + .landing-right { + padding: 2em; + } +} +@media print { + .logo-button { + display: none!important; + } +} + +/*-- scss:end --*/ diff --git a/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_nbis.scss b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_nbis.scss new file mode 100644 index 0000000..3890f23 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course-logo/templates/_nbis.scss @@ -0,0 +1,140 @@ +/*-- scss:uses --*/ +@use 'colors' as cl; +@use 'callout' as co; +@import url("https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Nunito:ital,wght@0,400;0,600;1,400;1,600&display=swap"); + + +/*-- scss:defaults --*/ +// Description: NBIS theme +$primary: cl.$lime; +$secondary: cl.$lime75; +$link-color: cl.$teal50; +$body-color: cl.$black; +$code-block-bg: WhiteSmoke; + +$font-family-sans-serif: "Nunito", "Source Sans Pro", "Helvetica", "sans-serif"; +$font-family-monospace: "JetBrains Mono", "Ubuntu Mono", "Lucida Console", + "Courier", "monospace"; + + +$code-block-border-color: none !default; +$code-bg: $code-block-bg; /* inline code */ + +/*-- scss:variables --*/ +$font-size-root: 18px; +$code-font-size: 14px; +/* major second scale */ +$h1-font-size: 1.802rem; +$h2-font-size: 1.602rem; +$h3-font-size: 1.424rem; +$h4-font-size: 1.266rem; +$h5-font-size: 1.125rem; +$h6-font-size: 1rem; +$code-block-font-size: 0.95rem; + +/* site */ +$navbar-bg: white; +$footer-bg: white; +$footer-fg: DimGray; + +/*-- scss:rules --*/ +:root { + --color-primary: cl.$lime75; + --color-primary-dark: cl.$lime; +} + +.highlight { + background-color: $secondary; + color: white; + font-size: 0.85em; + padding: 0.2em 0.3em; + border-radius: 4px; + vertical-align: baseline; + font-weight: bold; +} + +pre.out { + background-color: cl.$lime; +} +pre.sourceCode.src { + background-color: cl.$teal25; +} + +.label { + color: cl.$lime; + font-weight: bold; + background-color: white; + font-size: 1em; +} + +code { + color: cl.$grape; + font-weight: bold; + background-color: white; +} + +code.tree { + line-height: 8px; + font-size: 18px; +} + +code.stree { + line-height: 6.6px; + font-size: 12px; +} + +code.sstree { + line-height: 5.6px; + font-size: 10px; +} + +code.large { + font-size: 20px; +} + +.citation { + font-size: 1em; +} + +.flushright { + float: right; +} + +.translatey50 { + transform: translate(0, -50px); +} + +.translatey100 { + transform: translate(0, -100px); +} + +.center { + margin: auto; +} + +.graynote { + color: cl.$mediumgray; + font-size: 0.7em; +} + +dt { + font-weight: bold; + color: cl.$grape; +} + +.software { + font-weight: bold; + font-style: italic; + color: cl.$teal50; +} + +.dna { + font-weight: normal; + font-family: monospace; +} + +.dna .italic { + font-weight: bold; + font-family: monospace; + color: cl.$aqua75; +} diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/_extension.yml b/lectures/welcome/_extensions/percyfal/nbis-course/_extension.yml new file mode 100644 index 0000000..65f4632 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/_extension.yml @@ -0,0 +1,23 @@ +title: Nbiscourse +author: Per Unneberg +version: 1.0.0 +quarto-required: ">=1.4.0" +contributes: + formats: + html: + toc: true + theme: + - html.scss + revealjs: + theme: + - revealjs.scss + title-slide-attributes: + data-background-image: logo.svg + data-background-size: 500px + data-background-position: top left + template-partials: + - partials/title-slide.html + institute: NBIS + filters: + - filters/callout.lua + - filters/button.lua diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/filters/button.lua b/lectures/welcome/_extensions/percyfal/nbis-course/filters/button.lua new file mode 100644 index 0000000..ed05af4 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/filters/button.lua @@ -0,0 +1,10 @@ +function Div(div) + -- process exercise + if div.classes:includes("nbisbtn") then + if div.c[1].t == "Para" then + local newcontent = div.c[1].c + div.content = newcontent + end + end + return div +end diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/filters/callout.lua b/lectures/welcome/_extensions/percyfal/nbis-course/filters/callout.lua new file mode 100644 index 0000000..124d183 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/filters/callout.lua @@ -0,0 +1,61 @@ +-- callout.lua +-- +-- Custom filter to convert divs with class "callout" to callout +-- blocks for special callout classes exercise, answer, and hint +-- +local callout_attrs = { + exercise = { + type = "exercise", + collapse = false, + }, + answer = { + type = "answer", + collapse = true + }, + hint = { + type = "hint", + collapse = true + } +} + +local function isCallout(class) + return class == 'callout' or class:match("^callout%-") +end + +local function calloutType(div) + for _, class in ipairs(div.attr.classes) do + if isCallout(class) then + local type = class:match("^callout%-(.*)") + if type == nil then + type = "none" + end + return type + end + end + return nil +end + + +-- function Div +-- +-- Format div callout blocks. Only for html classes +function Div(div) + local title + local callout_type = calloutType(div) + if callout_type == nil then + return div + end + if div.content[1] ~= nil and div.content[1].t == "Header" then + title = pandoc.utils.stringify(div.content[1]) + div.content:remove(1) + else + title = callout_type:gsub("^%l", string.upper) + end + + return quarto.Callout({ + type = callout_type, + content = { pandoc.Div(div) }, + title = title, + collapse = callout_attrs[callout_type].collapse + }) +end diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/html.scss b/lectures/welcome/_extensions/percyfal/nbis-course/html.scss new file mode 100644 index 0000000..b8b0659 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/html.scss @@ -0,0 +1,29 @@ +/*-- scss:uses --*/ +@use 'templates/colors' as cl; +@use 'templates/callout' as co; +@use 'templates/course'; +@import 'templates/nbis'; + +/*-- scss:variables --*/ + +/*-- scss:rules --*/ + +/* sizes */ +.largest { + font-size: 1.953em; +} +.larger { + font-size: 1.563em; +} +.large { + font-size: 1.25em; +} +.small { + font-size: 0.8em; +} +.smallr { + font-size: 0.64em; +} +.smallest { + font-size: 0.512em; +} diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/logo.svg b/lectures/welcome/_extensions/percyfal/nbis-course/logo.svg new file mode 100644 index 0000000..9f86ab2 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/logo.svg @@ -0,0 +1 @@ + diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/revealjs.scss b/lectures/welcome/_extensions/percyfal/nbis-course/revealjs.scss new file mode 100644 index 0000000..9cd1d30 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/revealjs.scss @@ -0,0 +1,120 @@ +/*-- scss:uses --*/ +@use 'templates/colors' as cl; +@use 'templates/callout' as co; +@use 'templates/course'; +@import 'templates/nbis'; + +/*-- scss:defaults --*/ +$presentation-slide-text-align: left !default; + +/*-- scss:variables --*/ +/* presentation */ +$presentation-font-size-root: 28px; +$presentation-line-height: 1.3; +$presentation-h1-font-size: 1.802em; +$presentation-h2-font-size: 1.602em; +$presentation-h3-font-size: 1.424em; +$presentation-h4-font-size: 1.266em; +$presentation-h5-font-size: 1.125em; +$presentation-h6-font-size: 1em; +$code-block-font-size: 0.95em; + +/*-- scss:mixins --*/ +.reveal .slides section[data-vertical-align-top] { + top: 0 !important; +} + +.reveal strong { + color: cl.$lime; + font-weight: bold; +} + +.reveal blockquote { + color: cl.$grape75 !important; + border-left: 0.25rem solid cl.$grape25 !important; +} + +.reveal .compact > p { + margin-top: 0px; + margin-bottom: 0px; + padding-top: 0px; + padding-bottom: 0px; +} + +.reveal .compact img { + margin-top: 0px; + margin-bottom: 0px; + padding-top: 0px; + padding-bottom: 0px; +} + +.reveal .panel-tabset [role="tab"][aria-selected="true"] { + border: lightgray 2px solid; + border-radius: 6px 6px 0px 0px; + border-bottom: none; +} + +.reveal .panel-tabset .tab-content { + border-top: lightgray 2px solid; +} + +.reveal pre { + font-size: 0.92em; + line-height: 1.3; +} + +.reveal pre code { + background-color: #f5f5f5 !important; + border-radius: 6px; +} + +.reveal code { + font-size: 0.9em; + padding-left: 4px; + padding-right: 4px; +} + +.reveal div.sourceCode { + border: 0 !important; + border-radius: 6px; + margin-bottom: 0.6em !important; +} + +.reveal .sourceCode span.in { + background-color: $code-block-bg; +} + +// .reveal .callout.callout-captioned .callout-icon::before { +// margin-top: 0.5rem; +// scale: 0.8; +// } + +// .callout-answer > .callout-header::before { +// font-family: "Font Awesome 5 Free"; +// content: "\f002 "; +// font-weight: 900; +// margin-right: 10px; +// } + + +/* sizes */ +.largest { + font-size: 1.953em; +} +.larger { + font-size: 1.563em; +} +.large { + font-size: 1.25em; +} +.small { + font-size: 0.8em; +} +.smallr { + font-size: 0.64em; +} +.smallest { + font-size: 0.512em; +} + +/*-- scss:end --*/ diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/templates/_callout.scss b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_callout.scss new file mode 100644 index 0000000..2daa36b --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_callout.scss @@ -0,0 +1,66 @@ +/*-- scss:defaults --*/ +// Custom callout styles for exercises, answers, and hints +/*-- scss:uses --*/ +@use 'colors'; +@import url("https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.2/css/all.min.css"); + + +/*-- scss:variables --*/ +$callout-color-exercise: colors.$lime; +$callout-color-exercise-bg: colors.$lime25; +$callout-color-answer: colors.$grape; +$callout-color-answer-bg: colors.$grape25; +$callout-color-hint: colors.$aqua; +$callout-color-hint-bg: colors.$aqua25; + + +/*-- scss:rules --*/ + +// Exercise callout styling +div.callout-exercise { + border-left-color: $callout-color-exercise !important; +} + +div.callout-exercise.callout-style-default > .callout-header { + background-color: $callout-color-exercise-bg !important; +} + +.callout-exercise > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f303"; + font-weight: 900; + margin-right: 10px; +} + +// Answer callout styling +div.callout-answer { + border-left-color: $callout-color-answer !important; +} + +div.callout-answer.callout-style-default > .callout-header { + background-color: $callout-color-answer-bg !important; +} + +.callout-answer > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f002 "; + font-weight: 900; + margin-right: 10px; +} + + +// Hint callout styling +div.callout-hint { + border-left-color: $callout-color-hint !important; +} + +div.callout-hint.callout-style-default > .callout-header { + background-color: $callout-color-hint-bg !important; +} + +.callout-hint > .callout-header::before { + font-family: "Font Awesome 5 Free"; + content: "\f12e"; + font-weight: 900; + margin-right: 10px; +} diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/templates/_colors.scss b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_colors.scss new file mode 100644 index 0000000..29c2ef2 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_colors.scss @@ -0,0 +1,25 @@ +// SciLife color scheme: see https://www.scilifelab.se/community-pages/resources/visual-identity/ +$lime: #a7c947 !default; +$lime25: #e9f2d1 !default; +$lime50: #d3e4a3 !default; +$lime75: #bdd775 !default; + +$teal: #045c64 !default; +$teal25: #c0d6d8 !default; +$teal50: #82aeb2 !default; +$teal75: #43858b !default; + +$aqua: #4c979f !default; +$aqua25: #d2e5e7 !default; +$aqua50: #a6cbcf !default; +$aqua75: #79b1b7 !default; + +$grape: #491f53 !default; +$grape25: #d2c7d4 !default; +$grape50: #a48fa9 !default; +$grape75: #77577e !default; + +$lightgray: #e5e5e5 !default; +$mediumgray: #a6a6a6 !default; +$darkgray: #3f3f3f !default; +$black: #202020 !default; diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/templates/_course.scss b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_course.scss new file mode 100644 index 0000000..ee68835 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_course.scss @@ -0,0 +1,114 @@ +/*-- scss:use --*/ +@use 'colors' as cl; +@use 'nbis' as nb; + +/*-- scss:rules --*/ + +.organisers-parent { + display: grid; + gap: 1em; + grid-template-columns: repeat(5, 1fr); +} + +.organisers-child { + text-align: center; +} + +.organisers-child img { + height: 150px; + width: 150px; + object-fit: cover; + border-radius: 50%; +} + +.circular-image img { + height: 150px; + width: 150px; + object-fit: cover; + border-radius: 90%; +} +/* link colour */ +a, table a{ + color: nb.$link-color; + text-decoration: underline; +} + +a:hover, table a:hover { + color: var(--color-primary); + text-decoration: underline; +} + +a:focus, table a:focus{ + color: var(--color-primary-dark); + text-decoration: underline; +} + +/* table even shading colour */ +.even { + background-color: var(--color-code-bg); +} + +/* schedule table */ +.table-schedule { + display: inline-block; + padding: 25px; + border-radius: 4px; + -webkit-box-shadow: 1px 1px 6px var(--color-shadow); + -moz-box-shadow: 1px 1px 6px var(--color-shadow); + box-shadow: 1px 1px 6px var(--color-shadow); + margin-bottom: 20px; +} + +.table-schedule td, .table-schedule th { + border: none !important; +} + +// Table entries +.topic { + margin-right: 5px; +} +.topic-highlight { + color:white; + font-weight:bold; + border-radius:4px; + padding:4px; + background: cl.$grape25; +} +.day-highlight { + color:white; + border-radius:4px; + padding-right:4px; + padding-left:4px; + background: cl.$lime; + margin-left:9px; + margin-right:7px; +} +.table-date { + font-size:110%; + font-weight: bold; + padding-top:10px; + padding-bottom:5px; + margin-top:10px; +} + +@media only screen and (max-width: 575px) { + .landing { + grid-template-columns: 100%; + text-align: center; + } + + .landing-update { + text-align: center; + } + + .landing-right { + padding: 2em; + } +} +@media print { + .logo-button { + display: none!important; + } +} + +/*-- scss:end --*/ diff --git a/lectures/welcome/_extensions/percyfal/nbis-course/templates/_nbis.scss b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_nbis.scss new file mode 100644 index 0000000..3890f23 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbis-course/templates/_nbis.scss @@ -0,0 +1,140 @@ +/*-- scss:uses --*/ +@use 'colors' as cl; +@use 'callout' as co; +@import url("https://fonts.googleapis.com/css2?family=JetBrains+Mono&family=Nunito:ital,wght@0,400;0,600;1,400;1,600&display=swap"); + + +/*-- scss:defaults --*/ +// Description: NBIS theme +$primary: cl.$lime; +$secondary: cl.$lime75; +$link-color: cl.$teal50; +$body-color: cl.$black; +$code-block-bg: WhiteSmoke; + +$font-family-sans-serif: "Nunito", "Source Sans Pro", "Helvetica", "sans-serif"; +$font-family-monospace: "JetBrains Mono", "Ubuntu Mono", "Lucida Console", + "Courier", "monospace"; + + +$code-block-border-color: none !default; +$code-bg: $code-block-bg; /* inline code */ + +/*-- scss:variables --*/ +$font-size-root: 18px; +$code-font-size: 14px; +/* major second scale */ +$h1-font-size: 1.802rem; +$h2-font-size: 1.602rem; +$h3-font-size: 1.424rem; +$h4-font-size: 1.266rem; +$h5-font-size: 1.125rem; +$h6-font-size: 1rem; +$code-block-font-size: 0.95rem; + +/* site */ +$navbar-bg: white; +$footer-bg: white; +$footer-fg: DimGray; + +/*-- scss:rules --*/ +:root { + --color-primary: cl.$lime75; + --color-primary-dark: cl.$lime; +} + +.highlight { + background-color: $secondary; + color: white; + font-size: 0.85em; + padding: 0.2em 0.3em; + border-radius: 4px; + vertical-align: baseline; + font-weight: bold; +} + +pre.out { + background-color: cl.$lime; +} +pre.sourceCode.src { + background-color: cl.$teal25; +} + +.label { + color: cl.$lime; + font-weight: bold; + background-color: white; + font-size: 1em; +} + +code { + color: cl.$grape; + font-weight: bold; + background-color: white; +} + +code.tree { + line-height: 8px; + font-size: 18px; +} + +code.stree { + line-height: 6.6px; + font-size: 12px; +} + +code.sstree { + line-height: 5.6px; + font-size: 10px; +} + +code.large { + font-size: 20px; +} + +.citation { + font-size: 1em; +} + +.flushright { + float: right; +} + +.translatey50 { + transform: translate(0, -50px); +} + +.translatey100 { + transform: translate(0, -100px); +} + +.center { + margin: auto; +} + +.graynote { + color: cl.$mediumgray; + font-size: 0.7em; +} + +dt { + font-weight: bold; + color: cl.$grape; +} + +.software { + font-weight: bold; + font-style: italic; + color: cl.$teal50; +} + +.dna { + font-weight: normal; + font-family: monospace; +} + +.dna .italic { + font-weight: bold; + font-family: monospace; + color: cl.$aqua75; +} diff --git a/lectures/welcome/_extensions/percyfal/nbishome/_extension.yml b/lectures/welcome/_extensions/percyfal/nbishome/_extension.yml new file mode 100644 index 0000000..b941657 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbishome/_extension.yml @@ -0,0 +1,19 @@ +title: Nbishome +author: Per Unneberg +version: 1.0.0 +quarto-required: ">=1.4.0" +contributes: + revealjs-plugins: + - name: RevealNbisHome + script: + - nbishome.js + stylesheet: + - nbishome.css + config: + nbishome: + url: https://nbis.se + position: + left: 30px + bottom: 30px + right: auto + top: auto diff --git a/lectures/welcome/_extensions/percyfal/nbishome/nbishome.css b/lectures/welcome/_extensions/percyfal/nbishome/nbishome.css new file mode 100644 index 0000000..7c24d97 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbishome/nbishome.css @@ -0,0 +1,52 @@ +.reveal .reveal-nbis-home { + position: absolute; + left: 40px; +} + +.reveal .reveal-nbis-home.reveal-nbis-home-top { + top: 13px; +} + +.reveal .reveal-nbis-home.reveal-nbis-home-bottom { + bottom: 13px; +} + +.reveal .reveal-nbis-home .reveal-nbis-home-button { + display: inline-block; + color: $link-color; + font-size: .9em; + border-radius: 12px; + width: 30px; + text-align: center; + padding: 2px; + margin: 0 4px; +} + +.reveal .reveal-nbis-home .reveal-nbis-home-button:hover { + color: #ffffff; +} + +.hidden { + display: none; +} + +.reveal .reveal-nbis-home .slide-menu-button { + position: initial; + left: initial; + bottom: initial; +} + +.reveal .reveal-nbis-home .slide-menu-button a { + color: white; +} + +.reveal .reveal-nbis-home .playback { + position: relative; + left: 4px; + bottom: -17px; +} + +.reveal.paused .reveal-nbis-home .reveal-nbis-home-pause-button { + z-index: 110; + color: #757575; +} diff --git a/lectures/welcome/_extensions/percyfal/nbishome/nbishome.js b/lectures/welcome/_extensions/percyfal/nbishome/nbishome.js new file mode 100644 index 0000000..e59c219 --- /dev/null +++ b/lectures/welcome/_extensions/percyfal/nbishome/nbishome.js @@ -0,0 +1,44 @@ +window.RevealNbisHome = window.RevealNbisHome || { + id: "RevealNbisHome", + init: function (deck) { + initNBISHome(deck); + }, + configure: function (config) { + configure(config); + }, +}; + +// cf chalkboard plugin +const initNBISHome = function( Reveal ) { + var position = { left: "30px", bottom: "30px" }; + var url = "https://nbis.se"; + + var config = configure( Reveal.getConfig()['nbishome'] || {} ); + + function configure( config ) { + if ( config.url != undefined ) url = config.url; + if ( config.position != undefined ) position = config.position; + return config + } + + var button = document.createElement('div'); + button.className = 'reveal-nbis-home'; + button.id = 'reveal-nbis-home'; + button.style.visibility = 'visible'; + button.style.position = 'absolute'; + button.style.zIndex = 30; + button.style.fontSize = "24px"; + + button.style.left = position.left || "30px"; + button.style.bottom = position.bottom || "30px"; + + button.style.top = position.top || "auto"; + button.style.right = position.right || "auto"; + + button.innerHTML = ''; + document.querySelector('.reveal').appendChild(button); + + this.configure = configure; + + return this; +}; diff --git a/lectures/welcome/achievement-agreement-arms-1068523.jpg b/lectures/welcome/achievement-agreement-arms-1068523.jpg new file mode 100644 index 0000000..9d07242 Binary files /dev/null and b/lectures/welcome/achievement-agreement-arms-1068523.jpg differ diff --git a/lectures/welcome/john-sundh.jpg b/lectures/welcome/john-sundh.jpg index bc047ab..519eb45 100644 Binary files a/lectures/welcome/john-sundh.jpg and b/lectures/welcome/john-sundh.jpg differ diff --git a/lectures/welcome/verena-kutschera.jpg b/lectures/welcome/verena-kutschera.jpg index d847757..e93b7b2 100644 Binary files a/lectures/welcome/verena-kutschera.jpg and b/lectures/welcome/verena-kutschera.jpg differ diff --git a/lectures/welcome/welcome.Rmd b/lectures/welcome/welcome.Rmd deleted file mode 100644 index d8f1919..0000000 --- a/lectures/welcome/welcome.Rmd +++ /dev/null @@ -1,174 +0,0 @@ ---- -title: "Snakemake 'bring-your-own-code' workshop" -date: "`r format(Sys.time(), '%d %B, %Y')`" -output: - xaringan::moon_reader: - self-contained: true - seal: false - css: ["default", "../template.css"] - nature: - slideNumberFormat: "" ---- - -layout: true - - - ---- - -class: center, middle - -.HUGE[Welcome to the] -
-
-.HUGE[Snakemake] -
-.HUGE[_bring-your-own-code_] -
-.HUGE[Workshop] - -```{r Setup, echo = FALSE, message = FALSE} -# Knitr setup -knitr::opts_chunk$set(message = FALSE, - warning = FALSE) - -# Load packages -library("dplyr") -library("kableExtra") -``` - ---- - -# Your teachers - -.pull-left[ - -
-.small[John] - - -
-.small[Per] -] - -.pull-right[ - -
-.small[Marcel] - - -
-.small[Verena] -] - ---- - -# Zoom etiquette - -* Please keep your .green[webcam] on, unless you have problems with your internet connection - - -* .green[Lectures] - * Please stay muted - * If you have a question, unmute and speak up - - -* .green[Breakout rooms] - * Feel free to stay unmuted, unless there is a lot of background noise - - ---- - -# Slack - -* Please use the Slack workspace .green[nbissnakemakebyoc.slack.com] - during the workshop for communication - -* You can ask Snakemake questions to everyone in #2022-snakemake-questions - -* There are also separate channels for each of the groups that you - can use to communicate, e.g. to share code snippets and links - -* We will share the lecture slides with you in #general and - Canvas ([Discussions > Lecture slides](https://uppsala.instructure.com/courses/70024/discussion_topics/178575)) - ---- - -# Schedule - -.center[] - ---- - -# Schedule - -.center[] - ---- - -# Schedule - -.center[] - ---- - -# Breakout room feedback session (today) - -* Each of you .green[presents] their project to your group (incl. your teacher) - -* You have .green[30 minutes], incl. questions and feedback from your group - -* This will hopefully help you to find a .green[starting point] to work on your project - -.center[] - ---- - -# Breakout room coding sessions (days 2 and 3) - -* You will be in a .green[breakout room] with your group (incl. your teacher) - -* You will each .green[work individually] on your project - -* You can ask your group whenever you need help or feedback - -* Your teacher will try to help, but all participants are encouraged to .green[help each other] - -.center[] - ---- - -# Feedback sessions (days 2 and 3) - -* At the end of each day, after the coding sessions - -* In the main Zoom room - -* Time to .green[discuss open questions] that were difficult to solve in - the groups or any .green[interesting insights] about Snakemake - -.center[] - ---- - -# Breakout room groups - -Whenever we open the breakout rooms, please move yourselves into your group room - -.green[Breakout room 1: Marcel] -* Daniela, Camille, Kang, Karin - -.green[Breakout room 2: John] -* Trine, Tianyi, Anna - -.green[Breakout room 3: Per] -* Archana, Martyna, Begüm Serra, Magnus - - ---- - -class: center, middle - -.HUGE[Questions?] - ---- diff --git a/lectures/welcome/welcome.pdf b/lectures/welcome/welcome.pdf index eac0497..0e5690b 100644 Binary files a/lectures/welcome/welcome.pdf and b/lectures/welcome/welcome.pdf differ diff --git a/lectures/welcome/welcome.qmd b/lectures/welcome/welcome.qmd new file mode 100644 index 0000000..a7c0fd0 --- /dev/null +++ b/lectures/welcome/welcome.qmd @@ -0,0 +1,212 @@ +--- +title: Snakemake 'bring-your-own-code' workshop +footer: Welcome lecture +author: Verena Kutschera +institute: NBIS +date: "27 May, 2024" +format: + nbis-course-revealjs: + nbishome: + url: https://uppsala.instructure.com/courses/93486 + position: + left: 130px + bottom: 14px +revealjs-plugins: + - nbishome +--- + +## Your teachers + +:::: {.columns} +::: {.column width="50%"} +![](john-sundh.jpg){width=150} +
+John Sundh +
+ +![](per-unneberg.jpg){width=125} +
+Per Unneberg +
+::: + +::: {.column width="50%"} +![](marcel-martin.jpg){width=150} +
+Marcel Martin +
+ +![](verena-kutschera.jpg){width=150} +
+Verena Kutschera +
+ +::: +:::: + + +## SciLifeLab - Science for Life Laboratory + +- Institution for the advancement of molecular biosciences in Sweden +- **Research infrastructure services** and research groups +- Located at a number of universities across Sweden + + +## NBIS - National Bioinformatics Infrastructure Sweden + +- SciLifeLab's **Bioinformatics platform** +- Swedish ELIXIR node (the European infrastructure for biological information) +- Provides bioinformatics and data science **support to the Swedish life science research community** +- ~120 experts in different fields +- Distributed across six universities in Sweden + + +## NBIS - National Bioinformatics Infrastructure Sweden + +- **Bioinformatics support**, incl. bioimaging and medical imaging +- Software development +- Data management & data sharing guidance +- Compute resources +- **Training**: courses & workshops, advisory program for PhD students + + +## Snakemake BYOC workshop + +![](achievement-agreement-arms-1068523.jpg) + + +## Schedule + +**Monday, 27 May** + +| Time | Topic | +|---------------|-------------------------------------------------------| +| 09.00 - 09.20 | Welcome and introduction | +| 09.20 - 10.20 | **Lectures** | +| | - Anatomy of a Snakefile | +| | - Scatter-gather solutions | +| | - Example workflow | +| 10.20 - 12.50 | **Breakout rooms** (incl. coffee break) | +| | - Ice-breaker session | +| | - Feedback session | +| | - Coding session | +| 12.50 - 13.00 | Wrap-up day 1 | + + +## Breakout rooms + +- During most of the workshop, you will be in breakout rooms +- We have placed you in **small groups** based on your replies +in the application form +- Each group has their **own teacher** + +![](john-schnobrich-FlPc9_VocJ4-unsplash.jpg){width=50%} + + +## Feedback session (today) + +- You will be in a **breakout room** with your group (incl. your teacher) +- Each of you **presents** their project for the workshop +- You have **30 minutes**, incl. questions and feedback +- This will hopefully help you to find a **starting point** to work on your pipeline + +![](windows-p74ndnYWRY4-unsplash.jpg){width=50%} + + +## Coding sessions (every day) + +- You will be in a **breakout room** with your group (incl. your teacher) +- You will each **work individually** on your project +- You can ask your group whenever you need help or feedback +- Your teacher will try to help, but all participants are encouraged to **help each other** + +![](john-schnobrich-FlPc9_VocJ4-unsplash.jpg){width=50%} + + +## Schedule + +**Tuesday, 28 May** + +| Time | Topic | +|---------------|----------------------------------------------------------| +| 09.00 - 10.00 | **Lectures** | +| | - Using conda, containers and env-modules in Snakemake | +| | - Running Snakemake locally & on a cluster | +| 10.00 - 12.50 | **Breakout rooms** (incl. coffee break) | +| | - Coding session | +| 12.50 - 13.00 | Wrap-up day 2 | + + +## Schedule + +**Wednesday, 29 May** + +| Time | Topic | +|---------------|----------------------------------------------------------| +| 09.00 - 10.00 | **Lectures** | +| | - Best practices for Snakemake workflows | +| 10.00 - 12.50 | **Breakout rooms** (incl. coffee break) | +| | - Coding session | +| 12.50 - 13.00 | Wrap-up day 3 | + + +## Zoom etiquette + +- Please keep your webcam on +- **Lectures** + - Please stay muted + - If you have a question, unmute and speak up +- **Breakout rooms** + - Feel free to stay unmuted, unless there is a lot of background noise + + +## Slack + +- Workspace **nbissnakemakebyoc.slack.com** for communication during the workshop +- #general for questions or information concerning everyone +- Separate channels to communicate within each group, e.g. to share code snippets and links + + +## Workshop resources + +- We will share the **lecture slides** with you +- You will get a workshop **certificate** +- We kindly ask you to fill out a **feedback form** at the end of the +workshop + + +## Breakout room groups + +Whenever we open the breakout rooms, please move yourselves into your room: + +:::: {.columns} +::: {.column width="50%"} +| **Breakout room 1: Per** | +|-----------------------------| +| Johan | +| Daniel | +| Xinmeng | + + +| **Breakout room 3: Marcel** | +|-----------------------------| +| Abu Bakar | +| Atul | +| Veronica | +::: + +::: {.column width="50%"} +| **Breakout room 2: Verena** | +|-----------------------------| +| Tetsuichi | +| Aivars | +| Maike | + + +| **Breakout room 4: John** | +|-----------------------------| +| Ionut Sebastian | +| Mateus | +| Sybil | +::: +::::