Merge branch 'main' into jro-kraken-split-database-and-task

.github/workflows/build-documentation.yml

@@ -0,0 +1,38 @@
+name: build-documentation
+on:
+  push:
+    branches:
+      - main
+
+permissions: 
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout main
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Setup python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+
+      - name: Cache dependencies
+        uses: actions/cache@v2
+        with:
+          key: ${{ github.ref }}
+          path: .cache
+
+      - name: Install dependencies
+        run: |
+          pip install mkdocs-material mkdocs-material-extensions mkdocs-git-revision-date-localized-plugin mike mkdocs-glightbox
+      
+      - name: Build documentation
+        run: |
+          git config user.name sage-wright
+          git config user.email [email protected] 
+          mike deploy --push main

.github/workflows/build-version-release.yml

@@ -0,0 +1,39 @@
+name: version-documentation
+on:
+ release:
+   types: [published]
+
+permissions: 
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout main
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Setup python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+
+      - name: Cache dependencies
+        uses: actions/cache@v2
+        with:
+          key: ${{ github.ref }}
+          path: .cache
+
+      - name: Install dependencies
+        run: |
+          pip install mkdocs-material mkdocs-material-extensions mkdocs-git-revision-date-localized-plugin mike mkdocs-glightbox
+      
+      - name: Build documentation
+        run: |
+          git config user.name sage-wright
+          git config user.email [email protected] 
+          LATEST_RELEASE=$(curl -sL https://api.github.com/repos/theiagen/public_health_bioinformatics/releases/latest | jq -r ".tag_name")
+          mike deploy --push --update-aliases ${LATEST_RELEASE} latest
+          mike set-default --push latest

README.md

@@ -4,7 +4,7 @@ The Public Health Bioinformatics Bioinformatics repository contains workflows fo
 
 ## Introduction
 
-**More information about the steps undertaken in these workflows is available via the [Theiagen Public Resources Documentation](https://theiagen.notion.site/Theiagen-Public-Health-Resources-a4bd134b0c5c4fe39870e21029a30566).**
+**More information about the steps undertaken in these workflows is available via the [Theiagen Public Resources Documentation](https://theiagen.github.io/public_health_bioinformatics/latest/).**
 
 Support for running these workflows can be sought by raising a [GitHub issue](https://github.com/theiagen/public_health_bioinformatics/issues/new) or by contacting Theiagen at [email protected].
 
@@ -16,19 +16,19 @@ The PHB repository contains workflows for the characterization, genomic epidemio
 
 All workflows in the PHB repository end with `_PHB` in order to differentiate them from earlier versions and from the original tools they incorporate.
 
-Briefly, the main *genomic characterization* workflows are split by pathogen type:
+Briefly, the main _genomic characterization_  workflows are split by pathogen type:
 
 1. **Viral** (***TheiaCoV*** workflows)
 2. **Bacterial** (***TheiaProk*** workflows)
-3. **Fungal** (***TheiaEuk*** workflows)
+3. **Fungal** (*[**TheiaEuk**](../workflows/genomic_characterization/theiaeuk.md)* workflows)
 
-Many more workflows are available, and are documented in detail in the [Theiagen Public Resources Documentation](https://theiagen.notion.site/Theiagen-Public-Health-Resources-a4bd134b0c5c4fe39870e21029a30566).
+Many more workflows are available, and are documented in detail in the [Theiagen Public Resources Documentation](https://theiagen.github.io/public_health_bioinformatics/latest/).
 
 ## On the Shoulder of Giants
 
 The PHB repository would not be possible without its predecessors. We would like to acknowledge the following repositories, individuals, and contributors for their influence on the development of these workflows:
 
-The PHB repository originated from collaborative work with Andrew Lang, PhD & his [Genomic Analysis WDL workflows](https://github.com/AndrewLangvt/genomic_analyses). The workflows and task development were influenced by The Broad's [Viral Pipes](https://github.com/broadinstitute/viral-pipelines) repository. The TheiaCoV workflows for viral genomic characterization were influenced by UPHL's [Cecret](https://github.com/UPHL-BioNGS/Cecret) & StaPH-B's [Monroe](https://staph-b.github.io/staphb_toolkit/workflow_docs/monroe/) workflows. The TheiaProk workflows for bacterial genomic characterization were influenced by Robert Petit's [bactopia](https://github.com/bactopia/bactopia). Most importantly, the PHB user community drove the development of these workflows and we are grateful for their feedback and contributions.
+The PHB repository originated from collaborative work with Andrew Lang, PhD & his [Genomic Analysis WDL workflows](https://github.com/AndrewLangvt/genomic_analyses). The workflows and task development were influenced by The Broad's [Viral Pipes](https://github.com/broadinstitute/viral-pipelines) repository. The TheiaCoV workflows for viral genomic characterization were influenced by UPHL's [Cecret](https://github.com/UPHL-BioNGS/Cecret) & StaPH-B's Monroe (now deprecated) workflows. The TheiaProk workflows for bacterial genomic characterization were influenced by Robert Petit's [bactopia](https://github.com/bactopia/bactopia). Most importantly, the PHB user community drove the development of these workflows and we are grateful for their feedback and contributions.
 
 If you would like to provide feedback, please raise a [GitHub issue](https://github.com/theiagen/public_health_bioinformatics/issues/new).
 
@@ -83,8 +83,8 @@ The authors declare no conflict of interest.
 
 Please cite this paper if publishing work using any workflows:
 
-> Libuit, Kevin G., Emma L. Doughty, James R. Otieno, Frank Ambrosio, Curtis J. Kapsak, Emily A. Smith, Sage M. Wright, et al. 2023. “Accelerating Bioinformatics Implementation in Public Health.” Microbial Genomics 9 (7). https://doi.org/10.1099/mgen.0.001051.
+> Libuit, Kevin G., Emma L. Doughty, James R. Otieno, Frank Ambrosio, Curtis J. Kapsak, Emily A. Smith, Sage M. Wright, et al. 2023. "Accelerating Bioinformatics Implementation in Public Health." Microbial Genomics 9 (7). https://doi.org/10.1099/mgen.0.001051.
 
 Alternatively, please cite this paper if using the TheiaEuk workflow:
 
-> Ambrosio, Frank, Michelle Scribner, Sage Wright, James Otieno, Emma Doughty, Andrew Gorzalski, Danielle Siao, et al. 2023. “TheiaEuk: A Species-Agnostic Bioinformatics Workflow for Fungal Genomic Characterization.” Frontiers in Public Health 11. https://doi.org/10.3389/fpubh.2023.1198213.
+> Ambrosio, Frank, Michelle Scribner, Sage Wright, James Otieno, Emma Doughty, Andrew Gorzalski, Danielle Siao, et al. 2023. "TheiaEuk: A Species-Agnostic Bioinformatics Workflow for Fungal Genomic Characterization." Frontiers in Public Health 11. https://doi.org/10.3389/fpubh.2023.1198213.

docs/assets/files/GPSC_README_PopPUNK2.txt

@@ -0,0 +1,37 @@
+GPSC assignment
+
+Install PopPUNK 2.4 as per instructions at PopPUNK documentation and download 
+the GPS reference database and the GPS designation.
+
+Files required to run GPSC assignment using PopPUNK 2.4:
+1. A 2-column tab-delimited file containing sample name and path to the
+   corresponding assembly (no header)
+2. GPS reference database <GPS_v6>
+3. GPS designation <GPS_v6_external_clusters.csv>
+
+output directory name is assigned using --output
+number of threads can be changed using –threads
+
+Run GPSC assignment:
+
+poppunk_assign --db GPS_v6 \
+               --distances GPS_v6/GPS_v6.dists \
+               --query <2-column path to assembly> \
+               --output <GPSC_assignment> \
+               --external-clustering \
+               GPS_v6_external_clusters.csv
+
+Outputs:
+
+_clusters.csv: popPUNK clusters with dataset specific nomenclature
+_external_clusters.csv: GPSC v6 scheme designations
+
+Novel Clusters are assigned NA in the _external_clusters.csv as they have 
+not been defined in the v6 dataset used to designate the GPSCs. Please email: 
+[email protected] to have novel clusters added to the database and a 
+GPSC cluster name assigned after you have checked for low level contamination 
+which may contribute to biased accessory distances.
+
+Merged clusters: Unsampled diversity may represent missing variation linking two 
+clusters. GPSCs are then merged. For example if GPSC23 and GPSC362 merged, the 
+GPSC would be then reported as GPSC23, with a merge history of GPSC23;362.

docs/assets/files/TheiaEuk_qc_check_template.tsv

@@ -0,0 +1,18 @@
+taxon	num_reads_raw1	num_reads_raw2	num_reads_clean1	num_reads_clean2	est_coverage_raw	est_coverage_clean	combined_mean_q_raw	r1_mean_q_raw	r2_mean_q_raw	combined_mean_readlength_raw	r1_mean_readlength_raw	r2_mean_readlength_raw	combined_mean_q_clean	r1_mean_q_clean	r2_mean_q_clean	combined_mean_readlength_clean	r1_mean_readlength_clean	r2_mean_readlength_clean	assembly_length_min	assembly_length_max	number_contigs	n50_value	quast_gc_percent_min	quast_gc_percent_max	busco_completeness
+Aspergillus_terreus	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	27960000	31840000	1000	10000			95
+Aspergillus_flavus	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	35090000	42930000	1000	10000			95
+Aspergillus_fumigatus	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	27360000	33400000	1000	10000			95
+Candida_albicans	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	11650000	18700000	1000	10000			95
+Candida_auris	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	12100000	14250000	1000	10000	43	47	95
+Candida_glabrata	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	11810000	14520000	1000	10000			95
+Candida_parapsilosis	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	11570000	13330000	1000	10000			95
+Candida_tropicalis	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	14350000	15730000	1000	10000			95
+Clavispora_lusitaniae	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	11910000	12380000	1000	10000			95
+Coccidioides_immitis	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	27470000	28950000	1000	10000			95
+Cryptococcus_gattii_VGI	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	17230000	18370000	1000	10000			95
+Cryptococcus_neoformans	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	17990000	19750000	1000	10000			95
+Fusarium	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	26090000	72920000	1000	10000			95
+Kluyveromyces_marxianus	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	9555000	13340000	1000	10000			95
+Pichia_kudriavzevii	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	72470000	12940000	1000	10000			95
+Yarrowia_lipolytica	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	19720000	21250000	1000	10000			95
+Candida_haemuloni	1000000	1000000	1000000	1000000	30	30	30	30	30	135	135	135	30	30	30	135	135	135	12590000	13310000	1000	10000			95

docs/assets/files/TheiaProk_FASTA_qc_check_template.tsv

@@ -0,0 +1,15 @@
+taxon	assembly_length_min	assembly_length_max	number_contigs	n50_value	quast_gc_percent_min	quast_gc_percent_max	busco_completeness	ani_highest_percent	ani_highest_percent_bases_aligned
+Listeria_monocytogenes	2800000	3200000	500	10000			95	92	70
+Escherichia_coli	4900000	6000000	500	10000			95	92	70
+Shigella	4200000	4900000	500	10000			95	92	70
+Salmonella	4400000	5700000	500	10000			95	92	70
+Campylobacter	1400000	2200000	500	10000			95	92	70
+Vibrio_cholerae	3800000	4300000	500	10000			95	92	70
+Vibrio_parahaemolyticus	4900000	5500000	500	10000			95	92	70
+Vibrio_vulnificus	4700000	5300000	500	10000			95	92	70
+Pseudomonas			500	10000			95		
+Streptococcus			500	10000			95		
+Acinetobacter			500	10000			95		
+Mycobacterium			500	10000			95		
+Legionella			500	10000			95		
+Klebsiella			500	10000			95		

docs/assets/files/TheiaProk_Illumina_PE_qc_check_template.tsv

@@ -0,0 +1,15 @@
+taxon	num_reads_raw1	num_reads_raw2	num_reads_clean1	num_reads_clean2	est_coverage_raw	est_coverage_clean	combined_mean_q_raw	r1_mean_q_raw	r2_mean_q_raw	combined_mean_readlength_raw	r1_mean_readlength_raw	r2_mean_readlength_raw	combined_mean_q_clean	r1_mean_q_clean	r2_mean_q_clean	combined_mean_readlength_clean	r1_mean_readlength_clean	r2_mean_readlength_clean	assembly_length_min	assembly_length_max	midas_secondary_genus_abundance	number_contigs	n50_value	quast_gc_percent_min	quast_gc_percent_max	busco_completeness	ani_highest_percent	ani_highest_percent_bases_aligned
+Listeria_monocytogenes	100000	100000	100000	100000	20		30			135			30			135			2800000	3200000	0.01						92	70
+Escherichia_coli	100000	100000	100000	100000	40		30			135			30			135			4900000	6000000	0.01						92	70
+Shigella	100000	100000	100000	100000	40		30			135			30			135			4200000	4900000	0.01						92	70
+Salmonella	100000	100000	100000	100000	30		30			135			30			135			4400000	5700000	0.01						92	70
+Campylobacter	100000	100000	100000	100000	20		30			135			30			135			1400000	2200000	0.01						92	70
+Vibrio_cholerae	100000	100000	100000	100000	40		30			135			30			135			3800000	4300000	0.01						92	70
+Vibrio_parahaemolyticus	100000	100000	100000	100000	40		30			135			30			135			4900000	5500000	0.01						92	70
+Vibrio_vulnificus	100000	100000	100000	100000	40		30			135			30			135			4700000	5300000	0.01						92	70
+Pseudomonas	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		
+Streptococcus	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		
+Acinetobacter	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		
+Mycobacterium	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		
+Legionella	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		
+Klebsiella	100000	100000	100000	100000	30	30	30	30	30	135	135	135	30	30	30	135	135	135				500	10000			95		

docs/assets/files/example_excluded_samples.tsv

@@ -0,0 +1,10 @@
+Samples excluded for quality thresholds:
+sample_name	message	
+sample2	VADR skipped due to poor assembly
+sample3	VADR number alerts too high: 3 greater than limit of 0
+sample4	Number of Ns was too high: 10000 greater than limit of 5000
+
+Samples excluded for missing required metadata (will have empty values in indicated columns):
+tablename_id	organism	country	library_layout
+sample5			paired
+sample6	SARS-CoV-2	USA

docs/assets/files/input_jsons/TheiaCoV_FASTA_RSV-A_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_fasta.organism":"rsv_a"}

docs/assets/files/input_jsons/TheiaCoV_FASTA_RSV-B_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_fasta.organism":"rsv_b"}

docs/assets/files/input_jsons/TheiaCoV_FASTA_WNV_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_fasta.organism":"WNV"}

docs/assets/files/input_jsons/TheiaCoV_FASTA_flu_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_fasta.organism":"flu"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_HIV_v1_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"HIV","theiacov_illumina_pe.hiv_primer_version":"v1"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_HIV_v2_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"HIV","theiacov_illumina_pe.hiv_primer_version":"v2"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_RSV-A_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"rsv_a"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_RSV-B_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"rsv_b"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_WNV_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"WNV"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_PE_flu_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_pe.organism":"flu"}

docs/assets/files/input_jsons/TheiaCoV_Illumina_SE_WNV_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_illumina_se.organism":"WNV"}

docs/assets/files/input_jsons/TheiaCoV_ONT_HIV_v1_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_ont.organism":"HIV","theiacov_ont.hiv_primer_version":"v1"}

docs/assets/files/input_jsons/TheiaCoV_ONT_HIV_v2_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_ont.organism":"HIV","theiacov_ont.hiv_primer_version":"v2"}

docs/assets/files/input_jsons/TheiaCoV_ONT_flu_2024-04-19.json

@@ -0,0 +1 @@
+{"theiacov_ont.organism":"flu"}

docs/assets/files/theiavalidate/example_exact_differences.tsv

@@ -0,0 +1,8 @@
+	columnA-string	columnA-string	columnB-set	columnB-set	columnC-ignore	columnC-ignore	columnD-float	columnD-float	columnE-missing	columnE-missing
+	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv
+samples										
+sample1			item1,item2,item3	item1,item3,item2	cheese	cheesecake	1000.0	999.0		
+sample2	option1	option2	item1,item3,item2	item1,item2,item3	cheesecake	batter				
+sample3	option2	option1	item1,item2,item3	item1,item2	cake	cheese	14.0	24.0	present	
+sample4			item2,item1	item1,item2			3492.0	728.0		
+sample5			item1,item2	item1,item2,item3			3.0	4.0		

docs/assets/files/theiavalidate/example_validation_criteria_differences.tsv

@@ -0,0 +1,7 @@
+Column	columnB-set	columnB-set	columnD-float	columnD-float	columnE-missing	columnE-missing
+Table	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv	example_table1.tsv	example_table2.tsv
+sample1						
+sample2						
+sample3	item1,item2,item3	item1,item2	14.0	24.0	present	
+sample4			3492.0	728.0		
+sample5	item1,item2	item1,item2,item3	3.0	4.0		

docs/assets/files/theiavalidate/filtered_example_table1.tsv

@@ -0,0 +1,6 @@
+samples	columnA-string	columnB-set	columnC-ignore	columnD-float	columnE-missing
+sample1	option1	item1,item2,item3	cheese	1000	present
+sample2	option1	item1,item3,item2	cheesecake	12	present
+sample3	option2	item1,item2,item3	cake	14	present
+sample4	option1	item2,item1	cakebatter	3492	
+sample5	option2	item1,item2	batter	3	present

docs/assets/files/theiavalidate/filtered_example_table2.tsv

@@ -0,0 +1,6 @@
+samples	columnA-string	columnB-set	columnC-ignore	columnD-float	columnE-missing
+sample1	option1	item1,item3,item2	cheesecake	999	present
+sample2	option2	item1,item2,item3	batter	12	present
+sample3	option1	item1,item2	cheese	24	
+sample4	option1	item1,item2	cakebatter	728	
+sample5	option2	item1,item2,item3	batter	4	present

docs/assets/files/theiavalidate/theiacov-validation-criteria.txt

@@ -0,0 +1,24 @@
+column	criteria
+abricate_flu_subtype	EXACT
+abricate_flu_type	EXACT
+assembly_length_unambiguous	0.01
+assembly_mean_coverage	0.01
+irma_subtype	EXACT
+irma_type	EXACT
+kraken_human	EXACT
+kraken_human_dehosted	EXACT
+kraken_sc2	EXACT
+kraken_sc2_dehosted	EXACT
+kraken_target_org	EXACT
+kraken_target_org_dehosted	EXACT
+nextclade_aa_dels	SET
+nextclade_aa_subs	SET
+nextclade_clade	EXACT
+nextclade_lineage	EXACT
+nextclade_tamiflu_resistance_aa_subs	SET
+num_reads_clean1	EXACT
+num_reads_clean2	EXACT
+number_N	0.01
+pango_lineage	EXACT
+percent_reference_coverage	0.01
+vadr_num_alerts	EXACT

docs/assets/files/theiavalidate/theiaeuk-validation-criteria.txt

@@ -0,0 +1,13 @@
+column	criteria
+assembly_length	0.01
+busco_results	EXACT
+clade_type	EXACT
+est_coverage_clean	0.01
+est_coverage_raw	0.01
+gambit_predicted_taxon	EXACT
+n50_value	0.01
+num_reads_clean1	EXACT
+num_reads_clean2	EXACT
+number_contigs	0.01
+quast_gc_percent	0.01
+theiaeuk_snippy_variants_hits	SET