Use spatialdata #67

cavenel · 2024-01-31T08:55:15Z

This PR replaces h5ad anndata objects with zarr spatialdata objects on disk.

Update docker environment to add spatialdata
Add function to get legacy anndata objects for scanpy plotting
Update tests to check for zarr files instead of h5ad
Update markdown text of quarto reports to introduce SpatialData format

Replace all reads and writes with SpatialData zarr, and use a "legacy anndata" conversion function to add images and scales, in order to plot with scanpy.

github-actions · 2024-01-31T08:56:24Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 509f547

+| ✅ 171 tests passed       |+
#| ❔   6 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

files_exist - File is ignored: conf/igenomes.config
files_unchanged - File ignored due to lint config: .gitattributes
files_unchanged - File ignored due to lint config: assets/nf-core-spatialtranscriptomics_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-spatialtranscriptomics_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-spatialtranscriptomics_logo_dark.png
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-spatialtranscriptomics_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-spatialtranscriptomics_logo_light.png
files_exist - File found: docs/images/nf-core-spatialtranscriptomics_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-spatialtranscriptomics_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/WorkflowSpatialtranscriptomics.groovy
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.spaceranger_reference= https://cf.10xgenomics.com/supp/spatial-exp/refdata-gex-GRCh38-2020-A.tar.gz
nextflow_config - Config default value correct: params.qc_min_counts= 500
nextflow_config - Config default value correct: params.qc_min_genes= 250
nextflow_config - Config default value correct: params.qc_min_spots= 1
nextflow_config - Config default value correct: params.qc_mito_threshold= 20.0
nextflow_config - Config default value correct: params.qc_ribo_threshold= 0.0
nextflow_config - Config default value correct: params.qc_hb_threshold= 100.0
nextflow_config - Config default value correct: params.cluster_n_hvgs= 2000
nextflow_config - Config default value correct: params.cluster_resolution= 1.0
nextflow_config - Config default value correct: params.n_top_svgs= 14
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 64
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (149 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.13.1
Run at 2024-04-12 12:13:46

This reverts commit fe7a7dc. Lowering the test thresholds was part of troubleshooting the spatial DE module, but did solve the issue.

bin/quality_controls.qmd

bin/spatially_variable_genes.qmd

…ranscriptomics into use-spatialdata

cavenel added 2 commits January 31, 2024 09:34

Use spatialdata zarr objects instead of AnnData

e13cf66

Replace all reads and writes with SpatialData zarr, and use a "legacy anndata" conversion function to add images and scales, in order to plot with scanpy.

Fix typo

d866f37

Fix black and prettier

aec833f

nf-core deleted a comment from github-actions bot Jan 31, 2024

fasterius and others added 17 commits February 12, 2024 12:21

Update Conda environment and Dockerfile

12ca883

Add QUARTONOTEBOOK nf-core module

3ced22c

Use QUARTONOTEBOOK module in pipeline

2e595b7

Remove old local modules

508c0b2

Update changelog

393bc3c

Remove XDG paths, which are now in QUARTONOTEBOOK

64d3327

Lower test thresholds

fe7a7dc

Revert "Lower test thresholds"

548bc67

This reverts commit fe7a7dc. Lowering the test thresholds was part of troubleshooting the spatial DE module, but did solve the issue.

Fix report module output names and paths

66e32bc

Update Conda environment

5226398

Template update for nf-core/tools version 2.13

cd7a352

Template update for nf-core/tools version 2.13.1

e9d1089

Remove redundant environment specification

91ca58b

Update MultiQC module

9539d6e

Use Space Ranger output for MultiQC

1a42abf

Add missing MultiQC output documentation

a095713

Update changelog

f8c2863

fasterius mentioned this pull request Mar 1, 2024

Add MultiQC support for Space Ranger #70

Closed

6 tasks

fasterius linked an issue Mar 4, 2024 that may be closed by this pull request

Add SpatialData as output #39

Closed

cavenel and others added 6 commits March 4, 2024 15:39

Fix tables writing in SpatialData objects

01eefcc

Fix quarto formatting

d1bf3ec

Move from SpatialDE to Squidpy for SVG computation

c606f17

Merge remote-tracking branch 'origin/dev' into use-spatialdata

72f50c9

Add util-subworkflows and move most lib/ content

52e4c21

Merge branch 'TEMPLATE' into template-updates

eb23d32

fasterius mentioned this pull request Apr 4, 2024

Missing SpatialDE package in Docker container for ST_SPATIAL_DE process #74

Closed

fasterius marked this pull request as ready for review April 4, 2024 06:31

fasterius self-requested a review April 4, 2024 06:31

fasterius assigned cavenel and fasterius Apr 4, 2024

fasterius added 2 commits April 4, 2024 08:38

Update changelog

e103fc4

Remove duplicate citation for SpatialData

4f12ac8

fasterius reviewed Apr 4, 2024

View reviewed changes

bin/quality_controls.qmd Outdated Show resolved Hide resolved

bin/quality_controls.qmd Show resolved Hide resolved

bin/spatially_variable_genes.qmd Outdated Show resolved Hide resolved

fasterius added 4 commits April 4, 2024 09:21

Minor formatting

f82267d

Update quartonotebook module

8ae8c26

Update spaceranger/count module

d4861bd

Remove TODO

f253f56

grst mentioned this pull request Apr 8, 2024

Feedback on concatenate() scverse/spatialdata#541

Closed

cavenel and others added 14 commits April 12, 2024 12:37

Update SpatialData to v0.1.2

51d5918

Remove temporary fix for scanpy issue

a9a5570

Add back missing line in SVG quarto

b139755

Fix spelling of AnnData

d2a490b

Clean read_zarr local path

184dec8

Fix qmd headers

c364e3e

Move degs/ output to data/ folder

6754c6f

Add fix for pip install in Dockerfile

72e07a0

Fix adata import order

de3c8c1

Merge branch 'use-spatialdata' of https://github.com/nf-core/spatialt…

e6f156e

…ranscriptomics into use-spatialdata

Remove trailing whitespace

a465e0f

Export spatialdata_io version

7e1756f

Remove Conda profile for READ_DATA process

7ec707c

Update snapshots

509f547

fasterius approved these changes Apr 12, 2024

View reviewed changes

fasterius merged commit 5db31c4 into dev Apr 12, 2024
5 of 6 checks passed

fasterius deleted the use-spatialdata branch April 12, 2024 12:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use spatialdata #67

Use spatialdata #67

cavenel commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Use spatialdata #67

Use spatialdata #67

Conversation

cavenel commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

cavenel commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️