Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardize conversion workflow #369

Draft
wants to merge 66 commits into
base: dev
Choose a base branch
from
Draft

Standardize conversion workflow #369

wants to merge 66 commits into from

Conversation

fmalmeida
Copy link
Contributor

@fmalmeida fmalmeida commented Sep 16, 2024 •
edited by grst
Loading

Summary

Hi,
This work relates to #310 and discussion that is happening in slack.

Currently, the implementation as been done only for Star aligner so I am opening the PR so we can have a proper discussion and alignment over what the code should look like, before I can apply the same for the other aligners.

PoC overview

  • Aligners
    • star
    • alevin
    • cellranger
    • kallisto
  • Make mtx_to_h5ad module have aligner-specific template to avoid having a gigantic single module with multiple conditions
  • added a copy of cellbender subworkflow from scdownstream to perform a more straightforward empty drops filter that keeps format of h5ad input
    • Thus, not requiring further work after filter, since it will keep the format of its input, which should be the standardized h5ad
  • Moved h5ad to Rds conversion to use anndataR package so to keep it simple

TODOs summary

  • Define where (and what) h5ad format standardization should happen
    • Should it happen directly in the aligner-specific template script (would be the most straightforward) or should it be in a separate (maybe the anndataR conversion) module that will perform the standardization and generate a new h5ad for all aligners?
  • Create the anndataR docker container within nf-core (currently is using my personal repo)
  • Define what is the desired standardization for h5ad
    • which commands should be executed?
  • Cleanup repo (remove old modules currently as backup, documentation, etc.)

Addressed issues

Close #385
Close #310
Close #370
Close #330

zxBIB Almeida,Felipe (GCBDS) EXTERNAL added 23 commits August 19, 2024 11:22
comment out for development
52fa9ca
refact modules for STAR aligner
bbf299c
directly pass txp2gene
c3eb5ea
simplify module lines
4771ed1
emit h5ad on starsolo
af85f87
add versions emition
d2a5386
update module to use templates and cleanup way of priting versions
0e55051
fix h5ad generator script
994d5a5
simplify check
b317165
Fix h5ad structure
72e9d50
updated concat module
8829080
workflow misses emptydrops and seurat & mtx conversion modules
9fc75b4
start emptydrops cellbender subworkflow
dc69f47
fix paths
9c20420
started the anndatar standardization module
6831195
concat h5ad with anndatar h5ad
237d1ca
update tag information
f5fb4a2
update tags
752666b
module is only to convert to rds
d0ad7f6
update directives
20ac4ae
update comments
da5b036
add cellbender to workflow
b90f388
start organisation of files
7c304cc
@fmalmeida fmalmeida requested a review from grst September 16, 2024 08:46
@fmalmeida fmalmeida self-assigned this Sep 16, 2024
@fmalmeida fmalmeida linked an issue Sep 16, 2024 that may be closed by this pull request
Clean up mtx conversion code #310
Open
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 16, 2024 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 079bb7e

+| ✅ 216 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.2
  • Run at 2024-10-30 11:16:56

zxBIB Almeida,Felipe (GCBDS) EXTERNAL added 2 commits September 26, 2024 09:25
@grst
Copy link
Member

grst commented Oct 28, 2024

so we would double the execution of the mtx conversions? one to at least have it before cellbender, and one again afterwards?

only for filtered/unfiltered (i.e. what comes out of the aligner). The rest (e.g. cellbender) should run entirely in h5ad space.
The idea is really to generate a h5ad once that is standardized across all aligners, and then use this for everything downstream.

grst
grst reviewed Oct 28, 2024
View reviewed changes
modules/local/adata_barcodes.nf Show resolved Hide resolved
modules/local/templates/anndatar_convert.R Outdated Show resolved Hide resolved
modules/local/templates/concat_h5ad.py Outdated Show resolved Hide resolved
modules/local/templates/concat_h5ad.py Outdated Show resolved Hide resolved
modules/local/templates/concat_h5ad.py Outdated Show resolved Hide resolved
modules/local/templates/mtx_to_h5ad_cellranger.py Outdated Show resolved Hide resolved
modules/local/templates/mtx_to_h5ad_kallisto.py Outdated Show resolved Hide resolved
modules/local/templates/mtx_to_h5ad_kallisto.py Outdated Show resolved Hide resolved
modules/local/templates/mtx_to_h5ad_kallisto.py Outdated Show resolved Hide resolved
modules/local/templates/mtx_to_h5ad_kallisto.py Outdated
Comment on lines 108 to 116
for type in ['spliced', 'unspliced']:
input_to_adata(
matrix=glob.glob("${inputs}/" + f"{type}*.mtx")[0],
barcodes=glob.glob("${inputs}/" + f"{type}*.barcodes.txt")[0],
features=glob.glob("${inputs}/" + f"{type}*.genes.txt")[0],
output="${meta.id}/${meta.id}_${meta.input_type}" + f"_{type}_matrix.h5ad",
sample="${meta.id}",
t2g="${txp2gene}"
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to make a single anndata object and store this information in .layers.

This was referenced Oct 28, 2024
Open
Open
@fmalmeida @grst
write uncompressed
c1e8357 
Co-authored-by: Gregor Sturm <[email protected]>
@nf-core-bot
Copy link
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 2.14.1.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

fmalmeida and others added 24 commits October 30, 2024 09:04
@fmalmeida @grst
use .astype(str)
ae85710 
Co-authored-by: Gregor Sturm <[email protected]>
@fmalmeida @grst
simplify iteration
ae8809a 
Co-authored-by: Gregor Sturm <[email protected]>
@fmalmeida @grst
perform join left operation
68464de 
Co-authored-by: Gregor Sturm <[email protected]>
@fmalmeida @grst
do not compress output h5ad
fdedc4d 
Co-authored-by: Gregor Sturm <[email protected]>
@fmalmeida @grst
perform join left operation
187dbf6 
Co-authored-by: Gregor Sturm <[email protected]>
@fmalmeida
not compress h5ad output
98af608
@fmalmeida
simplify iteration
efd6299
@fmalmeida
simplify index iteration
f357fd7
@fmalmeida
fix unmatched parenthesis
bd1a74c
@fmalmeida
fix use of igenomes ... pipeline was not properly selecting igenomes …
4731e00 
…options
@fmalmeida
correct values parsing
a71348f
@fmalmeida
fix container registry
d2f9cfd
@fmalmeida
do not save versions files
5c31226
@fmalmeida
also convert concat h5ads
8278429
@fmalmeida
manage subdirectory in publishDir
2b199d0
@fmalmeida
match template scripts with new publishDir
cb67797
@fmalmeida
have outputs separated
7230095
@fmalmeida
make parsing inside sub-workflow
206a7c1
@fmalmeida
added kallisto to correct structure
c4a09e0
@fmalmeida
correct mix of channels
971667b
@fmalmeida
correct for cellranger multi
8122113
@fmalmeida
correct stub
44d1cb7
@fmalmeida
remove glob in txp2gene
18cfdd7
@fmalmeida
added small comment to local modules
079bb7e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grst grst grst left review comments

@apeltzer apeltzer Awaiting requested review from apeltzer

At least 1 approving review is required to merge this pull request.

Assignees

@fmalmeida fmalmeida

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Clean up mtx conversion code
3 participants
@fmalmeida @grst @nf-core-bot