Batch module updates - Fall 2024

[edmundmiller]

Warning

Please open PRs against the batch_update_staging branch. Then we can have small, easy to review, atomic PRs but also merge all batch updates to the modules repo in one go.

Note

[Edit - @ewels]: I've split into two groups, those that will require changes from pipeline developers (other than just pulling in updated modules as normal) and those that will not (those that will require changes to pipeline code).

Anything that doesn't need changes to pipeline code can be done iteratively if we wish, rather than batched.

✅ Will not require pipeline updates ✅ ✅ ✅ ✅ ✅

Beta Give feedback

1. Fix meta.yml structure for input and output tuples #4983
   bug +1
2. Add bio.tools identifiers in meta.yml #5830
   +0
3. Add EDAM ontologies for input and output channels in meta.yml #5831
   +0
4. Remove default channel from conda environment.yml files #5829
   9 of 9
   +0
   
5. Use Conda lock files #5835
   +0
6. Remove name from conda environment.yml #5836
   +0
   
7. Swap test file paths to use testdata_base_path #5848
   tests +1
   
8. [FEATURE] Add stub support to every module #4570
   enhancement +1
9. Remove tags.yml files if we are sure we no longer need them #6001
   +0
   
10. [FEATURE] Update modules to include arity in path. #6454
    +0
11. [FEATURE] Port pytest with --migrate-pytest-hard #6226
    +0
    
12. Add renovate comments to environment.yml #6504
    +0
    
13. Snapshot content of version files instead of a hash #6505
    +0
    
14. Update output to use type dir #6650
    +0

⚡ Will require pipeline updates ⚡⚡⚡⚡⚡⚡⚡⚡

Beta Give feedback

1. Migration to topic channels #4517
   0 of 5
   +0
2. Use eval to collect version #5834
   +0
3. Migrate modules to Seqera Containers #5832
   0 of 2
   container-issue +1
4. Support multiple different CPU architectures #5833
   +0

Just a heads up for bulk changes: there is a strict matrix limit of 256 tests in a test strategy on github, so that is the max number of module changes in a PR. @mashehu #2738

[SPPearce]

Can we automatically swap the test datasets paths too?

[ewels]

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

[SPPearce]

@SPPearce can you elaborate / open an issue about this please? (Or point to an existing one)

We have swapped the file paths used in tests from:
file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
to direct paths like:
file(params.modules_testdata_base_path + 'genomics/sarscov2/genome/genome.fasta', checkIfExists: true)

This is in the guidelines now, but could do with a bulk update.

Made an issue: #5848

[muffato]

My concern is that several of the changes require Nextflow 24.04. It means that from this point forward, nf-core will only support Nextflow 24.04+. Is there an analysis of 24.04 vs 22.04-23.10 and what developers have to do to make their pipelines compatible with 24.04 ? Any pipeline developer who is unable to move to 24.04 would remain stuck on pre-June 2024 modules and sub-workflows.

[ewels]

@muffato this has been policy for a long time now in nf-core. We are not adopting anything that requires an edge release of Nextflow here, everything is stable. There are other updates that are in 24.04 that we also want to update for pipelines, such as the use of resourceLimits. Some of those changes have been requested for ~6 years, and have been present in the Nextflow edge release for ~6 months.

We will soon bump the minimum required version of Nextflow in the pipeline template, and many / most of these changes should come through automatically via template syncs and module updates. We will do our best to write lint tests to cover most of them so that any required changes don't slip through the gaps. We've also recently started the practice of writing blog posts and doing bytesize talks where we talk over and demonstrate the updates that are required for new versions of the template.

I appreciate that it can be frustrating to keep updating pipelines with these kinds of changes. But that's also part of what makes nf-core great. We are a large pool of feature requests to Nextflow which makes it better. Stability is still possible through using previous versions of Nextflow and pipelines. I don't think that we should be held back from implementing new features once they are available in stable versions of Nextflow.

Any pipeline developer who is unable to move to 24.04

The hope is that no-one falls into this camp. If you can see any potential reasons for this, then please say and we can try to resolve them.

[muffato]

Oh yes @ewels , I don't deny the advantages of moving over. There will be a lot of benefits for people to move. Hopefully we can get away with it by just updating the Nextflow version number and can do pipeline template updates later.

[famosab]

Should this also include adding stubs? (That is not a breaking change but might still be worth to keep in mind?) Related to #4570.

[ewels]

Should this also include adding stubs? (That is not a breaking change but might still be worth to keep in mind?) Related to #4570.

Added, thanks!

[mahesh-panchal]

Can we add, including arity: to modules or is that out of scope? It'll require a more recent minimum version.

[ewels]

@mahesh-panchal - we will be requiring a pretty recent minimum version of Nextflow for a lot of this (see comments above), so I think that should be fine 👍🏻

Do we have an issue for that? Could you write one if not? Then we can add it to the list above. Thanks!

[mahesh-panchal]

There's not an issue, but a PR #5915. I'll make an issue.

[mahesh-panchal]

Added #6454 to list

[GallVp]

I also added pytest port #6226 . I have been experimenting with some automation. But there is still quite a bit of work remaining, round 170 single process pytests and around 200 multi-process pytests. @SPPearce has been very helpful in reviewing the PRs.

[chriswyatt1]

Happy to try tackling some of the issues here. Is the plan to tackle this module by module?, or try to script some of the changes, before bulk checking that the modules still function.

[SPPearce]

Happy to try tackling some of the issues here. Is the plan to tackle this module by module?, or try to script some of the changes, before bulk checking that the modules still function.

They need scripted changes, not manual updates per file. It is extremely painful to do anything manually across all ~1000 modules :|