Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BOWTIE2 task failing when scratch set as /tmp #465

Open
CarstenBaker opened this issue Jun 28, 2023 · 17 comments
Open

BOWTIE2 task failing when scratch set as /tmp #465

CarstenBaker opened this issue Jun 28, 2023 · 17 comments
Labels
bug Something isn't working

Comments

@CarstenBaker
Copy link

CarstenBaker commented Jun 28, 2023 •
edited by jfy133
Loading

Description of the bug

We are trying to redirect the scratch to /tmp on the BOWTIE2 task but it fails with:

FATAL: container creation failed: mount /tmp/nxf.cN0zaFFBsi->/tmp/nxf.cN0zaFFBsi error: while mounting /tmp/nxf.cN0zaFFBsi: destination /tmp/nxf.cN0zaFFBsi doesn't exist in container
cp: cannot stat '.command.trace': No such file or directory

This process works correctly on other tasks such as FASTP with same config (when /tmp set as scratch).
Looking in the /tmp directory the folder is not being created as part of this process, the issue seems to be before singularity is even called.

The nxf_mktemp() function looks correct when compared to a working task (same as FASTP).
If I set the scratch folder to something other than /tmp then it works correctly for BOWTIE step.

The function is working correctly for fastqc/fastp/Busco (and the other processes) using /tmp for scratch.

Can anyone try running the test profile with singularity and using /tmp as scratch to confirm ?

Command used and terminal output

nextflow run nf-core/mag -profile test,singularity -c local.config --outdir output

Relevant files

config file example:

process {
     withName: 'BOWTIE2_ASSEMBLY_ALIGN|BOWTIE2_PHIX_REMOVAL_ALIGN' {
        stageInMode = 'copy'
        scratch = '/tmp'
     }
}

.nextflow.log

Jun-28 20:05:38.302 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core/mag -r 2.3.2 -profile test,singularity -c local.2.config --outdir output
Jun-28 20:05:38.375 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 22.10.3
Jun-28 20:05:38.390 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jun-28 20:05:38.397 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-28 20:05:38.398 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-28 20:05:38.400 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jun-28 20:05:38.408 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-28 20:05:38.418 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /root/.nextflow/scm
Jun-28 20:05:39.213 [main] DEBUG nextflow.scm.AssetManager - Git config: /root/.nextflow/assets/nf-core/mag/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/mag.git
Jun-28 20:05:39.228 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jun-28 20:05:39.234 [main] DEBUG nextflow.scm.AssetManager - Git config: /root/.nextflow/assets/nf-core/mag/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/mag.git
Jun-28 20:05:40.074 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /root/.nextflow/assets/nf-core/mag/nextflow.config
Jun-28 20:05:40.076 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /root/local.2.config
Jun-28 20:05:40.076 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /root/.nextflow/assets/nf-core/mag/nextflow.config
Jun-28 20:05:40.076 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /root/local.2.config
Jun-28 20:05:40.084 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,singularity`
Jun-28 20:05:41.266 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,singularity`
Jun-28 20:05:41.280 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [cfc_dev, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, uge, rosalind_uge, lugh, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, tigem, tubingen_apg, google, ipop_up, googlels, eddie, medair, test_host_rm, bi, bigpurple, sbc_sharc, adcra, cedars, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, test_ancient_dna, ceres, munin, arm, rosalind, test_binrefinement, hasta, cfc, uzh, test_hybrid_host_rm, test_busco_auto, ebc, ku_sund_dangpu, ccga_dx, test_hybrid, crick, marvin, biohpc_gen, shifter, mana, mamba, wehi, awsbatch, imperial, maestro, genotoul, test_no_clipping, abims, janelia, nihbiowulf, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, sage, cambridge, podman, cheaha, xanadu, test, computerome, seg_globe, sanger, dkfz, pasteur, test_full, azurebatch, hki, test_adapterremoval, crukmi, docker, engaging, gis, hypatia, psmn, eva, fgcz, conda, crg, singularity, prince, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uct_hpc, aws_tower, binac]
Jun-28 20:05:41.311 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Jun-28 20:05:41.312 [main] INFO  nextflow.cli.CmdRun - Launching `https://github.com/nf-core/mag` [sleepy_allen] DSL2 - revision: 66cf53aff8 [2.3.2]
Jun-28 20:05:41.312 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-28 20:05:41.313 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-28 20:05:41.315 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jun-28 20:05:41.317 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@2e204155] - activable => nextflow.secret.LocalSecretsProvider@2e204155
Jun-28 20:05:41.358 [main] DEBUG nextflow.Session - Session UUID: e8d1bf64-2c14-4a7c-9737-645b7c7928ea
Jun-28 20:05:41.358 [main] DEBUG nextflow.Session - Run name: sleepy_allen
Jun-28 20:05:41.358 [main] DEBUG nextflow.Session - Executor pool size: 10
Jun-28 20:05:41.366 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=30; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-28 20:05:41.398 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 22.10.3 build 5834
  Created: 21-11-2022 21:37 UTC (21:37 BST)
  System: Linux 4.18.0-477.13.1.el8_8.x86_64
  Runtime: Groovy 3.0.13 on Java HotSpot(TM) 64-Bit Server VM 17.0.6+9-LTS-190
  Encoding: UTF-8 (UTF-8)
  Process: [email protected] [192.168.122.8]
  CPUs: 10 - Mem: 15.1 GB (13.3 GB) - Swap: 7.7 GB (7.7 GB)
Jun-28 20:05:41.415 [main] DEBUG nextflow.Session - Work-dir: /root/work [xfs]
Jun-28 20:05:41.430 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-28 20:05:41.439 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-28 20:05:41.539 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-28 20:05:41.546 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 11; maxThreads: 1000
Jun-28 20:05:41.597 [main] DEBUG nextflow.Session - Session start
Jun-28 20:05:41.601 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /root/output/pipeline_info/execution_trace_2023-06-28_20-05-40.txt
Jun-28 20:05:41.608 [main] DEBUG nextflow.Session - Using default localLib path: /root/.nextflow/assets/nf-core/mag/lib
Jun-28 20:05:41.611 [main] DEBUG nextflow.Session - Adding to the classpath library: /root/.nextflow/assets/nf-core/mag/lib
Jun-28 20:05:41.611 [main] DEBUG nextflow.Session - Adding to the classpath library: /root/.nextflow/assets/nf-core/mag/lib/nfcore_external_java_deps.jar
Jun-28 20:05:42.438 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-28 20:05:42.476 [main] INFO  nextflow.Nextflow - 

-�[2m----------------------------------------------------�[0m-
                                        �[0;32m,--.�[0;30m/�[0;32m,-.�[0m
�[0;34m        ___     __   __   __   ___     �[0;32m/,-._.--~'�[0m
�[0;34m  |\ | |__  __ /  ` /  \ |__) |__         �[0;33m}  {�[0m
�[0;34m  | \| |       \__, \__/ |  \ |___     �[0;32m\`-._,-`-,�[0m
                                        �[0;32m`._,._,'�[0m
�[0;35m  nf-core/mag v2.3.2-g66cf53a�[0m
-�[2m----------------------------------------------------�[0m-
�[1mCore Nextflow options�[0m
  �[0;34mrevision                   : �[0;32m2.3.2�[0m
  �[0;34mrunName                    : �[0;32msleepy_allen�[0m
  �[0;34mcontainerEngine            : �[0;32msingularity�[0m
  �[0;34mlaunchDir                  : �[0;32m/root�[0m
  �[0;34mworkDir                    : �[0;32m/root/work�[0m
  �[0;34mprojectDir                 : �[0;32m/root/.nextflow/assets/nf-core/mag�[0m
  �[0;34muserName                   : �[0;32mroot�[0m
  �[0;34mprofile                    : �[0;32mtest,singularity�[0m
  �[0;34mconfigFiles                : �[0;32m/root/.nextflow/assets/nf-core/mag/nextflow.config, /root/local.2.config�[0m

�[1mInput/output options�[0m
  �[0;34minput                      : �[0;32mhttps://raw.githubusercontent.com/nf-core/test-datasets/mag/samplesheets/samplesheet.csv�[0m
  �[0;34moutdir                     : �[0;32moutput�[0m

�[1mInstitutional config options�[0m
  �[0;34mconfig_profile_name        : �[0;32mTest profile�[0m
  �[0;34mconfig_profile_description : �[0;32mMinimal test dataset to check pipeline function�[0m

�[1mMax job request options�[0m
  �[0;34mmax_cpus                   : �[0;32m2�[0m
  �[0;34mmax_memory                 : �[0;32m6.GB�[0m
  �[0;34mmax_time                   : �[0;32m6.h�[0m

�[1mQuality control for short reads options�[0m
  �[0;34mphix_reference             : �[0;32m/root/.nextflow/assets/nf-core/mag/assets/data/GCA_002596845.1_ASM259684v1_genomic.fna.gz�[0m

�[1mQuality control for long reads options�[0m
  �[0;34mlambda_reference           : �[0;32m/root/.nextflow/assets/nf-core/mag/assets/data/GCA_000840245.1_ViralProj14204_genomic.fna.gz�[0m

�[1mTaxonomic profiling options�[0m
  �[0;34mcentrifuge_db              : �[0;32mhttps://raw.githubusercontent.com/nf-core/test-datasets/mag/test_data/minigut_cf.tar.gz�[0m
  �[0;34mkraken2_db                 : �[0;32mhttps://raw.githubusercontent.com/nf-core/test-datasets/mag/test_data/minigut_kraken.tgz�[0m
  �[0;34mskip_krona                 : �[0;32mtrue�[0m
  �[0;34mgtdb                       : �[0;32mfalse�[0m

�[1mBinning options�[0m
  �[0;34mskip_concoct               : �[0;32mtrue�[0m
  �[0;34mmin_length_unbinned_contigs: �[0;32m1�[0m
  �[0;34mmax_unbinned_contigs       : �[0;32m2�[0m

�[1mBin quality check options�[0m
  �[0;34mbusco_reference            : �[0;32mhttps://busco-data.ezlab.org/v5/data/lineages/bacteria_odb10.2020-03-06.tar.gz�[0m
  �[0;34mbusco_clean                : �[0;32mtrue�[0m
  �[0;34mcheckm_db                  : �[0;32mnull�[0m
  �[0;34mgunc_db                    : �[0;32mnull�[0m

!! Only displaying parameters that differ from the pipeline defaults !!
-�[2m----------------------------------------------------�[0m-
If you use nf-core/mag for your analysis please cite:

* The pipeline publication
  https://doi.org/10.1093/nargab/lqac007

* The pipeline
  https://doi.org/10.5281/zenodo.3589527

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/nf-core/mag/blob/master/CITATIONS.md
-�[2m----------------------------------------------------�[0m-
Jun-28 20:05:49.508 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MAG:MAG:ARIA2_UNTAR
Jun-28 20:05:49.509 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:ARIA2_UNTAR` matches process NFCORE_MAG:MAG:ARIA2_UNTAR
Jun-28 20:05:49.513 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.513 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.518 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jun-28 20:05:49.521 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=10; memory=15.1 GB; capacity=10; pollInterval=100ms; dumpInterval=5m
Jun-28 20:05:49.597 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:FASTQC_RAW
Jun-28 20:05:49.598 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:FASTQC_RAW` matches process NFCORE_MAG:MAG:FASTQC_RAW
Jun-28 20:05:49.601 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.601 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.614 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:FASTP
Jun-28 20:05:49.615 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:FASTP` matches process NFCORE_MAG:MAG:FASTP
Jun-28 20:05:49.617 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.617 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.624 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BOWTIE2_HOST_REMOVAL_BUILD|BOWTIE2_HOST_REMOVAL_ALIGN|BOWTIE2_PHIX_REMOVAL_BUILD|BOWTIE2_PHIX_REMOVAL_ALIGN` matches process NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD
Jun-28 20:05:49.626 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.626 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.630 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BOWTIE2_PHIX_REMOVAL_ALIGN` matches process NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_ALIGN
Jun-28 20:05:49.631 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BOWTIE2_HOST_REMOVAL_BUILD|BOWTIE2_HOST_REMOVAL_ALIGN|BOWTIE2_PHIX_REMOVAL_BUILD|BOWTIE2_PHIX_REMOVAL_ALIGN` matches process NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_ALIGN
Jun-28 20:05:49.632 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.632 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.638 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:FASTQC_TRIMMED
Jun-28 20:05:49.638 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:FASTQC_TRIMMED` matches process NFCORE_MAG:MAG:FASTQC_TRIMMED
Jun-28 20:05:49.640 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.640 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.645 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:NANOPLOT_RAW` matches process NFCORE_MAG:MAG:NANOPLOT_RAW
Jun-28 20:05:49.646 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.647 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.652 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:PORECHOP` matches process NFCORE_MAG:MAG:PORECHOP
Jun-28 20:05:49.653 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.653 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.658 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:NANOLYSE` matches process NFCORE_MAG:MAG:NANOLYSE
Jun-28 20:05:49.660 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.660 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.670 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:FILTLONG` matches process NFCORE_MAG:MAG:FILTLONG
Jun-28 20:05:49.671 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.671 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.687 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:NANOPLOT_FILTERED` matches process NFCORE_MAG:MAG:NANOPLOT_FILTERED
Jun-28 20:05:49.689 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.689 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.695 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.695 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.699 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:CENTRIFUGE` matches process NFCORE_MAG:MAG:CENTRIFUGE
Jun-28 20:05:49.700 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.700 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.706 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.706 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.709 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:KRAKEN2` matches process NFCORE_MAG:MAG:KRAKEN2
Jun-28 20:05:49.711 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.711 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.717 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MEGAHIT` matches process NFCORE_MAG:MAG:MEGAHIT
Jun-28 20:05:49.719 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.719 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.728 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:SPADES` matches process NFCORE_MAG:MAG:SPADES
Jun-28 20:05:49.730 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.730 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.744 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:SPADESHYBRID` matches process NFCORE_MAG:MAG:SPADESHYBRID
Jun-28 20:05:49.747 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.747 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.754 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:QUAST` matches process NFCORE_MAG:MAG:QUAST
Jun-28 20:05:49.755 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.755 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.760 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MAG:MAG:PRODIGAL
Jun-28 20:05:49.761 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:PRODIGAL` matches process NFCORE_MAG:MAG:PRODIGAL
Jun-28 20:05:49.762 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.762 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.770 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.770 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.778 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BOWTIE2_ASSEMBLY_ALIGN` matches process NFCORE_MAG:MAG:BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN
Jun-28 20:05:49.779 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.779 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.788 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:BINNING:METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS
Jun-28 20:05:49.788 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS` matches process NFCORE_MAG:MAG:BINNING:METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS
Jun-28 20:05:49.790 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.790 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.798 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.799 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.805 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:BINNING:METABAT2_METABAT2
Jun-28 20:05:49.806 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:METABAT2_METABAT2` matches process NFCORE_MAG:MAG:BINNING:METABAT2_METABAT2
Jun-28 20:05:49.807 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.808 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.817 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:BINNING:MAXBIN2
Jun-28 20:05:49.818 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MAXBIN2` matches process NFCORE_MAG:MAG:BINNING:MAXBIN2
Jun-28 20:05:49.819 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.819 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.823 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_low` matches labels `process_low` for process with name NFCORE_MAG:MAG:BINNING:ADJUST_MAXBIN2_EXT
Jun-28 20:05:49.824 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:ADJUST_MAXBIN2_EXT` matches process NFCORE_MAG:MAG:BINNING:ADJUST_MAXBIN2_EXT
Jun-28 20:05:49.825 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.825 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.833 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_low` matches labels `process_low` for process with name NFCORE_MAG:MAG:BINNING:SPLIT_FASTA
Jun-28 20:05:49.834 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:SPLIT_FASTA` matches process NFCORE_MAG:MAG:BINNING:SPLIT_FASTA
Jun-28 20:05:49.835 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.835 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.840 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MAG:MAG:BINNING:GUNZIP_BINS
Jun-28 20:05:49.842 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.842 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.845 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MAG:MAG:BINNING:GUNZIP_UNBINS
Jun-28 20:05:49.847 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.847 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.855 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MAG_DEPTHS` matches process NFCORE_MAG:MAG:BINNING:MAG_DEPTHS
Jun-28 20:05:49.856 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.856 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.873 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MAG_DEPTHS_PLOT|MAG_DEPTHS_SUMMARY|MAG_DEPTHS_PLOT_REFINED` matches process NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_PLOT
Jun-28 20:05:49.874 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.874 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.878 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MAG_DEPTHS_PLOT|MAG_DEPTHS_SUMMARY|MAG_DEPTHS_PLOT_REFINED` matches process NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_SUMMARY
Jun-28 20:05:49.879 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.879 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.887 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BUSCO_DB_PREPARATION` matches process NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION
Jun-28 20:05:49.888 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.888 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.898 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BUSCO` matches process NFCORE_MAG:MAG:BUSCO_QC:BUSCO
Jun-28 20:05:49.899 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.899 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.908 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BUSCO_SUMMARY|QUAST_BINS|QUAST_BINS_SUMMARY` matches process NFCORE_MAG:MAG:BUSCO_QC:BUSCO_SUMMARY
Jun-28 20:05:49.909 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.909 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.918 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BUSCO_SUMMARY|QUAST_BINS|QUAST_BINS_SUMMARY` matches process NFCORE_MAG:MAG:QUAST_BINS
Jun-28 20:05:49.919 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.919 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.923 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BUSCO_SUMMARY|QUAST_BINS|QUAST_BINS_SUMMARY` matches process NFCORE_MAG:MAG:QUAST_BINS_SUMMARY
Jun-28 20:05:49.924 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.924 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.928 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:CAT` matches process NFCORE_MAG:MAG:CAT
Jun-28 20:05:49.928 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.928 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.933 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_low` matches labels `process_low` for process with name NFCORE_MAG:MAG:CAT_SUMMARY
Jun-28 20:05:49.933 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:CAT_SUMMARY` matches process NFCORE_MAG:MAG:CAT_SUMMARY
Jun-28 20:05:49.934 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.934 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.941 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:BIN_SUMMARY` matches process NFCORE_MAG:MAG:BIN_SUMMARY
Jun-28 20:05:49.941 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.941 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.949 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_low` matches labels `process_low` for process with name NFCORE_MAG:MAG:PROKKA
Jun-28 20:05:49.949 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:PROKKA` matches process NFCORE_MAG:MAG:PROKKA
Jun-28 20:05:49.950 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.950 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.961 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MAG:MAG:CUSTOM_DUMPSOFTWAREVERSIONS
Jun-28 20:05:49.961 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:CUSTOM_DUMPSOFTWAREVERSIONS` matches process NFCORE_MAG:MAG:CUSTOM_DUMPSOFTWAREVERSIONS
Jun-28 20:05:49.962 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.962 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.981 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_medium` matches labels `process_medium` for process with name NFCORE_MAG:MAG:MULTIQC
Jun-28 20:05:49.981 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MULTIQC` matches process NFCORE_MAG:MAG:MULTIQC
Jun-28 20:05:49.982 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jun-28 20:05:49.982 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jun-28 20:05:49.987 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: NANOPLOT_FILTERED, NFCORE_MAG:MAG:PROKKA, BUSCO, NFCORE_MAG:MAG:NANOPLOT_RAW, CHECKM_LINEAGEWF, BOWTIE2_PHIX_REMOVAL_ALIGN, CONCOCT_CONCOCTCOVERAGETABLE, BUSCO_DB_PREPARATION, POOL_SINGLE_READS, NFCORE_MAG:MAG:BUSCO_QC:BUSCO_SUMMARY, NFCORE_MAG:MAG:QUAST_BINS, BCFTOOLS_INDEX_POST, CENTRIFUGE_DB_PREPARATION, NFCORE_MAG:MAG:SPADESHYBRID, BCFTOOLS_CONSENSUS, PROKKA, POOL_SHORT_SINGLE_READS, GUNZIP_UNBINS, GUNZIP, BOWTIE2_ASSEMBLY_ALIGN, QUAST, QUAST_BINS_SUMMARY, NFCORE_MAG:MAG:ARIA2_UNTAR, ADAPTERREMOVAL_SE, BUSCO_SAVE_DOWNLOAD, FILTLONG, BCFTOOLS_INDEX_PRE, DASTOOL_FASTATOCONTIG2BIN_METABAT2, KRAKEN2, ADJUST_MAXBIN2_EXT, KRONA_DB, NFCORE_MAG:MAG:FASTP, DASTOOL_FASTATOCONTIG2BIN_CONCOCT, NFCORE_MAG:MAG:CAT_SUMMARY, COMBINE_CHECKM_TSV, NFCORE_MAG:MAG:QUAST, PORECHOP, NANOPLOT, KRONA, NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD, NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_ALIGN, NFCORE_MAG:MAG:BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN, FASTQC, NFCORE_MAG:MAG:CUSTOM_DUMPSOFTWAREVERSIONS, CAT_DB_GENERATE, GTDBTK_SUMMARY, CONVERT_DEPTHS, MULTIQC, BCFTOOLS_INDEX, DASTOOL_FASTATOCONTIG2BIN_MAXBIN2, MAXBIN2, BIN_SUMMARY, BOWTIE2_REMOVAL_BUILD, NFCORE_MAG:MAG:SPADES, DASTOOL_FASTATOCONTIG2BIN, QUAST_BINS, CONCOCT_CUTUPFASTA, FREEBAYES, MAG_DEPTHS_REFINED, GTDBTK_CLASSIFY, SAMTOOLS_FAIDX, NANOPLOT_RAW, NFCORE_MAG:MAG:CENTRIFUGE, NFCORE_MAG:MAG:PRODIGAL, PYDAMAGE_FILTER, CAT_SUMMARY, PRODIGAL, NFCORE_MAG:MAG:CAT, BCFTOOLS_VIEW, NFCORE_MAG:MAG:BUSCO_QC:BUSCO, RENAME_POSTDASTOOL, MAG_DEPTHS_PLOT_REFINED, COMBINE_TSV, SPADESHYBRID, NFCORE_MAG:MAG:BINNING:METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS, NFCORE_MAG:MAG:BINNING:MAXBIN2, NFCORE_MAG:MAG:QUAST_BINS_SUMMARY, MAG_DEPTHS_PLOT, NFCORE_MAG:MAG:NANOPLOT_FILTERED, ADAPTERREMOVAL, BOWTIE2_ASSEMBLY_BUILD, ADAPTERREMOVAL_PE, NFCORE_MAG:MAG:BINNING:GUNZIP_BINS, NFCORE_MAG:MAG:BINNING:SPLIT_FASTA, FASTP, NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION, BOWTIE2_HOST_REMOVAL_ALIGN, NFCORE_MAG:MAG:CENTRIFUGE_DB_PREPARATION, NFCORE_MAG:MAG:BINNING:METABAT2_METABAT2, GUNC_RUN, PYDAMAGE_ANALYZE, GUNC_DOWNLOADDB, DASTOOL_DASTOOL, FASTQC_TRIMMED, NFCORE_MAG:MAG:FASTQC_TRIMMED, COMBINE_SUMMARY_TSV, FASTQC_RAW, SPLIT_FASTA, CONCOCT_MERGECUTUPCLUSTERING, RENAME_PREDASTOOL, CENTRIFUGE, KRAKEN2_DB_PREPARATION, NFCORE_MAG:MAG:BINNING:CONVERT_DEPTHS, POOL_PAIRED_READS, METABAT2_METABAT2, BOWTIE2_REMOVAL_ALIGN, CONCOCT_CONCOCT, NFCORE_MAG:MAG:PORECHOP, NFCORE_MAG:MAG:BINNING_PREPARATION:BOWTIE2_ASSEMBLY_BUILD, GUNZIP_BINS, NFCORE_MAG:MAG:MEGAHIT, CAT_DB, CAT, NFCORE_MAG:MAG:BINNING:ADJUST_MAXBIN2_EXT, MEGAHIT, NFCORE_MAG:MAG:BIN_SUMMARY, MAG_DEPTHS_SUMMARY, SPADES, BOWTIE2_HOST_REMOVAL_BUILD, MAG_DEPTHS, CHECKM_QA, NFCORE_MAG:MAG:KRAKEN2_DB_PREPARATION, POOL_LONG_READS, NFCORE_MAG:MAG:KRAKEN2, NFCORE_MAG:MAG:FILTLONG, NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_PLOT, GTDBTK_DB_PREPARATION, BOWTIE2_PHIX_REMOVAL_BUILD, MAG_DEPTHS_SUMMARY_REFINED, NFCORE_MAG:MAG:FASTQC_RAW, NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_SUMMARY, GUNC_MERGECHECKM, METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS, NFCORE_MAG:MAG:BINNING:GUNZIP_UNBINS, NFCORE_MAG:MAG:NANOLYSE, NFCORE_MAG:MAG:MULTIQC, BUSCO_SUMMARY, ARIA2, NANOLYSE, ARIA2_UNTAR, CUSTOM_DUMPSOFTWAREVERSIONS, CONCOCT_EXTRACTFASTABINS, FAIDX, NFCORE_MAG:MAG:BINNING:MAG_DEPTHS
Jun-28 20:05:49.995 [main] DEBUG nextflow.Session - Igniting dataflow network (77)
Jun-28 20:05:49.996 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:ARIA2_UNTAR
Jun-28 20:05:49.996 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:FASTQC_RAW
Jun-28 20:05:49.996 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:FASTP
Jun-28 20:05:49.996 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD
Jun-28 20:05:49.996 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_ALIGN
Jun-28 20:05:49.997 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:FASTQC_TRIMMED
Jun-28 20:05:49.997 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:NANOPLOT_RAW
Jun-28 20:05:49.997 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:PORECHOP
Jun-28 20:05:49.997 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:NANOLYSE
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:FILTLONG
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:NANOPLOT_FILTERED
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:CENTRIFUGE_DB_PREPARATION
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:CENTRIFUGE
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:KRAKEN2_DB_PREPARATION
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:KRAKEN2
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:MEGAHIT
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:SPADES
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:SPADESHYBRID
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:QUAST
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:PRODIGAL
Jun-28 20:05:49.998 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING_PREPARATION:BOWTIE2_ASSEMBLY_BUILD
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:METABAT2_JGISUMMARIZEBAMCONTIGDEPTHS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:CONVERT_DEPTHS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:METABAT2_METABAT2
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:MAXBIN2
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:ADJUST_MAXBIN2_EXT
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:SPLIT_FASTA
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:GUNZIP_BINS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:GUNZIP_UNBINS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:MAG_DEPTHS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_PLOT
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BINNING:MAG_DEPTHS_SUMMARY
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO_SUMMARY
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:QUAST_BINS
Jun-28 20:05:49.999 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:QUAST_BINS_SUMMARY
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:CAT
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:CAT_SUMMARY
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:BIN_SUMMARY
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:PROKKA
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:CUSTOM_DUMPSOFTWAREVERSIONS
Jun-28 20:05:50.000 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MAG:MAG:MULTIQC
Jun-28 20:05:50.000 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jun-28 20:05:50.000 [main] DEBUG nextflow.Session - Session await
Jun-28 20:05:50.078 [Actor Thread 2] WARN  nextflow.container.SingularityCache - Singularity cache directory has not been defined -- Remote image will be stored in the path: /root/work/singularity -- Use env variable NXF_SINGULARITY_CACHEDIR to specify a different location
Jun-28 20:05:50.079 [Actor Thread 2] INFO  nextflow.container.SingularityCache - Pulling Singularity image https://depot.galaxyproject.org/singularity/bowtie2:2.4.2--py38h1c8e9b9_1 [cache /root/work/singularity/depot.galaxyproject.org-singularity-bowtie2-2.4.2--py38h1c8e9b9_1.img]
Jun-28 20:05:50.079 [Actor Thread 4] INFO  nextflow.container.SingularityCache - Pulling Singularity image https://depot.galaxyproject.org/singularity/aria2:1.36.0 [cache /root/work/singularity/depot.galaxyproject.org-singularity-aria2-1.36.0.img]
Jun-28 20:05:50.110 [Actor Thread 18] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 1; slices: 1; internal sort time: 0.001 s; external sort time: 0.011 s; total time: 0.012 s
Jun-28 20:05:50.116 [Actor Thread 18] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /tmp/3b8b95bb1286ccf46253890bb6b58185.collect-file
Jun-28 20:05:50.121 [Actor Thread 18] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-14340507848116382675
Jun-28 20:05:50.132 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Copying foreign file https://raw.githubusercontent.com/nf-core/test-datasets/mag/test_data/minigut_cf.tar.gz to work dir: /root/work/stage/04/d80a4fa9fd92fc90639d8409483a7e/minigut_cf.tar.gz
Jun-28 20:05:50.358 [FileTransfer-2] DEBUG nextflow.file.FilePorter - Copying foreign file https://raw.githubusercontent.com/nf-core/test-datasets/mag/test_data/minigut_kraken.tgz to work dir: /root/work/stage/19/ea3953b07c872cb382d5b1cb799e46/minigut_kraken.tgz
Jun-28 20:05:50.437 [Actor Thread 18] INFO  nextflow.container.SingularityCache - Pulling Singularity image https://depot.galaxyproject.org/singularity/ubuntu:20.04 [cache /root/work/singularity/depot.galaxyproject.org-singularity-ubuntu-20.04.img]
Jun-28 20:05:50.712 [Actor Thread 4] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=https://depot.galaxyproject.org/singularity/aria2:1.36.0 path=/root/work/singularity/depot.galaxyproject.org-singularity-aria2-1.36.0.img
Jun-28 20:05:50.714 [Actor Thread 2] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=https://depot.galaxyproject.org/singularity/bowtie2:2.4.2--py38h1c8e9b9_1 path=/root/work/singularity/depot.galaxyproject.org-singularity-bowtie2-2.4.2--py38h1c8e9b9_1.img
Jun-28 20:05:50.730 [Actor Thread 9] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/mag/assets/data/GCA_002596845.1_ASM259684v1_genomic.fna.gz
Jun-28 20:05:50.776 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jun-28 20:05:50.777 [Task submitter] INFO  nextflow.Session - [05/b6c075] Submitted process > NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)
Jun-28 20:05:50.789 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jun-28 20:05:50.793 [Task submitter] INFO  nextflow.Session - [5b/fd6db1] Submitted process > NFCORE_MAG:MAG:ARIA2_UNTAR (https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz)
Jun-28 20:05:50.897 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz); status: COMPLETED; exit: 255; error: -; workDir: /root/work/05/b6c0752fd260da91354d133f6f35ad]
Jun-28 20:05:50.907 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)'

Caused by:
  Process `NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)` terminated with an error exit status (255)

Command executed:

  mkdir bowtie
  bowtie2-build --threads 1 GCA_002596845.1_ASM259684v1_genomic.fna.gz "bt2_index_base"
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD":
      bowtie2: $(echo $(bowtie2 --version 2>&1) | sed 's/^.*bowtie2-align-s version //; s/ .*$//')
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  FATAL:   container creation failed: mount /tmp/nxf.P0ZJLFhZ8C->/tmp/nxf.P0ZJLFhZ8C error: while mounting /tmp/nxf.P0ZJLFhZ8C: destination /tmp/nxf.P0ZJLFhZ8C doesn't exist in container
  cp: cannot stat '.command.trace': No such file or directory

Work dir:
  /root/work/05/b6c0752fd260da91354d133f6f35ad

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
Jun-28 20:05:50.913 [Actor Thread 21] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 20:05:50.917 [Task monitor] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Jun-28 20:05:50.919 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-28 20:05:50.924 [Actor Thread 22] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Jun-28 20:05:51.008 [Actor Thread 18] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=https://depot.galaxyproject.org/singularity/ubuntu:20.04 path=/root/work/singularity/depot.galaxyproject.org-singularity-ubuntu-20.04.img
Jun-28 20:05:51.197 [FileTransfer-3] DEBUG nextflow.file.FilePorter - Copying foreign file https://busco-data.ezlab.org/v5/data/lineages/bacteria_odb10.2020-03-06.tar.gz to work dir: /root/work/stage/b4/557ba1eb90919774b80c642a18eff4/bacteria_odb10.2020-03-06.tar.gz
Jun-28 20:05:53.196 [Actor Thread 10] INFO  nextflow.file.FilePorter - Staging foreign file: https://busco-data.ezlab.org/v5/data/lineages/bacteria_odb10.2020-03-06.tar.gz
Jun-28 20:06:11.935 [SIGINT handler] DEBUG nextflow.Session - Session aborted -- Cause: SIGINT
Jun-28 20:06:11.954 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-28 20:06:11.956 [main] INFO  nextflow.Nextflow - -�[0;35m[nf-core/mag]�[0;31m Pipeline completed with errors�[0m-
Jun-28 20:06:11.959 [main] WARN  n.processor.TaskPollingMonitor - Killing running tasks (1)
Jun-28 20:06:11.966 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=3; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=1; succeedDuration=0ms; failedDuration=66ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=2; peakCpus=2; peakMemory=12 GB; ]
Jun-28 20:06:11.966 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-28 20:06:11.968 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-28 20:06:11.986 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  [{"cpuUsage":null,"process":"BOWTIE2_PHIX_REMOVAL_BUILD","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":{"mean":66,"min":66,"q1":66,"q2":66,"q3":66,"max":66,"minLabel":"NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)","maxLabel":"NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)","q1Label":"NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)","q2Label":"NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)","q3Label":"NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)"},"writes":null},{"cpuUsage":null,"process":"ARIA2_UNTAR","mem":null,"memUsage":null,"timeUsage":null,"vmem":null,"reads":null,"cpu":null,"time":null,"writes":null}]
Jun-28 20:06:12.378 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-28 20:06:12.464 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-28 20:06:12.490 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

System information

Nextflow : Version: 22.10.3 build 5834
Hardware : tested on HPC system (EPYC CPU) and local VM (AMD processor, 10 CPU, 15GB RAM)
OS : RockyLinux 8.7
nf-core-mag-2.3.2
singularity-ce version 3.9.8
Executor : local
$TMPDIR is set to /tmp
@CarstenBaker CarstenBaker added the bug Something isn't working label Jun 28, 2023
@CarstenBaker
Copy link
Author

Get the same issue on Ubuntu 22.04.01

Setting 'export NXF_DEBUG=3' I can see that the process is trying to run nxf_mktemp to /dev/shm but then looking for the files in /tmp so this is causing an issue. Is there an option in nextflow to disable using /dev/shm completely ? Have tried removing the /dev/shm mount point but nextflow still trying to use it ?

.nextflow.log

  • cp -fRL /home/sbiae5/singularity-3.8.7/work/93/9487b1d7995ffe72cd5ddddecdfcbe/bt2_index_base.rev.2.bt2 bt2_index_base.rev.2.bt2
  • set +e
    ++ set +u
    ++ nxf_mktemp /dev/shm
  • ctmp=/dev/shm/nxf.60OBICl0Ok
  • local cout=/dev/shm/nxf.60OBICl0Ok/.command.out
  • mkfifo /dev/shm/nxf.60OBICl0Ok/.command.out
  • local cerr=/dev/shm/nxf.60OBICl0Ok/.command.err
  • mkfifo /dev/shm/nxf.60OBICl0Ok/.command.err
  • tee1=36248
  • tee .command.out
  • tee2=36249
  • tee .command.err
  • pid=36250
  • wait 36250
  • nxf_launch
  • set +u
    ++ nxf_container_env
    ++ cat
  • env - PATH=/root/.sdkman/candidates/java/current/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/root/go/bi
    n SINGULARITYENV_NXF_DEBUG=3 singularity exec --pid -B /root/.nextflow/assets/nf-core/mag/bin -B /home/sbiae5/singularity-3.8.7/work/5f/2cd3c0a8b57c9621013e1c2a47543a -B /tmp/nxf.eDXCAo
    9eWz /home/sbiae5/singularity-3.8.7/nf-core-mag-2.3.2/singularity-images/depot.galaxyproject.org-singularity-bowtie2-2.4.2--py38h1c8e9b9_1.img /bin/bash -c 'cd /tmp/nxf.eDXCAo9eWz; eval
    export PYTHONNOUSERSITE="1"
    export R_PROFILE_USER="/.Rprofile"
    export R_ENVIRON_USER="/.Renviron"
    export JULIA_DEPOT_PATH="/usr/local/share/julia"
    export PATH="$PATH:/root/.nextflow/assets/nf-core/mag/bin"; /bin/bash .command.run nxf_trace'
    WARNING: Skipping mount /usr/local/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
    FATAL: container creation failed: mount /tmp/nxf.eDXCAo9eWz->/tmp/nxf.eDXCAo9eWz error: while mounting /tmp/nxf.eDXCAo9eWz: destination /tmp/nxf.eDXCAo9eWz doesn't exist in containe
    r

@jfy133
Copy link
Member

jfy133 commented Jun 30, 2023

I'm trying this now

@jfy133
Copy link
Member

jfy133 commented Jun 30, 2023

I can confirm the bug:

  FATAL:   container creation failed: mount hook function failure: mount /tmp/nxf.8z9dbXc8N3->/tmp/nxf.8z9dbXc8N3 error: while mounting /tmp/nxf.8z9dbXc8N3: destination /tmp/nxf.8z9dbXc8N3 doesn't exist in container

I wonder if it's something to do with the container...

Googling came across this: nf-core/eager#894 (comment)

Maybe you can see if there is a similar singularity configuration that was recorded as being teh problem?

@CarstenBaker
Copy link
Author

I tried the "mount tmp = no" option and sadly didn't get that far, but should revisit under a virtual machine that I can test on a bit deeper.

If I rename the fastp container to the bowtie container then I don't get the same error, just the expected "bowtie command not found" which would seem to think a /tmp container is possibly working. Still can't see the folder in /tmp or /dev/shm but with the test data maybe it's failing and cleaning too quickly?

@CarstenBaker
Copy link
Author

If I go the opposite direction and rename the bowtie2 image to fastp I can recreate the "destination doesn't exist error" so would agree that it's the container that's the issue (stupidly didn't think of swapping container names before to test).

The other containers work correctly with /tmp so will have to check what's different with the container.

@jfy133
Copy link
Member

jfy133 commented Jun 30, 2023

To be honest we need to replace the local bowtie2 modules with the official nf-core module versions. We are getting there slowly, but MAG is a very complicated pipeline so ti's it's going slowly unfortunately...

@CarstenBaker
Copy link
Author

That does sound a bit of a nightmare with the number of modules required, let us know if you need someone to help with testing.

Just to confirm it is the singularity images that are the issue, looking inside the container it seems to be this symbolic link that causes the issue on /tmp
lrwxrwxrwx 1 root root 3 Nov 7 2020 run -> tmp

To replicate:

cd /tmp
folder=$(mktemp -d nxf.XXXXXX)
cd $folder

singularity exec -B "$PWD"  /mnt/scratch/singularity-images/depot.galaxyproject.org-singularity-bowtie2-2.4.2--py38h1c8e9b9_1.img /bin/bash

container creation failed: mount /tmp/nxf.bNjf5X->/tmp/nxf.bNjf5X error: while mounting /tmp/nxf.bNjf5X: destination /tmp/nxf.bNjf5X doesn't exist in container

I found that by also binding the /tmp folder to /run (this is where the link points) I could get the container to run correctly. I tried setting this in runOptions in nextflow config but it adds the extra Bind option to the end of the command and /tmp binding fails as earlier in the line. Managed to adapt the nf-core/eager/issues/894 comment to turn off autoMounts and manually mount /tmp and $PWD

Here is my example test config file in case anyone else has a similar issue (obviously just a test example)

nextflow run nf-core/mag -profile test -c local.tmp.config --outdir output

singularity {
  cacheDir = '/mnt/scratch/singularity-images'
  enabled = true
  autoMounts = false
  runOptions = "-B /tmp:/run -B $PWD"
}

process {
            executor = 'local'
            scratch = '/tmp/mag_scratch'
            stageInMode = 'copy'
            stageOutMode = 'rsync'

withName: 'BOWTIE2_PHIX_REMOVAL_BUILD|BOWTIE2_PHIX_REMOVAL_ALIGN|BOWTIE2_ASSEMBLY_BUILD' {
            scratch = '/tmp/bowtie'
     }
}

@CarstenBaker
Copy link
Author

CarstenBaker commented Jul 3, 2023 •
edited
Loading

I did have a look at swapping in the newer singularity images from official nf-core modules but some of these still have the similar issue, for example these 2 images:

curl -LO https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0
curl -LO https://depot.galaxyproject.org/singularity/metabat2:2.15--h986a166_1

A lot of the others are working, seems to be images from 2020/2021 have the /run -> /tmp symbolic link, images from late 2021 onwards no longer follow that trend.

@prototaxites
Copy link
Contributor

I have also just encountered this exact same bug running on a new cluster, where I have large amounts of scratch space I'd like to make use of. For me, the pipeline was failing at FASTQC_RAW, so I haven't had a chance to see which other containers are affected.

A lot of the others are working, seems to be images from 2020/2021 have the /run -> /tmp symbolic link, images from late 2021 onwards no longer follow that trend.

That's interesting - I am also having some trouble with an older container from that period: #462. I've rebuilt it (need to test it still), but this might warrant rebuilds of a good chuck of biocontainers from that period?

@jfy133
Copy link
Member

jfy133 commented Oct 9, 2023

To ocme back to this: is the 'best solution' to just update all the container versions?

@prototaxites
Copy link
Contributor

I think it's definitely worth a try, but it will probably depend on what the base image for the container is (busybox/ubuntu? I don't know which of these are affected), and if tools are old/haven't received a recent update, the container images will probably have to be rebuilt before the fix works by updating the build number in the bioconda recipe.

@jfy133
Copy link
Member

jfy133 commented Oct 9, 2023 •
edited
Loading

OK, then maybe we can schedule in 2.6 a mass update of all containers? (And if required request the custom ones...)

@prototaxites
Copy link
Contributor

Sounds good. Is there a quick way to dump a list of all the container modules? Happy to take some time and make a list of all the ones that are 2021 or older

@prototaxites
Copy link
Contributor

prototaxites commented Oct 9, 2023 •
edited
Loading

Assuming a cutoff date for replacement of 01/01/2022:

Module Container Date OK?
local/adjust_maxbin2_ext.nf- multiqc:1.12--pyhdfd78af_0 11-Feb-22 ✅
local/bin_summary.nf- pandas:1.4.3 03-Sep-22 ✅
local/bowtie2_assembly_align.nf- mulled-v2-ac74a7f02cebcfcc07d8e8d1d750af9c83b4d45a:577a697be67b5ae9b16f637fd723b8263a3898b3-0 14-Jan-21
local/bowtie2_assembly_build.nf- bowtie2:2.4.2--py38h1c8e9b9_1 07-Nov-20
local/bowtie2_removal_align.nf- bowtie2:2.4.2--py38h1c8e9b9_1 07-Nov-20
local/bowtie2_removal_build.nf- bowtie2:2.4.2--py38h1c8e9b9_1 07-Nov-20
local/busco_save_download.nf- ubuntu:20.04 08-Feb-21
local/cat.nf- mulled-v2-75e2a26f10cbf3629edf2d1600db3fed5ebe6e04:eae321284604f7dabbdf121e3070bda907b91266-0 15-Mar-21
local/cat_db.nf- ubuntu:20.04 08-Feb-21
local/cat_db_generate.nf- mulled-v2-75e2a26f10cbf3629edf2d1600db3fed5ebe6e04:eae321284604f7dabbdf121e3070bda907b91266-0 15-Mar-21
local/cat_summary.nf- bioawk:1.0--hed695b0_5 27-Oct-19
local/centrifuge.nf- centrifuge:1.0.4_beta--he513fc3_5 15-Jan-20
local/combine_tsv.nf- bioawk:1.0--hed695b0_5 27-Oct-19
local/convert_depths.nf- bioawk:1.0--hed695b0_5 27-Oct-19
local/filtlong.nf- filtlong:0.2.0--he513fc3_3 22-Aug-20
local/gtdbtk_db_preparation.nf- ubuntu:20.04 08-Feb-21
local/gtdbtk_summary.nf- pandas:1.4.3 03-Sep-22 ✅
local/kraken2.nf- kraken2:2.0.8_beta--pl526hc9558a2_2 17-Apr-20
local/kraken2_db_preparation.nf- ubuntu:20.04 08-Feb-21
local/krona.nf- krona:2.7.1--pl526_5 03-Mar-21
local/krona_db.nf- krona:2.7.1--pl526_5 03-Mar-21
local/mag_depths.nf- mulled-v2-e25d1fa2bb6cbacd47a4f8b2308bd01ba38c5dd7:75310f02364a762e6ba5206fcd11d7529534ed6e-0 11-Feb-21
local/mag_depths_summary.nf- pandas:1.4.3 03-Sep-22 ✅
local/nanolyse.nf- nanolyse:1.1.0--py36_1 18-Aug-19
local/nanoplot.nf- nanoplot:1.26.3--py_0 01-Sep-19
local/pool_paired_reads.nf- ubuntu:20.04 08-Feb-21
local/pool_single_reads.nf- ubuntu:20.04 08-Feb-21
local/porechop.nf- porechop:0.2.3_seqan2.1.1--py36h2d50403_3 22-Aug-19
local/quast.nf- quast:5.0.2--py37pl526hb5aa323_2 18-Jul-20
local/quast_bins.nf- quast:5.0.2--py37pl526hb5aa323_2 18-Jul-20
local/quast_bins_summary.nf- ubuntu:20.04 08-Feb-21
local/rename_postdastool.nf- multiqc:1.12--pyhdfd78af_0 11-Feb-22 ✅
local/rename_predastool.nf- multiqc:1.12--pyhdfd78af_0 11-Feb-22 ✅
local/spades.nf- spades:3.15.3--h95f258a_0 24-Jul-21
local/spadeshybrid.nf- spades:3.15.3--h95f258a_0 24-Jul-21
local/split_fasta.nf- mulled-v2-e25d1fa2bb6cbacd47a4f8b2308bd01ba38c5dd7:75310f02364a762e6ba5206fcd11d7529534ed6e-0 11-Feb-21
local/tiara_classify.nf- mulled-v2-1021c2bc41756fa99bc402f461dad0d1c35358c1:b0c847e4fb89c343b04036e33b2daa19c4152cf5-0 12-Jan-23 ✅
local/busco.nf- busco:5.4.3--pyhdfd78af_0': 12-Aug-22 ✅
local/busco_db_preparation.nf- ubuntu:20.04 08-Feb-21
local/busco_summary.nf- pandas:1.4.3 03-Sep-22 ✅
local/centrifuge_db_preparation.nf- ubuntu:20.04 08-Feb-21
local/mag_depths_plot.nf- mulled-v2-d14219255233ee6cacc427e28a7caf8ee42e8c91:0a22c7568e4a509925048454dad9ab37fa8fe776-0 06-Jul-21
local/megahit.nf- megahit:1.2.9--h2e03b76_1 25-Mar-21
nf-core/adapterremoval/main.nf- adapterremoval:2.3.2--hb7ba0dd_0 17-Mar-21
nf-core/aria2/main.nf- aria2:1.36.0 08-Oct-22 ✅
nf-core/bbmap/bbnorm/main.nf- mulled-v2-008daec56b7aaf3f162d7866758142b9f889d690:e8a286b2e789c091bac0a57302cdc78aa0112353-0': 25-Nov-22 ✅
nf-core/bcftools/consensus/main.nf- bcftools:1.17--haef29d1_0': 08-Apr-23 ✅
nf-core/bcftools/index/main.nf- bcftools:1.17--haef29d1_0': 08-Apr-23 ✅
nf-core/bcftools/view/main.nf- bcftools:1.17--haef29d1_0': 08-Apr-23 ✅
nf-core/cat/fastq/main.nf- ubuntu:20.04 08-Feb-21
nf-core/checkm/lineagewf/main.nf- checkm-genome:1.2.1--pyhdfd78af_0 05-Aug-22 ✅
nf-core/checkm/qa/main.nf- checkm-genome:1.2.1--pyhdfd78af_0 05-Aug-22 ✅
nf-core/concoct/concoct/main.nf- concoct:1.1.0--py311h245ed52_4': 09-Aug-23 ✅
nf-core/concoct/concoctcoveragetable/main.nf- concoct:1.1.0--py311h245ed52_4': 09-Aug-23 ✅
nf-core/concoct/cutupfasta/main.nf- concoct:1.1.0--py311h245ed52_4': 09-Aug-23 ✅
nf-core/concoct/extractfastabins/main.nf- concoct:1.1.0--py311h245ed52_4': 09-Aug-23 ✅
nf-core/concoct/mergecutupclustering/main.nf- concoct:1.1.0--py311h245ed52_4': 09-Aug-23 ✅
nf-core/custom/dumpsoftwareversions/main.nf- multiqc:1.15--pyhdfd78af_0 05-Aug-23 ✅
nf-core/dastool/dastool/main.nf- das_tool:1.1.6--r42hdfd78af_0 21-Dec-22 ✅
nf-core/dastool/fastatocontig2bin/main.nf- das_tool:1.1.6--r42hdfd78af_0 21-Dec-22 ✅
nf-core/fastp/main.nf- fastp:0.23.4--h5f740d0_0 31-May-23 ✅
nf-core/fastqc/main.nf- fastqc:0.11.9--0 25-Jan-20
nf-core/freebayes/main.nf- freebayes:1.3.6--hbfe0e7f_2 18-Sep-22 ✅
nf-core/genomad/download/main.nf- genomad:1.5.2--pyhdfd78af_0': 12-May-23 ✅
nf-core/genomad/endtoend/main.nf- genomad:1.5.2--pyhdfd78af_0': 12-May-23 ✅
nf-core/gtdbtk/classifywf/main.nf- gtdbtk:2.1.1--pyhdfd78af_1 01-Aug-22 ✅
nf-core/gunc/downloaddb/main.nf- gunc:1.0.5--pyhdfd78af_0 30-Oct-21
nf-core/gunc/mergecheckm/main.nf- gunc:1.0.5--pyhdfd78af_0 30-Oct-21
nf-core/gunc/run/main.nf- gunc:1.0.5--pyhdfd78af_0 30-Oct-21
nf-core/gunzip/main.nf- ubuntu:20.04 08-Feb-21
nf-core/maxbin2/main.nf- maxbin2:2.2.7--he1b5a44_2 07-Nov-20
nf-core/metabat2/jgisummarizebamcontigdepths/main.nf- metabat2:2.15--h986a166_1 04-Jul-20
nf-core/metabat2/metabat2/main.nf- metabat2:2.15--h986a166_1 04-Jul-20
nf-core/metaeuk/easypredict/main.nf- metaeuk:6.a5d39d9--pl5321hf1761c0_2': 12-May-23 ✅
nf-core/mmseqs/databases/main.nf- mmseqs2:14.7e284--pl5321h6a68c12_2': 16-May-23 ✅
nf-core/multiqc/main.nf- multiqc:1.15--pyhdfd78af_0 05-Aug-23 ✅
nf-core/prodigal/main.nf- mulled-v2-2e442ba7b07bfa102b9cf8fac6221263cd746ab8:57f05cfa73f769d6ed6d54144cb3aa2a6a6b17e0-0 27-01-22 ✅
nf-core/prokka/main.nf- prokka%3A1.14.6--pl5321hdfd78af_4 24-Mar-22 ✅
nf-core/pydamage/analyze/main.nf- pydamage:0.70--pyhdfd78af_0 11-Mar-22 ✅
nf-core/pydamage/filter/main.nf- pydamage:0.70--pyhdfd78af_0 11-Mar-22 ✅
nf-core/samtools/faidx/main.nf- samtools:1.17--h00cdaf9_0 08-Apr-23 ✅
nf-core/seqtk/mergepe/main.nf- seqtk:1.3--h5bf99c6_3 27-Mar-21
nf-core/tiara/tiara/main.nf- tiara:1.0.3 03-Feb-23 ✅

@jfy133
Copy link
Member

jfy133 commented Oct 9, 2023

Nice, I guess the next thing to do is to identify which local modules are pretty close to official nf-core ones, and also switch them out (and presumably wil have the latest container too)

@CarstenBaker
Copy link
Author

Before I turned off automounts I started swapping containers out for alternative versions, there are some official nf-core versions that have the same issue (fastqc and metabat as an example), but they do have alternative versions on https://depot.galaxyproject.org/singularity which worked correctly.

I got stuck when I hit the mulled-v2 images as some of these didn't have alternative versions I could easily swap in.

@prototaxites
Copy link
Contributor

prototaxites commented Oct 13, 2023 •
edited
Loading

I got stuck when I hit the mulled-v2 images as some of these didn't have alternative versions I could easily swap in.

I think all that's needed is a PR to the multi-package-containers repo with an update to the relevant hash.tsv line, increasing the build number (last entry on the line). Though this might be base image dependent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jfy133 @prototaxites @CarstenBaker