[Bug]: Pipeline not working on Snakemake version 8, running without internet connection

[fawaz-dabbaghieh]

Contact Details

[email protected]

What happened?

I was trying to run the test pipeline, I created a new conda environment following the Usage instruction, but I am getting an NotImplementedError and it seems to be related to the new Snakemake version 8, as I was able to circumvent this error when using Snakemake version 7. The error I am getting with Snakemake version 8 is pasted in "Relevant log output".

The other issue is that when running with Snakemake version 7 I can run the dry run. However, when I run the complete test, it halts and then exits due to requests.exceptions.ConnectionError:. Our HPC is not connected to the internet. Is there a way to run the pipeline offline? I.e. download all the requirements separately so I am not forced to have internet access when running the pipeline?

Thank you very much for your help

Steps To Reproduce

Basic conda environment using the .tests/config/simple_config.yaml config file

Mosaicatcher-pipeline Version

1.5.1 (Default)

Command used

$ snakemake     --cores 1     --configfile .tests/config/simple_config.yaml     --profile workflow/snakemake_profiles/local/conda/     --report report.zip     --report-stylesheet workflow/report/custom-stylesheet.css

How did you run the pipeline?

Conda only

What did you use to run the pipeline? (local execution, HPC, cloud)

HPC in conda environment

Pipeline configuration file

.tests/config/simple_config.yaml

Relevant log output

Using profile workflow/snakemake_profiles/local/conda/ for setting default command line arguments.
Config file config/config.yaml is extended by additional config specified via the command line.
Traceback (most recent call last):
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/cli.py", line 1893, in args_to_api
    dag_api = workflow_api.dag(
              ^^^^^^^^^^^^^^^^^
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/api.py", line 326, in dag
    return DAGApi(
           ^^^^^^^
  File "<string>", line 6, in __init__
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/api.py", line 436, in __post_init__
    self.workflow_api._workflow.dag_settings = self.dag_settings
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/api.py", line 383, in _workflow
    workflow.include(
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/workflow.py", line 1379, in include
    exec(compile(code, snakefile.get_path_or_uri(), "exec"), self.globals)
  File "/gpfs/project/projects/medbioinf/users/fawaz/cancer_project/paper_analysis/mosaicatcher-pipeline/workflow/Snakefile", line 80, in <module>
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/workflow.py", line 1379, in include
    exec(compile(code, snakefile.get_path_or_uri(), "exec"), self.globals)
  File "/gpfs/project/projects/medbioinf/users/fawaz/cancer_project/paper_analysis/mosaicatcher-pipeline/workflow/rules/external_data.smk", line 4, in <module>
    HTTP = HTTPRemoteProvider()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fawaz/miniconda3/envs/mosacatcher/lib/python3.12/site-packages/snakemake/remote/__init__.py", line 8, in __init__
    raise NotImplementedError(
NotImplementedError: Remote providers have been replaced by Snakemake storage plugins. Please use the corresponding storage plugin instead (snakemake-storage-plugin-*).

Anything else?

No response

[weber8thomas]

Hi @fawaz-dabbaghieh, thanks for reaching and sorry for the delay!

I'm aware of the issues regarding snakemake v8 and will update the guidelines concerning MosaiCatcher. I'm working on a fix and raised an issue on the snakemake side but nothing is moving at the moment. The best workaround is to install snakemake 7.32.4 that should be compatible with the latest versions of the pipeline.

Regarding the offiline mode, I see your point, snakemake is looking at the files that are hosted on different providers, I can try to work on something to disable these rules using an extra argument, once the pipeline was correctly configured. I'll update you once this is done.

Cheers!

[fawaz-dabbaghieh]

Hi Thomas,

No problem, I just wanted to mention the Snakemake version 8 in case other users face this.

As for the internet access, I know my situation might be a special one and not general. But after some work I was able to edit and tweak both this pipeline and Ashley's qc pipeline to get them to work without direct connection to the internet, so it wasn't impossible :)

I will look into the edits I did and send them to you, you can then choose if you'd want to integrate them or not (I don't want to do a pull request as some of these fixes might be somewhat hacky). Worse case, you can put my edits somewhere in the documentation and if this situation is ever faced by someone else, they can follow what I've done to get it to work.

Best!