Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

annotate_esm error #63

Open
yotsa opened this issue Apr 23, 2024 · 11 comments
Open

annotate_esm error #63

yotsa opened this issue Apr 23, 2024 · 11 comments

Comments

@yotsa
Copy link

yotsa commented Apr 23, 2024

Hi,

I installed CADD v 1.7 and downloaded the annotations/prescored files separately and put them in the appropriate folders. I then run ./CADD.sh test/input.vcf to test the software, then I get this error

CADD-v1.7 (c) University of Washington, Hudson-Alpha Institute for Biotechnology and Berlin Institute of Health at Charité - Univ
ersitätsmedizin Berlin 2013-2023. All rights reserved.
Running snakemake pipeline:
snakemake /tmp/tmp.5xupN9W1bE/input.tsv.gz --use-conda --conda-prefix /RUN/CADD-scripts-1.7/envs/conda --cores 1
--configfile /RUN/CADD-scripts-1.7/config/config_GRCh38_v1.7_noanno.yml --snakefile /RUN/CADD-scripts-1.7/Snakefile -q
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job         count
--------  -------
join            1
prepare         1
prescore        1
total           3

Select jobs to execute...
Activating conda environment: envs/conda/000a02fda4f9a3dacbe9a7c94df2b69c_
Select jobs to execute...
Activating conda environment: envs/conda/000a02fda4f9a3dacbe9a7c94df2b69c_
Removing temporary output /tmp/tmp.5xupN9W1bE/input.prepared.vcf.
Select jobs to execute...
Activating conda environment: envs/conda/fb032738807041aeaadfd1bd59628e1f_
Smartmatch is experimental at /RUN/CADD-scripts-1.7/envs/conda/fb032738807041aeaadfd1bd59628e1f_/share/ensembl-vep-110.1-0/module
s/Bio/EnsEMBL/VEP/AnnotationSource/File.pm line 472.
Removing temporary output /tmp/tmp.5xupN9W1bE/input.novel.vcf.
Select jobs to execute...
Activating conda environment: envs/conda/57cd93e005aa8f80b32c43193d51c129_
[Tue Apr 23 09:22:54 2024]
Error in rule annotate_esm:
    jobid: 10
    input: /tmp/tmp.5xupN9W1bE/input.vep.vcf.gz, data/annotations/GRCh38_v1.7/esm/esm1v_t33_650M_UR90S_1.pt, data/annotations/GRC
h38_v1.7/esm/esm1v_t33_650M_UR90S_2.pt, data/annotations/GRCh38_v1.7/esm/esm1v_t33_650M_UR90S_3.pt, data/annotations/GRCh38_v1.7/
esm/esm1v_t33_650M_UR90S_4.pt, data/annotations/GRCh38_v1.7/esm/esm1v_t33_650M_UR90S_5.pt, data/annotations/GRCh38_v1.7/esm/pep.1
10.fa
    output: /tmp/tmp.5xupN9W1bE/input.esm_missens.vcf.gz, /tmp/tmp.5xupN9W1bE/input.esm_frameshift.vcf.gz, /tmp/tmp.5xupN9W1bE/in
put.esm.vcf.gz
    log: /tmp/tmp.5xupN9W1bE/input.annotate_esm.log (check log file(s) for error details)
    conda-env: /RUN/CADD-scripts-1.7/envs/conda/57cd93e005aa8f80b32c43193d51c129_
    shell:

        model_directory=`dirname data/annotations/GRCh38_v1.7/esm/esm1v_t33_650M_UR90S_1.pt`;
        model_directory=`dirname $model_directory`;

        python /RUN/CADD-scripts-1.7/src/scripts/lib/tools/esmScore/esmScore_missense_av_fast.py         --input /tmp/tmp.5xupN9W
1bE/input.vep.vcf.gz         --transcripts data/annotations/GRCh38_v1.7/esm/pep.110.fa         --model-directory $model_directory
 --model esm1v_t33_650M_UR90S_1  --model esm1v_t33_650M_UR90S_2  --model esm1v_t33_650M_UR90S_3  --model esm1v_t33_650M_UR90S_4
--model esm1v_t33_650M_UR90S_5          --output /tmp/tmp.5xupN9W1bE/input.esm_missens.vcf.gz --batch-size 1 &> /tmp/tmp.5xupN9W1
bE/input.annotate_esm.log

        python /RUN/CADD-scripts-1.7/src/scripts/lib/tools/esmScore/esmScore_frameshift_av.py         --input /tmp/tmp.5xupN9W1bE
/input.esm_missens.vcf.gz         --transcripts data/annotations/GRCh38_v1.7/esm/pep.110.fa         --model-directory $model_dire
ctory --model esm1v_t33_650M_UR90S_1  --model esm1v_t33_650M_UR90S_2  --model esm1v_t33_650M_UR90S_3  --model esm1v_t33_650M_UR90
S_4  --model esm1v_t33_650M_UR90S_5          --output /tmp/tmp.5xupN9W1bE/input.esm_frameshift.vcf.gz --batch-size 1 &>> /tmp/tmp
.5xupN9W1bE/input.annotate_esm.log

        python /RUN/CADD-scripts-1.7/src/scripts/lib/tools/esmScore/esmScore_inFrame_av.py         --input /tmp/tmp.5xupN9W1bE/in
put.esm_frameshift.vcf.gz         --transcripts data/annotations/GRCh38_v1.7/esm/pep.110.fa         --model-directory $model_dire
ctory --model esm1v_t33_650M_UR90S_1  --model esm1v_t33_650M_UR90S_2  --model esm1v_t33_650M_UR90S_3  --model esm1v_t33_650M_UR90
S_4  --model esm1v_t33_650M_UR90S_5          --output /tmp/tmp.5xupN9W1bE/input.esm.vcf.gz --batch-size 1 &>> /tmp/tmp.5xupN9W1bE
/input.annotate_esm.log

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-04-23T092250.542307.snakemake.log

Can you diagnose what's wrong and how to solve this?

Thank you
Yot

@zihhuafang
Copy link

zihhuafang commented Apr 24, 2024

I had the same issue. Below is the error from log:

Activating conda environment: .snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_
Traceback (most recent call last):
  File "CADD_scripts/src/scripts/lib/tools/esmScore/esmScore_missense_av_fast.py", line 17, in <module>
    import torch
  File ".snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/__init__.py", line 218, in <module>
    from torch._C import *  # noqa: F403
ImportError: .snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent
Traceback (most recent call last):
  File "CADD_scripts/src/scripts/lib/tools/esmScore/esmScore_frameshift
_av.py", line 21, in <module>
    import torch
  File ".snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/__init__.py", line 218, in <module>
    from torch._C import *  # noqa: F403
ImportError: .snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent
Traceback (most recent call last):
  File "CADD_scripts/src/scripts/lib/tools/esmScore/esmScore_inFrame_av.py", line 19, in <module>
    import torch
  File ".snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/__init__.py", line 218, in <module>
    from torch._C import *  # noqa: F403
ImportError: .snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent

Seems like from torch._C import * is not working.

@visze
Copy link
Collaborator

visze commented Apr 25, 2024

@yotsa To figure out what went wrong we need the log of the rule that failed. In your case it is rule annotate_esm /tmp/tmp.5xupN9W1bE/input.annotate_esm.log. Log might be already deleted because in the tmp directory. Maybe rerun again.

@zihhuafang Not sure if it is the same error because right now we do not know the exact caus eof the faile. Just that the rule exited unexpected. About you error: seems that the conda environment is not correct. Did you use conda or mamba? (recommend to use mamba). And the channel priority have to be set to strict.

@yotsa
Copy link
Author

yotsa commented Apr 25, 2024 via email

@visze
Copy link
Collaborator

visze commented Apr 25, 2024

Don't use the CADD.sh script :-)

You can run snakemake seperately as described also in the readme. E.g. via snakemake test/input.tsv.gz --use-conda --conda-prefix envs/conda --configfile config/config_GRCh38_v1.7.yml --snakefile Snakefile -c 1

@zihhuafang
Copy link

zihhuafang commented Apr 25, 2024

I confirmed that I use mamba and set the channel priority to strict.
I tried again and added --conda-frontend mamba to be sure that envs will be installed by mamba though it should be default. I got the same error.

I attached the list of packages installed by snakemake (7.32.3).

# packages in environment at .snakemake/conda/ab3bc29d2de38afcf46948cc1048db3c_:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
biopython                 1.83            py310h2372a71_0    conda-forge
blas                      2.122                       mkl    conda-forge
blas-devel                3.9.0            22_linux64_mkl    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.28.1               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cuda                      11.6.1                        0    nvidia
cuda-cccl                 11.6.55              hf6102b2_0    nvidia
cuda-command-line-tools   11.6.2                        0    nvidia
cuda-compiler             11.6.2                        0    nvidia
cuda-cudart               11.6.55              he381448_0    nvidia
cuda-cudart-dev           11.6.55              h42ad0f4_0    nvidia
cuda-cuobjdump            11.6.124             h2eeebcb_0    nvidia
cuda-cupti                11.6.124             h86345e5_0    nvidia
cuda-cuxxfilt             11.6.124             hecbf4f6_0    nvidia
cuda-driver-dev           11.6.55                       0    nvidia
cuda-gdb                  12.4.127                      0    nvidia
cuda-libraries            11.6.1                        0    nvidia
cuda-libraries-dev        11.6.1                        0    nvidia
cuda-memcheck             11.8.86                       0    nvidia
cuda-nsight               12.4.127                      0    nvidia
cuda-nsight-compute       12.4.1                        0    nvidia
cuda-nvcc                 11.6.124             hbba6d2d_0    nvidia
cuda-nvdisasm             12.4.127                      0    nvidia
cuda-nvml-dev             11.6.55              haa9ef22_0    nvidia
cuda-nvprof               12.4.127                      0    nvidia
cuda-nvprune              11.6.124             he22ec0a_0    nvidia
cuda-nvrtc                11.6.124             h020bade_0    nvidia
cuda-nvrtc-dev            11.6.124             h249d397_0    nvidia
cuda-nvtx                 11.6.124             h0630a44_0    nvidia
cuda-nvvp                 12.4.127                      0    nvidia
cuda-runtime              11.6.1                        0    nvidia
cuda-samples              11.6.101             h8efea70_0    nvidia
cuda-sanitizer-api        12.4.127                      0    nvidia
cuda-toolkit              11.6.1                        0    nvidia
cuda-tools                11.6.1                        0    nvidia
cuda-visual-tools         11.6.1                        0    nvidia
curl                      7.88.1               hdc1c0ab_1    conda-forge
fair-esm                  2.0.0                    pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
freetype                  2.12.1               h267a509_2    conda-forge
gds-tools                 1.9.1.3                       0    nvidia
gmp                       6.3.0                h59595ed_1    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
htslib                    1.10                 h244ad75_0    bioconda
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 hfd0df8a_0    conda-forge
ld_impl_linux-64          2.40                 h55db66e_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0            22_linux64_mkl    conda-forge
libcblas                  3.9.0            22_linux64_mkl    conda-forge
libcublas                 11.9.2.110           h5e84587_0    nvidia
libcublas-dev             11.9.2.110           h5c901ab_0    nvidia
libcufft                  10.7.1.112           hf425ae0_0    nvidia
libcufft-dev              10.7.1.112           ha5ce4c0_0    nvidia
libcufile                 1.9.1.3                       0    nvidia
libcufile-dev             1.9.1.3                       0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcurand-dev             10.3.5.147                    0    nvidia
libcurl                   7.88.1               hdc1c0ab_1    conda-forge
libcusolver               11.3.4.124           h33c3c4e_0    nvidia
libcusparse               11.7.2.124           h7538f96_0    nvidia
libcusparse-dev           11.7.2.124           hbbe9722_0    nvidia
libdeflate                1.17                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               hc881cc4_6    conda-forge
libgfortran-ng            13.2.0               h69a702a_6    conda-forge
libgfortran5              13.2.0               h43f5ff8_6    conda-forge
libhwloc                  2.10.0          default_h2fb2949_1000    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0            22_linux64_mkl    conda-forge
liblapacke                3.9.0            22_linux64_mkl    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnpp                    11.6.3.124           hd2722f0_0    nvidia
libnpp-dev                11.6.3.124           h3c42840_0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjpeg                 11.6.2.124           hd473ad6_0    nvidia
libnvjpeg-dev             11.6.2.124           hb5906b9_0    nvidia
libpng                    1.6.43               h2797004_0    conda-forge
libsqlite                 3.45.3               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h95c4c6d_6    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.6               h232c23b_2    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvm-openmp               18.1.3               h4dfa4b3_0    conda-forge
mkl                       2024.1.0           ha957f24_692    conda-forge
mkl-devel                 2024.1.0           ha770c72_692    conda-forge
mkl-include               2024.1.0           ha957f24_692    conda-forge
ncurses                   6.4.20240210         h59595ed_0    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
nsight-compute            2024.1.1.4                    0    nvidia
numpy                     1.26.4          py310hb13e2d6_0    conda-forge
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
pillow                    9.4.0           py310h023d228_1    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.14         hd12c33a_0_cpython    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
pytorch-cuda              11.6                 h867d48c_1    pytorch
pytorch-mutex             1.0                        cuda    pytorch
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
setuptools                69.5.1             pyhd8ed1ab_0    conda-forge
tbb                       2021.12.0            h00ab1b0_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
torchaudio                0.13.1              py310_cu116    pytorch
torchvision               0.14.1              py310_cu116    pytorch
typing_extensions         4.11.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge


@raghvendra44
Copy link

So what's the final fix? did any of you figure out?
let me know guys, Even i am facing the same issue!

@raghvendra44
Copy link

I confirmed that I use mamba and set the channel priority to strict.

how did you set the priority? and confirm that you are using mamba itself?

@raghvendra44
Copy link

raghvendra44 commented Apr 29, 2024

Even my log file prints the same error.

Traceback (most recent call last):
  File "/gpfs/data/user/raghvendra/CADD-1.7/CADD-scripts/src/scripts/lib/tools/esmScore/esmScore_missense_av_fast.py", line 17, in <module>
    import torch
  File "/gpfs/data/user/raghvendra/CADD-1.7/CADD-scripts/envs/conda/e1f044d61505834b60c1d2fffa15eda7_/lib/python3.10/site-packages/torch/__init__.py", line 218, in <module>
    from torch._C import *  # noqa: F403
ImportError: /gpfs/data/user/raghvendra/CADD-1.7/CADD-scripts/envs/conda/e1f044d61505834b60c1d2fffa15eda7_/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent

not sure how to go ahead!
any suggestions?

@ThorbenMaa
Copy link
Contributor

seems to be a recent bug with torch and mkl 2024.1.0 (new mkl release from 26th of March), see here: pytorch/pytorch#123097 and here https://pypi.org/project/mkl/#history.

Apparently downgrading to mkl 2024.0.0 solves it for now. Can you try this by manually going into the respective environment (envs/conda/57cd93e005aa8f80b32c43193d51c129_ in your case)? I guess they will fix it with the next mkl release :-)

@raghvendra44
Copy link

seems to be a recent bug with torch and mkl 2024.1.0 (new mkl release from 26th of March), see here: pytorch/pytorch#123097 and here https://pypi.org/project/mkl/#history.

Apparently downgrading to mkl 2024.0.0 solves it for now. Can you try this by manually going into the respective environment (envs/conda/57cd93e005aa8f80b32c43193d51c129_ in your case)? I guess they will fix it with the next mkl release :-)

Hey, I tried convert the mkl 2024.1.0 to 2024.0.0
Initially I faced a lot of issues, then after deleting the mkl 2024.1.0 lead to deleting its dependencies as well. And then upon installing the 2024.0.0 manually helped but then since the dependencies were deleted, I had to restore them (I was able to do that coz I had noted the dependencies and its versions before deleting them). Doing all this was quite an exercise but finally it worked.

After all this, I faced another issue! Which has been raised and closed i.e mmsplice1.0.1 but hasn't been resolved yet. The issue is that installing mmsplice1.0.1 has too many issues with its dependency libraries and upon installing the latest version of mmsplice does work out at the end but the predicted values arent right.

@visze
Copy link
Collaborator

visze commented Jul 16, 2024

I released a new CADD-scripts version v1.7.1. Maybe you try that one. Now it is recommended to use apptainer/singularity and all environments are packed within a container and no conda builds are needed (container is 17GB large). You also need now snakemake 8.

Also I updated the environments. So If you use mamba/conda instead I hope you will not face the issues you had above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants