InternalError: cudaGetDevice() failed #762

cosimogonnelli · 2022-05-26T15:43:42Z

cosimogonnelli
May 26, 2022

Problem description

Hi All,

I am encountering the following error when running SLEAP:

tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: initialization error

I run:

tensorflow version:			2.6.3
cudatoolkit                            11.3.1               
cudnn                                   8.2.1.32

Please let me know if you need more information.

Thanks for your help!

Expected behaviour

Actual behaviour

Your personal set up

OS:

system:			Linux, x86_64, 3.10.0-693.5.2.el7.x86_64, #1 SMP Fri Oct 20 20:32:50 UTC 2017

Version(s):

sleap version:			1.2.2
python:		        	3.7.12

SLEAP installation method (listed here):

Environment packages

Launch SLEAP conda environment and start evaluation...
>>> CHECK SLEAP SUPPORT <<<
GPUs: 4/4 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:1
         Available: True
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:2
         Available: True
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:3
         Available: True
        Initalized: False
     Memory growth: None
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]

==========SYSTEM==========

utc:			2022-05-26 14:48:51.428716
python:			3.7.12
system:			Linux, x86_64, 3.10.0-693.5.2.el7.x86_64, #1 SMP Fri Oct 20 20:32:50 UTC 2017
path:			/home/cg226285e/anaconda/envs/sleap/bin:/home/cg226285e/anaconda/condabin:/opt/openmpi/bin:/usr/local/bin:/usr/bin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/usr/local/sbin:/usr/sbin:/home/cg226285e/.local/bin:/home/cg226285e/bin

==========IMPORTS==========

sleap import:			True
sleap path:			/home/cg226285e/anaconda/envs/sleap/lib/python3.7/site-packages/sleap/__init__.py
sleap version:			1.2.2
pyside2 import:			True
pyside path:			/home/cg226285e/anaconda/envs/sleap/lib/python3.7/site-packages/PySide2/__init__.py
call to /home/cg226285e/anaconda/envs/sleap/bin/python failed
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.


cv2 import:			True

==========GIT==========

d3bacf0616eabca30dfd6c5e6acc128fb541224b

master


==========TENSORFLOW==========

tensorflow import:			True
tensorflow version:			2.6.3
tensorflow path:			/home/cg226285e/anaconda/envs/sleap/lib/python3.7/site-packages/tensorflow/__init__.py
gpus:			[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]

==========CONDA==========

# packages in environment at /home/cg226285e/anaconda/envs/sleap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.15.0                   pypi_0    pypi
aom                       3.3.0                h27087fc_1    conda-forge
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
backports-zoneinfo        0.2.1                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4                    pypi_0    pypi
cattrs                    1.1.1                    pypi_0    pypi
certifi                   2021.10.8                pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
clang                     5.0                      pypi_0    pypi
colorama                  0.4.4                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
cuda-nvcc                 11.3.58              h2467b9f_0    nvidia
cudatoolkit               11.3.1               ha36c431_9    nvidia
cudnn                     8.2.1.32             h86fa8c9_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
efficientnet              1.0.0                    pypi_0    pypi
ffmpeg                    5.0.1                h594f047_0    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.31.2                   pypi_0    pypi
freetype                  2.10.4               h0708190_1    conda-forge
gast                      0.4.0                    pypi_0    pypi
geos                      3.9.1                h9c3ff4c_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.44.0                   pypi_0    pypi
h5py                      3.1.0           nompi_py37h1e651dc_100    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.3                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imageio                   2.15.0                   pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.11.1                   pypi_0    pypi
jbig                      2.1               h7f98852_2003    conda-forge
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h7f98852_0    conda-forge
jsmin                     3.0.1                    pypi_0    pypi
jsonpickle                1.2                      pypi_0    pypi
keras                     2.6.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.2                    pypi_0    pypi
krb5                      1.19.3               h3790be6_0    conda-forge
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libcurl                   7.82.0               h7bff187_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libdrm                    2.4.109              h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_15    conda-forge
libgfortran-ng            11.2.0              h69a702a_15    conda-forge
libgfortran5              11.2.0              h5c6108e_15    conda-forge
libgomp                   11.2.0              h1d223b6_15    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libpciaccess              0.16                 h516909a_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_15    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libva                     2.14.0               h7f98852_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.13               h22db469_0    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markdown                  3.3.6                    pypi_0    pypi
matplotlib                3.5.1                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.6.3                    pypi_0    pypi
numpy                     1.19.5           py37h3e96413_3    conda-forge
oauthlib                  3.2.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
opencv-python             4.5.5.62                 pypi_0    pypi
opencv-python-headless    4.5.5.62                 pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1n               h166bdaf_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3                     pypi_0    pypi
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pillow                    8.4.0            py37h0f21c89_0    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.9.0                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.11.2                   pypi_0    pypi
pykalman                  0.9.5                    pypi_0    pypi
pyparsing                 3.0.7                    pypi_0    pypi
pyside2                   5.14.1                   pypi_0    pypi
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.6                      pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0              pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pyzmq                     22.3.0                   pypi_0    pypi
qimage2ndarray            1.8.3                    pypi_0    pypi
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rich                      10.16.1                  pypi_0    pypi
scikit-image              0.19.2                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
seaborn                   0.11.2                   pypi_0    pypi
segmentation-models       1.0.1                    pypi_0    pypi
setuptools                59.8.0           py37h89c1867_1    conda-forge
setuptools-scm            6.3.2                    pypi_0    pypi
shapely                   1.7.1            py37h48c49eb_5    conda-forge
shiboken2                 5.14.1                   pypi_0    pypi
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sleap                     1.2.2                    pypi_0    pypi
sqlite                    3.38.2               h4ff8645_0    conda-forge
svt-av1                   0.9.1                h27087fc_0    conda-forge
tensorboard               2.6.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.6.3                    pypi_0    pypi
tensorflow-estimator      2.6.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1                    pypi_0    pypi
typing-extensions         3.10.0.2                 pypi_0    pypi
tzdata                    2022.1                   pypi_0    pypi
tzlocal                   4.2                      pypi_0    pypi
urllib3                   1.26.8                   pypi_0    pypi
werkzeug                  2.0.3                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
x264                      1!161.3030           h7f98852_1    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.7.0                    pypi_0    pypi
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge


==========PIP==========

absl-py==0.15.0
astunparse==1.6.3
attrs==21.2.0
backports.zoneinfo==0.2.1
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools==4.2.4
cattrs==1.1.1
certifi==2021.10.8
charset-normalizer==2.0.12
clang==5.0
colorama==0.4.4
commonmark==0.9.1
cycler==0.11.0
efficientnet==1.0.0
flatbuffers==1.12
fonttools==4.31.2
gast==0.4.0
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.44.0
h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1604753641401/work
idna==3.3
image-classifiers==1.0.0
imageio==2.15.0
imgaug==0.4.0
imgstore==0.2.9
importlib-metadata==4.11.1
joblib==1.1.0
jsmin==3.0.1
jsonpickle==1.2
keras==2.6.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.4.2
Markdown==3.3.6
matplotlib==3.5.1
networkx==2.6.3
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1649281349439/work
oauthlib==3.2.0
olefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/work
opencv-python @ git+https://github.com/talmolab/wrap_opencv-python-headless.git@ede49f6a23a73033216339f29515e59d594ba921
opencv-python-headless==4.5.5.62
opt-einsum==3.3.0
packaging==21.3
pandas==1.3.5
Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1636558793805/work
protobuf==3.19.4
psutil==5.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.11.2
pykalman==0.9.5
pyparsing==3.0.7
PySide2==5.14.1
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-rapidjson==1.6
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1647961439546/work
pytz-deprecation-shim==0.1.0.post0
PyWavelets==1.3.0
PyYAML==6.0
pyzmq==22.3.0
qimage2ndarray==1.8.3
requests==2.27.1
requests-oauthlib==1.3.1
rich==10.16.1
rsa==4.8
scikit-image==0.19.2
scikit-learn==1.0.2
scikit-video==1.1.11
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1637806658031/work
seaborn==0.11.2
segmentation-models==1.0.1
setuptools-scm==6.4.2
Shapely @ file:///home/conda/feedstock_root/build_artifacts/shapely_1628205363317/work
shiboken2==5.14.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
sleap==1.2.2
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.6.3
tensorflow-estimator==2.6.0
termcolor==1.1.0
threadpoolctl==3.1.0
tifffile==2021.11.2
tomli==2.0.1
typing-extensions==3.10.0.2
tzdata==2022.1
tzlocal==4.2
urllib3==1.26.8
Werkzeug==2.0.3
wrapt==1.12.1
zipp==3.7.0


==========NVIDIA==========

Thu May 26 08:49:11 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:3B:00.0 Off |                  N/A |
| 40%   72C    P2   247W / 250W |    544MiB / 11019MiB |     90%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:5E:00.0 Off |                  N/A |
| 42%   78C    P2   244W / 250W |    544MiB / 11019MiB |     89%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:AF:00.0 Off |                  N/A |
| 43%   81C    P2   248W / 250W |    542MiB / 11019MiB |     90%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:D8:00.0 Off |                  N/A |
| 42%   78C    P2   242W / 250W |    492MiB / 11019MiB |     93%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0    589266      C   ...apps/amber_build/amber18/bin/pmemd.cuda   533MiB |
|    1    589394      C   ...apps/amber_build/amber18/bin/pmemd.cuda   533MiB |
|    2    589619      C   ...apps/amber_build/amber18/bin/pmemd.cuda   531MiB |
|    3    591300      C   ...apps/amber_build/amber18/bin/pmemd.cuda   481MiB |
+-----------------------------------------------------------------------------+

Logs

Train centroid model
INFO:sleap.nn.training:Versions:
SLEAP: 1.2.2
TensorFlow: 2.6.3
Numpy: 1.19.5
Python: 3.7.12
OS: Linux-3.10.0-693.5.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core
INFO:sleap.nn.training:Training labels file: /home/cg226285e/48HoursDegus/SLEAP/test1/project_1_labels.v000.pkg.slp
INFO:sleap.nn.training:Training profile: /home/cg226285e/48HoursDegus/SLEAP/test1/configuration_files_topdown_pipeline/centroid.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/home/cg226285e/48HoursDegus/SLEAP/test1/configuration_files_topdown_pipeline/centroid.json",
    "labels_path": "/home/cg226285e/48HoursDegus/SLEAP/test1/project_1_labels.v000.pkg.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "tensorboard": false,
    "save_viz": false,
    "zmq": false,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": 0
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": null,
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": null,
            "validation_inds": null,
            "test_inds": null,
            "search_path_hints": [],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": true,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": null,
            "resize_and_pad_to_target": true,
            "target_height": null,
            "target_width": null
        },
        "instance_cropping": {
            "center_on_part": "shoulder",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "shoulder",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        }
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -15.0,
            "rotation_max_angle": 15.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": true
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": null,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": null,
        "min_val_batches_per_epoch": 10,
        "epochs": 200,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "220524_084659",
        "run_name_prefix": "topdown_test",
        "run_name_suffix": ".centroid",
        "runs_folder": "models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": false,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": false,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.2.2",
    "filename": "/home/cg226285e/48HoursDegus/SLEAP/test1/configuration_files_topdown_pipeline/centroid.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/4 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
  Device: /physical_device:GPU:1
         Available: False
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:2
         Available: False
        Initalized: False
     Memory growth: None
  Device: /physical_device:GPU:3
         Available: False
        Initalized: False
     Memory growth: None
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /home/cg226285e/48HoursDegus/SLEAP/test1/project_1_labels.v000.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 6 / Validation = 1.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
Inference

Screenshots

How to reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

roomrys · 2022-05-26T16:35:58Z

roomrys
May 26, 2022
Maintainer

Hi @cosimogonnelli,

Do you mind filling out the edited description above? This seems to occur due to the version of cudatoolkit and tensorflow conflicting (similar problem seen here); thus, it might be beneficial to know how you are installing/running SLEAP (which should handle these dependencies for you).

Thanks,
Liezl

1 reply

cosimogonnelli May 26, 2022
Author

Hi @roomrys,

thanks for your quick help. I edited the description above and I believed I added the full diagnostic. Please let me know if you need more information.

Thanks!

talmo · 2022-05-27T12:43:04Z

talmo
May 27, 2022
Maintainer

This is usually because your GPU drivers aren't updated enough for the specific version of cudatoolkit (10.2 requires drivers >= 440.33). Yours (440.33.01) are ok for your system CUDA 10.2, but it looks like you're using the newer cudatoolkit in the SLEAP conda environment:

cudatoolkit               11.3.1               ha36c431_9    nvidia
cudnn                     8.2.1.32             h86fa8c9_0    conda-forge

You could try to conda install -c nvidia cudatoolkit=10.2, but you might actually be better off updating your system drivers to >= 450.80.02.

Let us know if either works for you!

3 replies

cosimogonnelli May 30, 2022
Author

Hi @talmo,

Unfortunately I can't update my system drivers because I am working on a remote machine. I tried to use 'conda install -c nvidia cudatoolkit=10.2' but I am failing the installation and it seems that is due to a lot of packages conflicts. How can I fix this issue? I am concerned about going down in a rabbit hole. Here you can see the out put of my terminal. Thanks for your help!

conda install -c nvidia cudatoolkit=10.2.pdf

talmo May 30, 2022
Maintainer

Hmmm, this is going to be tricky.

TensorFlow 2.6 only supports cudatoolkit 11.

If you can't request a driver update (it might help to mention that the current version on that machine was released in 2019!), then the alternative would be to try to use TensorFlow 2.3.

You'll have to follow the same steps as installing from source except that before you create the environment with conda env create..., you'll want to:

Edit environment.yml to set cudatoolkit=10 and cudnn=8 (possibly cudnn=7 if that doesn't work)
Edit requirements.txt and set tensorflow==2.3.*

That might still not work, however, since it's been a little while since we've update SLEAP to use TensorFlow 2.6 or newer.

If it's easier to do that than send the admin an email asking for a driver update (which shouldn't break existing applications btw) -- then give the steps above a go or let us know if you need any clarifications.

Sorry about the headache! Version compatibility issues are the hardest thing we deal with given the GPU dependency...

cosimogonnelli Jun 2, 2022
Author

Hi @talmo,

I followed the instruction and I still have conflicts that make the installation to fail.
How can I fix this?
Attached is my terminal output.
Thanks for your help!

conda env create -f environment.yml -n sleap.pdf

Logs

conda env create -f environment.yml -n sleap
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages
libstdcxx-ng-11.2.0 | 4.7 MB |
#####################################################
########################################## | 100%
libgfortran5-11.2.0 | 2.0 MB |
#####################################################
########################################## | 100%
libgcc-ng-11.2.0 | 5.3 MB |
#####################################################
########################################## | 100%
libgfortran-ng-11.2. | 20 KB |
#####################################################
########################################## | 100%
libgomp-11.2.0 | 474 KB |
#####################################################
########################################## | 100%
cudnn-7.6.5 | 184.3 MB |
#####################################################
########################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/cg226285e/anaconda/envs/sleap_downgraded/bin/python', '-
m', 'pip', 'install', '-U', '-r', '/home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt']
Pip subprocess output:
Obtaining file:///home/cg226285e/sleap (from -r /home/cg226285e/
sleap/condaenv.5o9fsb0x.requirements.txt (line 1))
 Preparing metadata (setup.py): started
 Preparing metadata (setup.py): finished with status 'done'
Collecting pytest
 Using cached pytest-7.1.2-py3-none-any.whl (297 kB)
Collecting pytest-qt
 Using cached pytest_qt-4.0.2-py2.py3-none-any.whl (33 kB)
Collecting pytest-cov
 Using cached pytest_cov-3.0.0-py3-none-any.whl (20 kB)
Collecting pytest-xvfb
 Using cached pytest_xvfb-2.0.0-py3-none-any.whl (5.1 kB)
Collecting ipython
 Using cached ipython-7.34.0-py3-none-any.whl (793 kB)
Collecting sphinx
 Using cached Sphinx-5.0.0-py3-none-any.whl (3.1 MB)
Collecting sphinx-book-theme
 Using cached sphinx_book_theme-0.3.2-py3-none-any.whl (335 kB)
Collecting sphinx-copybutton
 Using cached sphinx_copybutton-0.5.0-py3-none-any.whl (12 kB)
Collecting nbformat==5.1.3
 Using cached nbformat-5.1.3-py3-none-any.whl (178 kB)
Collecting myst-nb==0.13.2
 Using cached myst_nb-0.13.2-py3-none-any.whl (41 kB)
Collecting myst-parser==0.15.2
 Using cached myst_parser-0.15.2-py3-none-any.whl (46 kB)
Collecting linkify-it-py
 Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting sphinx-autobuild
 Using cached sphinx_autobuild-2021.3.14-py3-none-any.whl (9.9 kB)
Collecting black==21.6b0
 Using cached black-21.6b0-py3-none-any.whl (140 kB)
Collecting pre-commit
 Using cached pre_commit-2.19.0-py2.py3-none-any.whl (199 kB)
Collecting twine==3.3.0
 Using cached twine-3.3.0-py3-none-any.whl (34 kB)
Collecting PyGithub
 Using cached PyGithub-1.55-py3-none-any.whl (291 kB)
Collecting jupyterlab
 Using cached jupyterlab-3.4.2-py3-none-any.whl (8.8 MB)
Collecting jedi==0.17.2
 Using cached jedi-0.17.2-py2.py3-none-any.whl (1.4 MB)
Collecting ipykernel
 Using cached ipykernel-6.13.0-py3-none-any.whl (131 kB)
Collecting click==8.0.4
 Using cached click-8.0.4-py3-none-any.whl (97 kB)
Collecting ipython-genutils
 Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting jupyter-core
 Using cached jupyter_core-4.10.0-py3-none-any.whl (87 kB)
Collecting jsonschema!=2.5.0,>=2.4
 Downloading jsonschema-4.6.0-py3-none-any.whl (80 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
━━━━━━━ 80.4/80.4 KB 2.8 MB/s eta 0:00:00
Collecting traitlets>=4.1
 Using cached traitlets-5.2.2.post1-py3-none-any.whl (106 kB)
Collecting docutils<0.18,>=0.15
 Using cached docutils-0.17.1-py2.py3-none-any.whl (575 kB)
Collecting sphinx-togglebutton~=0.3.0
 Using cached sphinx_togglebutton-0.3.1-py3-none-any.whl (8.0 kB)
Collecting ipywidgets<8,>=7.0.0
 Using cached ipywidgets-7.7.0-py2.py3-none-any.whl (123 kB)
Collecting jupyter-cache~=0.4.1
 Using cached jupyter_cache-0.4.3-py3-none-any.whl (31 kB)
Collecting jupyter-sphinx~=0.3.2
 Using cached jupyter_sphinx-0.3.2-py3-none-any.whl (20 kB)
Collecting nbconvert<7,>=5.6
 Using cached nbconvert-6.5.0-py3-none-any.whl (561 kB)
Collecting sphinx
 Using cached Sphinx-4.5.0-py3-none-any.whl (3.1 MB)
Collecting importlib-metadata
 Using cached importlib_metadata-4.11.4-py3-none-any.whl (18 kB)
Collecting pyyaml
 Using cached PyYAML-6.0-cp37-cp37mmanylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.m
anylinux2010_x86_64.whl (596 kB)
Collecting jinja2
 Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting mdit-py-plugins~=0.2.8
 Using cached mdit_py_plugins-0.2.8-py3-none-any.whl (41 kB)
Collecting markdown-it-py<2.0.0,>=1.0.0
 Using cached markdown_it_py-1.1.0-py3-none-any.whl (83 kB)
Collecting regex>=2020.1.8
 Using cached regex-2022.4.24-cp37-cp37mmanylinux_2_17_x86_64.manylinux2014_x86_64.whl (749 kB)
Collecting typed-ast>=1.4.2
 Using cached typed_ast-1.5.4-cp37-cp37mmanylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.m
anylinux2010_x86_64.whl (843 kB)
Collecting appdirs
 Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pathspec<1,>=0.8.1
 Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting typing-extensions>=3.7.4
 Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting toml>=0.10.1
 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting mypy-extensions>=0.4.3
 Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting colorama>=0.4.3
 Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: setuptools>=0.7.0 in /home/cg226285e/
anaconda/envs/sleap_downgraded/lib/python3.7/site-packages (from
twine==3.3.0->-r /home/cg226285e/sleap/./dev_requirements.txt (line
17)) (58.0.4)
Collecting requests>=2.20
 Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting pkginfo>=1.4.2
 Using cached pkginfo-1.8.2-py2.py3-none-any.whl (26 kB)
Collecting tqdm>=4.14
 Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting rfc3986>=1.4.0
 Using cached rfc3986-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
 Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting keyring>=15.1
 Using cached keyring-23.5.1-py3-none-any.whl (33 kB)
Collecting readme-renderer>=21.0
 Using cached readme_renderer-35.0-py3-none-any.whl (14 kB)
Collecting parso<0.8.0,>=0.7.0
 Using cached parso-0.7.1-py2.py3-none-any.whl (109 kB)
Requirement already satisfied: numpy<=1.21.5,>=1.19.5 in /home/
cg226285e/anaconda/envs/sleap_downgraded/lib/python3.7/sitepackages (from sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (1.21.5)
Collecting attrs==21.2.0
 Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting cattrs==1.1.1
 Using cached cattrs-1.1.1-py3-none-any.whl (16 kB)
Collecting jsonpickle==1.2
 Using cached jsonpickle-1.2-py2.py3-none-any.whl (32 kB)
Collecting networkx
 Using cached networkx-2.6.3-py3-none-any.whl (1.9 MB)
Collecting tensorflow==2.3.*
 Using cached tensorflow-2.3.4-cp37-cp37mmanylinux2010_x86_64.whl (320.6 MB)
Requirement already satisfied: h5py<=3.6.0,>=3.1.0 in /home/
cg226285e/anaconda/envs/sleap_downgraded/lib/python3.7/sitepackages (from sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (3.6.0)
Collecting python-rapidjson
 Using cached python_rapidjson-1.6-cp37-cp37mmanylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting opencv-python-headless<=4.5.5.62,>=4.2.0.34
 Using cached opencv_python_headless-4.5.5.62-cp36-abi3-
manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.7 MB)
Requirement already satisfied: pandas in /home/cg226285e/anaconda/
envs/sleap_downgraded/lib/python3.7/site-packages (from
sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (1.3.5)
Collecting psutil
 Using cached psutil-5.9.1-cp37-cp37mmanylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_6
4.manylinux2014_x86_64.whl (281 kB)
Requirement already satisfied: PySide2<=5.14.1,>=5.13.2 in /home/
cg226285e/anaconda/envs/sleap_downgraded/lib/python3.7/sitepackages (from sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (5.13.2)
Collecting pyzmq
 Using cached pyzmq-23.0.0-cp37-cp37mmanylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)
Collecting imageio<=2.15.0
 Using cached imageio-2.15.0-py3-none-any.whl (3.3 MB)
Collecting imgaug==0.4.0
 Using cached imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
Requirement already satisfied: scipy<=1.7.3,>=1.4.1 in /home/
cg226285e/anaconda/envs/sleap_downgraded/lib/python3.7/sitepackages (from sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (1.7.3)
Collecting scikit-image
 Using cached scikit_image-0.19.2-cp37-cp37mmanylinux_2_12_x86_64.manylinux2010_x86_64.whl (13.5 MB)
Collecting scikit-learn==1.0.*
 Using cached scikit_learn-1.0.2-cp37-cp37mmanylinux_2_17_x86_64.manylinux2014_x86_64.whl (24.8 MB)
Collecting scikit-video
 Using cached scikit_video-1.1.11-py2.py3-none-any.whl (2.3 MB)
Collecting imgstore==0.2.9
 Using cached imgstore-0.2.9-py2.py3-none-any.whl (904 kB)
Collecting qimage2ndarray<=1.8.3,>=1.8.2
 Using cached qimage2ndarray-1.8.3-py3-none-any.whl (11 kB)
Collecting jsmin
 Using cached jsmin-3.0.1.tar.gz (13 kB)
 Preparing metadata (setup.py): started
 Preparing metadata (setup.py): finished with status 'done'
Collecting seaborn
 Using cached seaborn-0.11.2-py3-none-any.whl (292 kB)
Collecting pykalman==0.9.5
 Using cached pykalman-0.9.5.tar.gz (228 kB)
 Preparing metadata (setup.py): started
 Preparing metadata (setup.py): finished with status 'done'
Collecting segmentation-models==1.0.1
 Using cached segmentation_models-1.0.1-py3-none-any.whl (33 kB)
Collecting rich==10.16.1
 Using cached rich-10.16.1-py3-none-any.whl (214 kB)
Collecting certifi<=2021.10.8,>=2017.4.17
 Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting opencv-python
 Using cached opencv_python-4.5.5.64-cp36-abi3-
manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.5 MB)
Requirement already satisfied: Pillow in /home/cg226285e/anaconda/
envs/sleap_downgraded/lib/python3.7/site-packages (from
imgaug==0.4.0->sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (8.4.0)
Collecting matplotlib
 Using cached matplotlib-3.5.2-cp37-cp37mmanylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)
Requirement already satisfied: six in /home/cg226285e/anaconda/envs/
sleap_downgraded/lib/python3.7/site-packages (from imgaug==0.4.0-
>sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (1.16.0)
Requirement already satisfied: Shapely in /home/cg226285e/anaconda/
envs/sleap_downgraded/lib/python3.7/site-packages (from
imgaug==0.4.0->sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (1.7.1)
Requirement already satisfied: python-dateutil in /home/cg226285e/
anaconda/envs/sleap_downgraded/lib/python3.7/site-packages (from
imgstore==0.2.9->sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (2.8.2)
Requirement already satisfied: pytz in /home/cg226285e/anaconda/
envs/sleap_downgraded/lib/python3.7/site-packages (from
imgstore==0.2.9->sleap==1.2.3->-r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1)) (2021.3)
Collecting tzlocal
 Using cached tzlocal-4.2-py3-none-any.whl (19 kB)
Collecting pygments<3.0.0,>=2.6.0
 Using cached Pygments-2.12.0-py3-none-any.whl (1.1 MB)
Collecting commonmark<0.10.0,>=0.9.0
 Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting joblib>=0.11
 Using cached joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting threadpoolctl>=2.0.0
 Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting efficientnet==1.0.0
 Using cached efficientnet-1.0.0-py3-none-any.whl (17 kB)
Collecting image-classifiers==1.0.0
 Using cached image_classifiers-1.0.0-py3-none-any.whl (19 kB)
Collecting keras-applications<=1.0.8,>=1.0.7
 Using cached Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
Collecting tensorflow==2.3.*
 Using cached tensorflow-2.3.3-cp37-cp37mmanylinux2010_x86_64.whl (320.5 MB)
 Using cached tensorflow-2.3.2-cp37-cp37mmanylinux2010_x86_64.whl (320.4 MB)
 Using cached tensorflow-2.3.1-cp37-cp37mmanylinux2010_x86_64.whl (320.4 MB)
 Using cached tensorflow-2.3.0-cp37-cp37mmanylinux2010_x86_64.whl (320.4 MB)
INFO: pip is looking at multiple versions of segmentation-models to
determine which version is compatible with other requirements. This
could take a while.
INFO: pip is looking at multiple versions of scikit-learn to determine
which version is compatible with other requirements. This could take a
while.
Collecting scikit-learn==1.0.*
 Using cached scikit_learn-1.0.1-cp37-cp37mmanylinux_2_12_x86_64.manylinux2010_x86_64.whl (23.2 MB)
 Using cached scikit_learn-1.0-cp37-cp37mmanylinux_2_12_x86_64.manylinux2010_x86_64.whl (23.1 MB)
INFO: pip is looking at multiple versions of rich to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pykalman to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jsonpickle to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of imgstore to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of imgaug to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cattrs to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of attrs to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of sleap to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of click to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of jedi to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twine to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of black to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of myst-parser to determine
which version is compatible with other requirements. This could take a
while.
INFO: pip is looking at multiple versions of myst-nb to determine which
version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from RequiresPython> to determine which version is compatible with other
requirements. This could take a while.
INFO: pip is looking at multiple versions of nbformat to determine which
version is compatible with other requirements. This could take a while.
The conflict is caused by:
 sleap 1.2.3 depends on numpy<=1.21.5 and >=1.19.5
 imgaug 0.4.0 depends on numpy>=1.15
 imgstore 0.2.9 depends on numpy
 scikit-learn 1.0 depends on numpy>=1.14.6
 tensorflow 2.3.4 depends on numpy<1.19.0 and >=1.16.0
 sleap 1.2.3 depends on numpy<=1.21.5 and >=1.19.5
 imgaug 0.4.0 depends on numpy>=1.15
 imgstore 0.2.9 depends on numpy
 scikit-learn 1.0 depends on numpy>=1.14.6
 tensorflow 2.3.3 depends on numpy<1.19.0 and >=1.16.0
 sleap 1.2.3 depends on numpy<=1.21.5 and >=1.19.5
 imgaug 0.4.0 depends on numpy>=1.15
 imgstore 0.2.9 depends on numpy
 scikit-learn 1.0 depends on numpy>=1.14.6
 tensorflow 2.3.2 depends on numpy<1.19.0 and >=1.16.0
 sleap 1.2.3 depends on numpy<=1.21.5 and >=1.19.5
 imgaug 0.4.0 depends on numpy>=1.15
 imgstore 0.2.9 depends on numpy
 scikit-learn 1.0 depends on numpy>=1.14.6
 tensorflow 2.3.1 depends on numpy<1.19.0 and >=1.16.0
 sleap 1.2.3 depends on numpy<=1.21.5 and >=1.19.5
 imgaug 0.4.0 depends on numpy>=1.15
 imgstore 0.2.9 depends on numpy
 scikit-learn 1.0 depends on numpy>=1.14.6
 tensorflow 2.3.0 depends on numpy<1.19.0 and >=1.16.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the
dependency conflict
Pip subprocess error:
ERROR: Cannot install -r /home/cg226285e/sleap/
condaenv.5o9fsb0x.requirements.txt (line 1) and sleap because these
package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/
topics/dependency-resolution/#dealing-with-dependency-conflicts
failed
CondaEnvException: Pip failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

InternalError: cudaGetDevice() failed #762

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

InternalError: cudaGetDevice() failed #762

cosimogonnelli May 26, 2022

Problem description

Expected behaviour

Actual behaviour

Your personal set up

Screenshots

How to reproduce

Replies: 2 comments · 4 replies

roomrys May 26, 2022 Maintainer

cosimogonnelli May 26, 2022 Author

talmo May 27, 2022 Maintainer

cosimogonnelli May 30, 2022 Author

talmo May 30, 2022 Maintainer

cosimogonnelli Jun 2, 2022 Author

cosimogonnelli
May 26, 2022

Replies: 2 comments 4 replies

roomrys
May 26, 2022
Maintainer

cosimogonnelli May 26, 2022
Author

talmo
May 27, 2022
Maintainer

cosimogonnelli May 30, 2022
Author

talmo May 30, 2022
Maintainer

cosimogonnelli Jun 2, 2022
Author