ValueError: Anchors should be Tuple[Tuple[int]] ... with GPU RTX 3000 series

[MounirB]

Hello,
If I try to run test.py the pretrained_model you provided on CroHD, I am facing a problem with Anchors:

python test.py --test_dataset CroHD/test/HT21-11/img1 --plot_folder outputs --outfile outputs --pretrained_model FT_R50_epoch_24.pth --context cpm

Output, with the Traceback:

256
FT_R50_epoch_24.pth
0it [00:00, ?it/s]/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torch/nn/functional.py:3502: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
  warnings.warn(
0it [00:02, ?it/s]
Traceback (most recent call last):
  File "test.py", line 176, in <module>
    test()
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "test.py", line 165, in test
    outputs = model(images)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py", line 97, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 345, in forward
    anchors = self.anchor_generator(images, features)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torchvision/models/detection/anchor_utils.py", line 150, in forward
    anchors_over_all_feature_maps = self.cached_grid_anchors(grid_sizes, strides)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torchvision/models/detection/anchor_utils.py", line 139, in cached_grid_anchors
    anchors = self.grid_anchors(grid_sizes, strides)
  File "/mnt/sdb/anaconda3/envs/headhunter-TT/lib/python3.8/site-packages/torchvision/models/detection/anchor_utils.py", line 103, in grid_anchors
    raise ValueError("Anchors should be Tuple[Tuple[int]] because each feature "
ValueError: Anchors should be Tuple[Tuple[int]] because each feature map could potentially have different sizes and aspect ratios. There needs to be a match between the number of feature maps passed and the number of sizes / aspect ratios specified.

These are the contents of my virtual environment:

name: headhunter-TT
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - ca-certificates=2021.4.13=h06a4308_1
  - certifi=2020.12.5=py38h06a4308_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - ncurses=6.2=he6710b0_1
  - openssl=1.1.1k=h27cfd23_0
  - pip=21.1.1=py38h06a4308_0
  - python=3.8.10=hdb3f193_7
  - readline=8.1=h27cfd23_0
  - setuptools=52.0.0=py38h06a4308_0
  - sqlite=3.35.4=hdfb4753_0
  - tk=8.6.10=hbc83047_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - chardet==4.0.0
    - cycler==0.10.0
    - decorator==4.4.2
    - h5py==3.2.1
    - idna==2.10
    - imageio==2.9.0
    - kiwisolver==1.3.1
    - matplotlib==3.4.2
    - networkx==2.5.1
    - numpy==1.20.3
    - ordered-set==4.0.2
    - pillow==8.2.0
    - plyfile==0.7.4
    - pyparsing==2.4.7
    - python-dateutil==2.8.1
    - pywavelets==1.1.1
    - requests==2.25.1
    - scikit-image==0.18.1
    - scipy==1.6.3
    - six==1.16.0
    - tifffile==2021.4.8
    - torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
    - torchmeta==1.7.0
    - torchvision==0.9.1
    - tqdm==4.61.0
    - trimesh==3.9.19
    - typing-extensions==3.10.0.0
    - urllib3==1.26.5
    - munkres
    - albumentations==0.5.2
    - pyyaml 

If I try to downgrade torch to 1.6.0 and torchvision to 0.7.0, I run through the following error:

RuntimeError: CUDA error: no kernel image is available for execution on the device

Moreover, I get this warning message if I use torch 1.6.0, when I try to get a device info via: torch.cuda.get_device_name(0)

NVIDIA GeForce RTX 3xxx with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.