RuntimeError: Errors during CUDA kernel calls forward

[zhangpangpang-mi]

When I integrated xLSTM into my project, I encountered the following issue, but I am not sure how to resolve it
File "/remote-home/chengmi/PycharmProjects/Music-Source-Separation-Training_loss/Music-Source-Separation-Training-main/train.py", line 462, in train_model
y_ = model(x)
^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/PycharmProjects/Music-Source-Separation-Training_loss/Music-Source-Separation-Training-main/models/dp_tdf/xlstm_mamba_self.py", line 344, in forward
x = self.xlstm_stack(x)
^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/xlstm_block_stack.py", line 120, in forward
x = block(x, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/blocks/xlstm_block.py", line 77, in forward
x = x + self.xlstm(self.xlstm_norm(x), **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/blocks/slstm/layer.py", line 151, in forward
y, slstm_state = self.slstm_cell(
^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/blocks/slstm/cell.py", line 500, in forward
all_states = self._impl(self.training, input, states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/blocks/slstm/cell.py", line 766, in _impl
return self.func.apply(
^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 465, in decorate_fwd
return fwd(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/remote-home/chengmi/anaconda3/envs/xlstm/lib/python3.11/site-packages/xlstm/blocks/slstm/cell.py", line 560, in forward
states, cache_g_r, cache_g_i = slstm_mod.forward(training, *inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Errors during CUDA kernel calls forward.

[WangYLon]

Looks like it might be because of CUDA? Please check if your cuda is available

[zhangpangpang-mi]

Thank you. My original code was running fine, but after adding the xlstm module, this issue started appearing. However, when I changed backend="cuda" to backend="vanilla", the error disappeared.