Cupy backpropagation error

[tangzhenjie]

I encountered this problem when using cupy for backpropagation. I don’t know what’s going on? There is no problem in the forward direction, but in the reverse direction, there is a dimension mismatch.

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

code location:
mgbx = mgradbselect * xselect
gradA = torch.sum(mgbx, dim=1)

[tvercaut]

Can you provide a minimal reproducible example?

[tangzhenjie]

I wrote a sample code that will cause this problem.

import torch
import torch.optim as optim
import torchsparsegradutils.cupy.cupy_sparse_solve as cupy_solve


A = torch.randn(12, 12, requires_grad=True)
b = torch.randn(12, requires_grad=True)

target = torch.randn(12, 1, requires_grad=True)

learning_rate = 0.05
optimizer = optim.SGD([A, b], lr=learning_rate)

for i in range(100):

    x = torch.unsqueeze(cupy_solve.sparse_solve_c4t(A.to_sparse(), b), 1)

    loss = torch.mean(target - x * 10)

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

[tvercaut]

The issue is that we currently expect b to be a matrix in the backward pass. This is not cupy related.

You can workaround the issue by making b a 4x1 matrix (i.e. unsqueeze it). I'll open up a specific issue for handling vectors rhs in teh sparse solver routines.

[tangzhenjie]

Thank you very much for your prompt answer, but I changed the code to the one below and the same problem still occurs.

import torch
import torchsparsegradutils.cupy.cupy_sparse_solve as cupy_solve

A = torch.randn(12, 12, requires_grad=True).to_sparse()
b = torch.randn(12, 1, requires_grad=True)

x = cupy_solve.sparse_solve_c4t(A, b)
loss = x.sum()
loss.backward()

[tvercaut]

Indeed, there was another similar issue in the cupy solver. This should now be fixed albeit with missing unit tests:
https://colab.research.google.com/drive/1vZ23g_tamJPFGCoeV9RbKB5ObEMfkkv7?usp=sharing

Completion of unit tests will be tracked in #51