Tensor with weird strides encountered when writing a backend

[marty1885]

Hi,

I'm writing a new backend for NPUs and I encountered this VIEW operation when trying to run LLaMA and Qwen2. I assume this is a part of KV-cache looking at the source tensor shape. However I don't understand what the target tensor. The stride doesn't make sense to me as it is not a permutation of the normal strides.

I expect the "normal" stride be 2, 256, 8192, 8192 Where did the 512 come from and where did the 8192 go?

The following is the debug print I got when running Qwen2 1.5B.

VIEW offset: 0
VIEW source shape: 8388608 1 1 1
VIEW source stride: 2 16777216 16777216 16777216
VIEW target shape: 128 32 2 1
VIEW target stride: 2 512 256 512
SLICE type: f16

Can someone help me understand where this view operation comes from and what is it supposed to do?

[slaren]

It's probably the result of a ggml_view_3d operation, which leaves the stride of the 4th dimension unchanged. You can print the tensor->name to easily find what operation it comes from. The strides of dimensions with size 1 are not relevant and can be ignored.

[marty1885]

Hi, that part I understand. What is the 256 doing in nb[2]? It suggests moving one step in that dimension moves half step in dimension 1. Is this even valid?

[ggerganov]

What is the 256 doing in nb[2]?

This is a permuted tensor.

For example, a normal F16 tensor that you are probably used to would look like this:

ne: 128   2   32    1
nb:   2 256  512 8192

elements in consecutive memory (4-index ne):

  0 0 0 0   (offs = 0 bytes)
  1 0 0 0
  2 0 0 0
...
127 0 0 0
  0 1 0 0   (offs = 256 bytes)
  1 1 0 0
  2 1 0 0
...
127 1 0 0
  0 0 1 0   (offs = 512 bytes)
  1 0 1 0
  2 0 1 0
...
127 0 1 0
  0 1 1 0   (offs = 768 bytes)
  1 1 1 0
  2 1 1 0
...
127 1 1 0
  0 0 2 0   (offs = 1024 bytes)
...

This tensor, since it is permuted, looks like this:

ne: 128  32    2    1
nb:   2 512  256  512

elements in consecutive memory (4-index ne):

  0 0 0 0   (offs = 0 bytes)
  1 0 0 0
  2 0 0 0
...
127 0 0 0
  0 0 1 0   (offs = 256 bytes)
  1 0 1 0
  2 0 1 0
...
127 0 1 0
  0 1 0 0   (offs = 512 bytes)
  1 1 0 0
  2 1 0 0
...
127 1 0 0
  0 1 1 0   (offs = 768 bytes)
  1 1 1 0
  2 1 1 0
...
127 1 1 0
  0 2 0 0   (offs = 1024 bytes)
...

[marty1885]

Ahhh D'oh do did not not see it. Thanks!