Tensor Parallel All Reduce

[mjkpolo]

Hi, I am struggling to find where partial results from matrix multiply are reduced together. My understanding is when I use -sm row, that a row-wise tensor parallel approach is employed, where the result of every single matrix multiply needs to be all reduced. However, I only really see a warp_reduce_sum which I assume is used for the tiling that happens across threads in a warp on a single gpu, but not an operation between GPUs reducing the whole matrix.

[mjkpolo]

It seems that llama actually does a column parallel approach which is misleading since the option passed is sm -row but after ggml_mat_mul is done the result is just copied to a destination as shown for column parallel in the link above.