This is my first fork project. There are many irregularities. Please forgive me. I made some small modifications to the input. The current code supports multi-user input, and converted the previous gemv operator into gemm, which has improved performance to a certain extent. At the same time, I am still exploring the introduction of more technologies to improve the efficiency of the project, such as KV-cache optimization, etc.
MASALab