How to make vLLM support a new hardware (Tenstorrent Grayskull)

[yocox]

Hi, vLLM community

I want to make vLLM support a new hardware, the Tenstorrent's Grayskull (which is a general purpose DLA, just like CUDA, but not CUDA). After reading the document and the code, I have some understanding and some questions, need the community's help to clarify my thoughts and check my understanding. Please correct me if I have any misunderstandings.

My understandings

1. The essential part of the vLLM is the PagedAttention, which is a highly optimized "memory paging mechanism" implemented on CUDA.
   • The CUDA source is at attention_kernel.cu
   • The Python binding to expose the kernel to Python is at torch_bindings.cpp
2. To utilized the Tenstorrent Grayskull, I have to do:
   • Implement the PagedAttention with Tenstorrent Grayskull kernel. (that will a huge work)
   • Expose the kernel to Python with bindings.
3. What I DON'T have to do:
   • Modify LLM's implementation which already support vLLM, because they are already using the vLLM's interface.

My questions

1. I saw there are 2 versions of kernels, v1 and v2. Do I need to implement v1, or I can just go with v2?
2. Where can I find a list of API's that I have to implement? I am afraid I missed anything. In the torch_binding.py I saw there binds a lot of operations, but do I need to implement them all or just the paged_attention_v2()?
3. Can I first only modify the forward() function to adapt vLLM's interface, without the PagedAttention? will it work but just with worse performance?
4. Does quantization cause anything special considerations?
5. Is there are anything I missed but I should know?

Thank you for reading my long questions and thanks in advance for the helping :D

[cglagovichTT]

Please give our fork a look! Tenstorrent implemented the paged kernels needed for vLLM https://github.com/tenstorrent/vllm/blob/dev/tt_metal/README.md

[jeejeelee]

There are some RFCs related to hardware support in the issues. You can look into them