Popular repositories Loading

1. llama.cpp llama.cpp Public

   Forked from ggerganov/llama.cpp

   LLM inference in C/C++

   C++

2. tensorrtx tensorrtx Public

   Forked from wang-xinyu/tensorrtx

   Implementation of popular deep learning networks with TensorRT network definition API

   C++

3. InfiniGen InfiniGen Public

   Forked from snu-comparch/InfiniGen

   InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (OSDI'24)

   Python

4. H2O H2O Public

   Forked from FMInference/H2O

   [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

   Python

5. prompt-cache prompt-cache Public

   Forked from yale-sys/prompt-cache

   Modular and structured prompt caching for low-latency LLM inference

   Python

6. serverlesskv serverlesskv Public