Infrence Batching in VLLM

[HimanshuJanbandhu]

So I used vLLM with

1. Individual prompt inferencing
2. Batching prompts for inferencing

I saw that the Batch inferences took on average way more time than individual prompts took, per prompt.

This medium post claims, batch inferencing should take less time, but doesn't provide any proofs,
https://medium.com/@wearegap/a-brief-introduction-to-optimized-batched-inference-with-vllm-deddf5423d0c

Is there any reason why vLLM should be faster while batch infrencing, if yes, why so?

[Chocobi-1129]

Current LLM inference speed bottle neck is memory I/O, which is bring the model weight and data from vram to computation core (eg. CUDA) through HBM. Running in batch means if you have two requests in the same time, you only have to load the weight from VRAM once and deal with the two requests.
The average time per batch will be lower but the latency of the first request will be higher.