How to maximize vLLM throughput

[smennes]

The following post vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction | vLLM Blog mentions a 10 req/s on vLLM on prefill heavy prompts with limited output.
I'm running 8 bit quantified llama3.1-70b on 4xA100 40GB with python -m vllm.entrypoints.openai.api_server --model /lama3.1-70b/ --host 0.0.0.0 --port 8000 --tensor-parallel-size 4 --gpu-memory-utilization 0.9 --quantization bitsandbytes --load-format bitsandbytes --served-model-name llama --num-scheduler-steps 10 --enable-chunked-prefill --max-num-batched-tokens 2048

and get the following results
+-----------------+---------------+---------+-------------------+
| Output Length | Input Length | Req/s | Output Tokens/s |
+=================+===============+=========+===================+
| 2 words | 263 | 4.77 | 14.59 |
+-----------------+---------------+---------+-------------------+
| 4 words | 263 | 4.31 | 26.36 |
+-----------------+---------------+---------+-------------------+
| 6 words | 263 | 4.16 | 36.34 |
+-----------------+---------------+---------+-------------------+
Why am I getting half the performance with less heavy prompts? Is it because i'm running the 8 bit precision model on 40GB GPUs instead of 16 bit precision on 80GB GPUs?