[Model] Remove intermediate states copying in Mllama

[jkaniecki]

Mllama model - This PR changes the way intermediate hidden states are kept inside encoder part of the model. Appending a tuple with tensors and stacking them at the end can cause memcopying on different devices. This sollution avoids using tuple and uses pre-located tensor instead.
That can help to improve encoder speed by avoiding tensors stacking (which triggers memcopy)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[heheda12345]

Do you have benchmark result on this optimization? I think we still need a copy operation here:

encoder_states.index_copy_(0, hidden_states_idx, hidden_states.unsqueeze(0))

[jkaniecki]

Do you have benchmark result on this optimization? I think we still need a copy operation here:

encoder_states.index_copy_(0, hidden_states_idx, hidden_states.unsqueeze(0))

@heheda12345 I observe stable improvement in prompt time when using Gaudi devices. Depends on hidden states size, but always getting a few ms boost with this change.

[heheda12345]

Can you show how did you do the benchmarking and your exact number here?

[jkaniecki]

@heheda12345 Sorry for long response time, I was offline for last 2,5 weeks. I checked the change impact using benchmark_throughput.py from vllm/benchmarks. Using such a command:

python benchmark_throughput.py --model Meta-Llama-3.2-11B-Vision-Instruct --max-model-len 2048 --dataset datasets/sharegpt4v_instruct_gpt4-vision_cap100k.json --num-prompts 1000 --max-num-seqs 128 --output-len 4

I was able to see ~1,5% gain in all metrics reported by the test.
I made following assumptions:

• I reduced output len to 4 to expose change impact on prompt time in the final outcome (encoder is only used during prompt phase)
• I used sharegpt4v_instruct_gpt4-vision_cap100k dataset available here
• To make benchmark_throughput.py work with mllama I needed to adjust test script by adding proper prompt pattern:
  

[heheda12345]

I think that is not a very serious problem on Gaudi, and not sure whether it can have similar performance gain on other platforms.