[core] overhaul memory profiling and fix backward compatibility

[youkaichao]

fixes #10451 , and clearly explain the memory classification and the procedure.

I also added the initial pytorch memory, to be aligned with the pytorch memory profiler.

the profiling procedure is extracted into vllm/utils , so that we can use it later in v1 too.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

cc @joerunde

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @youkaichao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[JaheimLee]

any progress? I do need to run multi-instance with one GPU.

[Pydataman]

any progress? I do need to run multi-instance with one GPU.

is this function available?

[youkaichao]

let me finish it this week.

[joerunde]

Thanks for taking this on @youkaichao!

I think the docs for the --gpu-memory-utilization flag should also be updated in this PR to reflect the changes once this is working properly

[joerunde]

Some suggestions on making this a bit nicer:

1. The log would be much easier to parse if the numbers were aligned on the same column, right now they're all over the place
2. The descriptions are a mix of plain words and variable names, for logs maybe we should just use words. available_kv_cache_memory: -> KV Cache Size: etc.
3. "Non torch memory" is the one item on this list that I think might not be easily understood by somebody reading the logs. Maybe calling it something a little more generic like "Memory overhead" would be less distracting

[mgoin]

I think all of the newlines take too much space in the logs. IMO it would be more simple to keep the same single-line comma-separated result as before

[youkaichao]

addressed in #10511 (comment) , PTAL

[joerunde]

@youkaichao Any chance you can add in a quick test for the profiling context manager itself? As an example the one I wrote up here was very simple to do: https://github.com/vllm-project/vllm/pull/11120/files#diff-33c13e0b177bacd2f02e29bcb8aea5b49e7ce34901fd8f41fefb65defba1bd33R277-R312

[joerunde]

@youkaichao 🤔🤔🤔 Loading facebook/opt-125 twice in the same process on an A100 measures a negative non_torch_memory value:

from vllm import LLM
m1 = LLM("facebook/opt-125m", gpu_memory_utilization=0.25)
m2 = LLM("facebook/opt-125m", gpu_memory_utilization=0.25)

...
INFO 12-16 15:20:32 worker.py:243] non_torch_memory	-0.02GiB

Might not be super important to fix- I think the main use case to unblock here is multi-process vllm serving. But it is interesting, I can't immediately see why that would happen

[mgoin]

Excellent work Kaichao, I appreciate the walkthrough example in memory_profiling. This passed my local usage and I didn't see the issue Joe saw, or think it is a serious issue. My only nit is on adding all the newlines to the log, I think it was fine as comma-separated list

[mgoin]

I think all of the newlines take too much space in the logs. IMO it would be more simple to keep the same single-line comma-separated result as before

[youkaichao]

@mgoin changed the logging to be:

INFO 12-16 10:14:30 worker.py:241] Memory profiling takes 1.01 seconds
INFO 12-16 10:14:30 worker.py:241] the current vLLM instance can use total_gpu_memory (79.22GiB) x gpu_memory_utilization (0.90) = 71.29GiB
INFO 12-16 10:14:30 worker.py:241] model weights take 14.96GiB; non_torch_memory takes 0.18GiB; PyTorch activation peak memory takes 1.26GiB; the rest of the memory reserved for KV Cache is 54.90GiB.

Let me know if you have further ideas on how to improve the readability.

[youkaichao]

@youkaichao 🤔🤔🤔 Loading facebook/opt-125 twice in the same process on an A100 measures a negative non_torch_memory value:

from vllm import LLM
m1 = LLM("facebook/opt-125m", gpu_memory_utilization=0.25)
m2 = LLM("facebook/opt-125m", gpu_memory_utilization=0.25)

...
INFO 12-16 15:20:32 worker.py:243] non_torch_memory	-0.02GiB

Might not be super important to fix- I think the main use case to unblock here is multi-process vllm serving. But it is interesting, I can't immediately see why that would happen

@joerunde this is because PyTorch's internal memory fragmentation. If PyTorch allocates 2MiB from cuda, and allocate 1MiB only, then this 1 MiB will be accounted as non-torch memory. And when you run it the next time, maybe you allocate another 1 MiB, and the internal memory fragmentation reduces.

[youkaichao]

@youkaichao Any chance you can add in a quick test for the profiling context manager itself? As an example the one I wrote up here was very simple to do: #11120 (files)

@joerunde that's a great idea! I added it now, PTAL.

[joerunde]

Looks pretty good to me! Thanks for looking at this so thoroughly

[youkaichao]

errors are unrelated, merging