Different Models with different GPUs

[CallterC]

Hey guys,
I am trying to deploy a pipeline where I need to initialize two vllm's LLM instance with different models (in this case Llama-3.1-70B and Gemma-2-9B).
I am wondering is there a way to set different devices directly while initializing the vllm.LLM instance? Upon trying to use the device parameter, it seems that I can't initialize the second instance on the second GPU (it will throw a input and model are on different devices error). The error only occurs when change the model's device to the second GPU.
Looking for some help,
Thanks in advance