Looking for sample code to integrate streaming output for VLLM through Langchain

[GaneshDoosa]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.llms.vllm import VLLM
llm = VLLM(
                model="llama3.1-8B-instruct",
                trust_remote_code=True,  # mandatory for hf models
                max_new_tokens=2048,
                top_k=10,
                top_p=0.95,
                temperature=0,
                vllm_kwargs={
                    # "max_model_len": 1024,
                    "gpu_memory_utilization": 0.95,
                },
            )

Description

Even after adding stream=True the output tokens not being parsed as stream

System Info

langchain : 0.2.15
vllm : 0.6.1

[feijoes]

@GaneshDoosa VLLM is a BaseLLM try:

llm = VLLM(...)
async for chunk in llm.astream("Question"):
    print(chunk, end="", flush=True)