You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
在ChatChat项目中没有找到关于使用vLLM加速的介绍，想必是没有引入这个效率工具，希望可以提供官方支持啊。
我自己利用 @canghongjian 提供的vLLM加速版本（支持ChatGLM2-6B模型加速）本地demo可以跑起来，但是没法把它集成到ChatChat项目中。希望ChatChat项目组可以考虑引入使用LangChain已经集成的VLLM API实现推理加速。
Beta Was this translation helpful? Give feedback.
All reactions