LLM Serving

[MrMondragon]

Sorry if this is a noob question but is there a way to create a server using flask that is able to load and keep an LLM in memory so that I don't need to reload it between calls?

[davidism]

Your LLM is a separate service from your Flask frontend. You'll want to run that, then have a way for Flask views to use it. For example, you could use multiprocessing.Manager to manage the LLM service and expose an interface for it: https://stackoverflow.com/questions/28423069/store-large-data-or-a-service-connection-per-flask-session

[RonnyPfannschmidt]

i suspect the problem can actually collapse to something like a globalish object that loads the llm once per model
for confirmation details on the used llm api would be needed

(i presume llm in memory means a actual loaded llm one runs in process as opposed to a rest api)

while the initial post is not clear i think its fair to assume op is tryingto create the/a LLM service, not using one