LLM rails not working on Kubernetes - Pod restarts with out any errors

[moka1602]

Hi ,

I am using input guardrails for my chatbot which is working perfectly fine in local. The problem is starts deploying the fast API on kubernetes. As soon as we hit the api , the pod restarts at LLMRails(config=config, verbose=True) . No errors !!!
Using the vertexAI model for both LLM and embeddings. PLS HELP!!!

config.yaml

models:
- type: main
engine: vertexai
model: gemini-1.0-pro
- type: embeddings
engine: vertexai
model: text-embedding-004

core:
embedding_search_provider:
name: default
parameters:
embedding_engine: vertexai
embedding_model: text-embedding-004
cache:
enabled: False
key_generator: md5
store: in_memory
store_config: {}

knowledge_base:
embedding_search_provider:
name: default
parameters:
embedding_engine: vertexai
embedding_model: text-embedding-004
cache:
enabled: False
key_generator: md5
store: in_memory
store_config: {}

[moka1602]

Anyone tried deploying a chatbot with guardrails on Kubernetes faced this issue ? Pls suggest.

@drazvan

[Pouyanpi]

Hi @moka1602 , when you say that you are deploying FastAPI app, how are you doing it exactly? Are you using the Dockerfile in the repo to build the image? When you run the image would it work?

[fedewj]

are there any updates? I'm facing a similar problem

[Pouyanpi]

@jingwangfe would you please open a new issue and refer to this issue there? We need more info to be able to reproduce this issue but currently we don't have much. Opening a new issue would make this easier for you and us.