Not able to make Presidio sensitive data masking flows work with NeMo guardrails and Langchain

[Krish4101983]

Hello everyone, greetings! reaching out for help as having issues making the Presidio sensitive data masking flows work with NeMo guardrails and Langchain, any help will be greatly appreciated

I am implementing NeMo guard rails with Langchain and trying out the Presidio lib supported by NeMo for sensitive data masking

Issues - 1. Although some events related to sensitive data masking flow is getting triggered and, on the input flow, some masking happening as per the logs, BUT, it looks like that the data supposed to be masked is still sent to the LLM,
2. On the output flow, its always giving 'I'm not sure what to say.'
3. This output behavior is happening even when there is no maskable data is there in output

I am attaching logs for both the scenarios where maskable entity is present in the input and output as well as when its not.
In both cases Presidio rails are present

YAML code:

models: []
rails:
    dialog:
        single_call:
          enabled: False
          fallback_to_multiple_calls: True
    config:
        sensitive_data_detection:
          input:
            entities:
              - PERSON
              - EMAIL_ADDRESS
          output:
            entities:
              - PERSON
              - EMAIL_ADDRESS
    input:
        flows:
          - mask sensitive data on input

    output:
        flows:
          - mask sensitive data on output

The CoLang looks like (This is not related to the YAML one, the YAML one is for Presidio sensitive data which we are testing ) -

define user ask india question
    "any question related India"

define bot does not answer
    "Can't answer anything on India, how can I help you today?"

define flow ask india related question
    user ask india question
    bot does not answer

The Langchain and NeMo guardrails versions -

langchain==0.1.20
langchain-community==0.0.38
langchain-core==0.1.52
nemoguardrails==0.9.0

Guardrail creation code :

    def getGuardrailChain(payload, llm, chain):
      logger.debug("setting up nemo guardrail")
      config = RailsConfig.from_content(colang_content=payload.colang_content, yaml_content=payload.yaml_content)
  
      chain_with_guardrails = RunnableRails(config=config, llm=llm, runnable=chain)
      #guardrails = RunnableRails(config=config, llm=llm)
      #chain_with_guardrails = guardrails | chain
      return chain_with_guardrails

chain invocation code:

llm_chain = getGuardrailChain(payload, llm, llm_chain )
rsp = llm_chain.invoke({"input": payload.prompt, "image": payload.image},
                         config={"callbacks": [langfuse_handler]})

I am using Azure OpenAI gpt-35-turbo model

Logs:

Logs-when-maskable-entity-in-input-output-with-presidio-rails-present.txt
Logs-when-no-maskable-entity-ininput-output-with-presidio-rails-present.txt

[NiklasF98]

Having the same Issue. Need help ASAP please.