[Obs AI Assistant] Fix chat on the Alerts page

[viduni94]

Closes #184214

Summary

Problem

The Observability AI Assistant doesn't work on the Alerts page - errors out with a 400 status code from OpenAI.
The reason for this is that the description for the function get_data_on_screen is too long, and there is a token limit for function descriptions by OpenAI.

Note: This error does not occur with Gemini or Bedrock because simulated function calling is enabled by default for them. With simulated function calling, all functions and their descriptions are appended to the system message, therefore doesn't run into a token limit error as opposed to OpenAI.

Solution

Append the function description to the system message instead of sending it with the function.

The implementation includes:

• Registering an AdHoc instruction
• Retrieving AdHoc instructions
• Combine the retrieved AdHoc instructions with the other adHoc instructions passed to the chat and pass all AdHoc instructions to the getSystemMessageFromInstructions function.

This correctly orders the description for the get_data_on_screen function at the end of the system message

[elasticmachine]

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[dgieselaar]

I think we can go for a different fix that doesn't special-case the get_data_on_screen function. Instead of using the function description, we can add an instruction in the same place where we are registering the get_data_on_screen function:

kibana/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.ts

Line 51 in ee341d5

this.registerFunction(

.

[viduni94]

Sounds good. Thanks @dgieselaar
I'm looking into it.

[viduni94]

Hi @dgieselaar

I implemented the above change as follows:

if (allData.length) {
      this.registerFunction(
        {
          name: GET_DATA_ON_SCREEN_FUNCTION_NAME,
          description: 'Get data that is on the screen',
          visibility: FunctionVisibility.AssistantOnly,
          parameters: {....},
        async ({ arguments: { data: dataNames } }) => {
          return {
            content: allData.filter((data) => dataNames.includes(data.name)),
          };
        },
        ['all']
      );

      const instruction: RegisterInstructionCallback = ({ availableFunctionNames }) =>
        availableFunctionNames.includes(GET_DATA_ON_SCREEN_FUNCTION_NAME)
          ? `The ${GET_DATA_ON_SCREEN_FUNCTION_NAME} function will ${dedent(`Get data that is on the screen:
            ${allData.map((data) => `${data.name}: ${data.description}`).join('\n')}
          `)}`
          : undefined;

      this.registerInstruction({ instruction, scopes: ['all'] });
    }

However, when testing I noticed that,

• When I register the instruction at the time the function is being registered, the instruction is appended to the top of the prompt, as follows:

{
      role: 'system',
      content: 'The get_data_on_screen function will Get data that is on the screen:\n' +
        '.es-query: An available rule is Elasticsearch query.\n' +
        'observability.rules.custom_threshold: An available rule is Custom threshold.\n' +
        'xpack.ml.anomaly_detection_alert: An available rule is Anomaly detection.\n' +
        'slo.rules.burnRate: An available rule is SLO burn rate.\n' +
        'metrics.alert.threshold: An available rule is Metric threshold.\n' +
        'metrics.alert.inventory.threshold: An available rule is Inventory.\n' +
        'logs.alert.document.count: An available rule is Log threshold.\n' +
        'xpack.uptime.alerts.tlsCertificate: An available rule is Uptime TLS.\n' +
        'xpack.uptime.alerts.monitorStatus: An available rule is Uptime monitor status.\n' +
        'xpack.uptime.alerts.durationAnomaly: An available rule is Uptime Duration Anomaly.\n' +
        'xpack.synthetics.alerts.monitorStatus: An available rule is Synthetics monitor status.\n' +
        'xpack.synthetics.alerts.tls: An available rule is Synthetics TLS certificate.\n' +
        'apm.error_rate: An available rule is Error count threshold.\n' +
        'apm.transaction_error_rate: An available rule is Failed transaction rate threshold.\n' +
        'apm.transaction_duration: An available rule is Latency threshold.\n' +
        'apm.anomaly: An available rule is APM Anomaly.\n' +
        '\n' +
        'You are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n' +
        '\n' +
        "  It's very important to not assume what the user is meaning. Ask them for clarification if needed.\n" +
        '\n' +
        '  If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation........',
      function_call: undefined,
      name: undefined
    }

This is probably because the system instructions are registered afterwards.. do you have any thoughts on this?

[dgieselaar]

right, @sphilipse you also have a requirement to treat certain instructions differently, correct? is that about ordering or some other mechanism?

[sphilipse]

Nah that was about the way we use scopes to filter out some things in some cases, but I think that's resolved with the work we're doing in #196322.

We already have ways to fix ordering, right? System instructions are added to the top, but there are separate instructions that are added lower. You just need to not register them as an AdHocInstruction (see: https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts and https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/util/get_system_message_from_instructions.ts)

[viduni94]

Thank you @sphilipse
I've updated the PR with the implementation to register adhoc instructions.

With this implementation, I am able to register the description of the get_data_on_screen function as an adhoc instruction and order it at the end of the system message as specified by the getSystemMessageFromInstructions function.

cc: @dgieselaar

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e3af6ec
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-197126-e3af6ec01a85

Metrics [docs]

✅ unchanged

History

• 💔 Build #247008 failed d135fac
• 💚 Build #246414 succeeded 9260ab1
• 💔 Build #246377 failed a4974d3
• 💔 Build #246332 failed b19f5ab
• 💛 Build #246280 was flaky 72e430183adb55a5b7a05d84abb03c4ca5276711

cc @viduni94

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11597841867

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Could not create pull request: Validation Failed: {"resource":"Issue","code":"custom","field":"body","message":"body is too long (maximum is 65536 characters)"}

Manual backport

To create the backport manually run:

node scripts/backport --pr 197126

Questions ?

Please refer to the Backport tool documentation