[Request]: Update the reindex method on the Observability AI Assistant doc

[eedugon]

Description

The procedure described on the section === Use the {es} Reindex API of the document has the following issues:

1. It's too advanced.
2. The destination index proposed might not be valid if data is rolled over.
3. The first step about indexing data with the Index API is now useless considering they can do that with Search Connectors and also integrate the data directly with the AI Assistant knowledge base.

Resources

Background:

Now that we have added in #4326 the information about how Search Connectors can help to enrich the AI Assistant Knowledge base, the method of using Reindex API proposed in the doc on the section...

=== Use the {es} Reindex API

... is probably not needed (or at least not needed as it is).

I would suggest to:

• Determine the need of sharing such an advanced procedure to reindex data directly towards a system index (.kibana-observability-ai-assistant-kb-000001) now that there are other ways to enrich the AI Assistant KB.
• Consider moving this type of procedure to an Elastic KB article instead of keeping it in the public document.

If we want to keep the content, then the following updates should be needed (IMO):

• Add an introduction about when that procedure might be useful (for example to include existing data in Elasticsearch to the AI Assistant knowledge base)
• Update the destination index of the reindex operation to use an alias and not a hardcoded index that could be rolled over.
• Explain or add a note about this being an advanced procedure.
• Consider a troubleshooting section (that's also another reason to move this to a Knowledge Article instead of keeping in the main doc.

cc: @lucabelluccini , if possible I'd love to hear your opinion on this.
cc: @emma-raffenne , as this is a follow up of our previous work, could you check this with @dgieselaar or someone else from the team?

Which documentation set does this change impact?

Stateful and Serverless

Feature differences

Both docs are very similar so in case of moving forward with this we should update both documents.

What release is this request related to?

8.16

Collaboration model

The documentation team

Point of contact.

Main contact: @eedugon

Stakeholders: @emma-raffenne

[emma-raffenne]

@eedugon
Thank you for creating this issue.
As @dgieselaar mentioned elsewhere, we don't want to recommend this method for adding content to the Knowledge Base, so IMO it's safe to remove it entirely from the user documentation.

[eedugon]

I'm closing this issue as we have already removed the method as part of the linked PR. Adding some final notes:

• Serverless docs still have this method documented, because in serverless the Search Connectors are not an option (yet) to populate the AI Assistant knowledge base.

• I still believe that this method could be useful on certain advanced use cases, and it might be worthy to mention it slightly and linked to a Knowedge Article or to the original blog post where this was well explained:

https://www.elastic.co/observability-labs/blog/sre-troubleshooting-ai-assistant-observability-runbooks

[eedugon]

Reopening again after a conversation with @leemthompo :) cc: @bmorelli25

I really believe the method is useful and interesting for certain use cases, but the way it was documented before was an opened door for issues and misunderstandings, so let's try to do any of:

• Improve the way we present it to not cause confusion.
• Just describe the method and point to the example on the blog post or to a specific knowledge base article showing it in details.

[emma-raffenne]

cc @teknogeek0

[teknogeek0]

I'm just catching up on this. Can someone directly link the docs page(s) in question?

[eedugon]

The content was removed in 8.16 in favor of using search connectors to populate the AI assistant knowledge base.

But you can still go through it in the 8.15 version of the doc: https://www.elastic.co/guide/en/observability/8.15/obs-ai-assistant.html#obs-ai-index-api

The method is still useful for certain use cases, but it would require a few more explanations and improvements to not create confusion.
Also that method was added to the docs from this blog post (which I believe it's the original seed): https://www.elastic.co/observability-labs/blog/sre-troubleshooting-ai-assistant-observability-runbooks

We have to:

• Decide if we bring the content back to the docs (or to a KB article and link the KB article from the doc).
• Ensure we provide proper background and introduction for the method, like:

"If you already have data in your Elasticsearch cluster and you want to use it to populate the AI Assistant KB you can use the reindex API in the following way.
or...
if the external data you want to bring into the AI assistant KB cannot be fetched with search connectors you can still index it manually using the index API (procedure out of the scope of this document) and then reindex it in the following way."

[emma-raffenne]

Since we are migrating the Obs AI Assistant Knowledge Base (KB) to use the (new) inference API, I would hold on reintroducing this in the doc as the method will change.