Add warnings about cloning index templates

[kilfoyle]

As part of #1395, this updates the Best practices for integration assets page to add new warnings.

@lucabelluccini one question: i've added the warning about "Cloning the index template of an integration package" as suggested in the issue, but isn't this kind of repeating what's already covered in the warning about Copying Fleet and Elastic Agent integration assets?

Anyhow, please let me know whatever changes we might need.

Closes: #1395

---

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[kilfoyle]

@elasticmachine run docs-build

[lucabelluccini]

@lucabelluccini one question: i've added the warning about "Cloning the index template of an integration package" as suggested in the issue, but isn't this kind of repeating what's already covered in the warning about Copying Fleet and Elastic Agent integration assets?

No, it's not the same thing - let me expand.

A very common task for users is to customize the ILM policy.

Customizing the ILM policy for integration data streams should be done just using the @custom component template of a specific data stream.
Example: I want metrics-elastic_agent.metricbeat* data streams to have an ILM policy which deletes data after 1 day?
I create metrics-elastic_agent.metricbeat@custom component template, I add the ILM policy in its setting and I rollover. The metrics-elastic_agent.metricbeat@custom will be picked up automatically by the default Index Template of the integration.

Unfortunately, we dedicate a whole page to how to customize the ILM policy of a namespace specific data stream (here).
This is an even more power-user use-case, where the customer uses separate namespaces for their data streams.
But it seems users see this as the "default" way of customizing ILM policies.

The technique of cloning the index templates to make them namespace specific is known to lead to possible problems on stack upgrades which introduce major changes to the index templates.
We have a major issue on ES side on how to make those customizations without the need of cloning index templates elastic/elasticsearch#97664 (comment).

So my recommendation with this doc PR is:

• If the user just wants to change the ILM Policy or the number of shards (or other stuff) for the datastreams of one or more integrations, but they do not need to be namespace specific, then we should promote the use of the <package>@custom component template. This usually is fine and doesn't interfere in upgrades (except if users start to use it for heavy customizations). This should be the PROMINENT/MOST ADVERTISED option as it's likely to generate less problems to users.
• If the user wants to change the ILM Policy or the number of shards (or other stuff) for the datastreams of one or more integrations, but they really need to be namespace specific, then we can recommend what we say in this doc page.

[lucabelluccini]

I've read the changes you've done @kilfoyle and they seem excellent.

I would rework the part we've mentioned in #1408 (comment)

[kilfoyle]

Thanks for the explanation @lucabelluccini! Sorry it took me a while to circle back to this PR.

• On the Best practices for integration assets page, I've expanded the warning about "Cloning the index template of an integration package":

---

• In the "Customize data retention policies" tutorial, the cloning happens in Step 3, so there I've added in similar guidance with the two different options you mentioned:

---

Please let me know whatever needs fixing up.