Automatically migrate default ILM policy to Data Stream Lifecycle Management

[axw]

In 8.15.0 we enable the apm-data plugin by default in Elasticsearch, which ships index templates that make use of data stream lifecycle by default. It is still possible to opt into ILM (except in Serverless projects, which do not support ILM).

There was a bug in 8.15.0 that we fixed in 8.15.1 (#13898): data streams upgraded from older versions would be "unmanaged" on upgrade. Data stream lifecycle is only automatically configured on newly data streams, so we need to keep some ILM configuration for existing data streams.

This leaves us in an indefinite undesirable state:

• new data streams use data stream lifecycle
• old data streams use ILM, unless manually migrated

We should automatically migrate existing data streams from ILM to data stream lifecycle management when the existing data streams use the default ILM policy. The retention durations are aligned, and the default ILM policy does not use any advanced functionality (e.g. hot/warm/cold tiering), so this should be considered a non-breaking change.

[pilsy]

It's not really an enhancement, it's more of a bug -- how do i migrate this myself? none of the tracing stuff works anymore in our instances... (we are spending 25k a year on them, and they no longer work)

[axw]

@pilsy which part's a bug? There's definitely a bug in 8.15.0, for which there's a fix in 8.15.1: #13898

Is there something else affecting you too?

[pilsy]

@axw support eventually reached out and helped me fix the problem, basically when the cluster was upgraded to 8.15.0 all tracing, logging and metrics stopped displaying in the interface, even tho the data was still being ingested... and a simple upgrade to 8.15.1 did not automagically fix it.

[axw]

@pilsy thanks, I will follow up on the support case and see what else we can/need to do. Sorry about the trouble this caused you.

EDIT: I see the resolution was to force a rollover to the data streams. This is related to elastic/elasticsearch#112781