[BUG] Tracing index is not re-created in opensearch. Dataprepper needs restart?

[AdaptiveStep]

Describe the bug
When events are sent to opensearch, usually the index is created if it doesn't exist. This happens for all data except when dataprepper recieves traces. When dataprepper starts up, it creates the necessary tracing indexes for spans and servicemaps, once but never again unless restarted.

If the index is removed during dataprepper runtime, an error saying "index is missing" shows up extremely often, possibly filling up the buffer and eventually causing packetdrops.

To Reproduce
Send traces to dataprepper as per usual. And you will see the trace index in the management/index page on in the "opensearch dashboards gui".

However, if you delete the index, it never gets recreated again! Even if new traces are being sent to dataprepper! Only re-starting dataprepper seems to "recreate" the index again. This can probably be easily fixed so that indexes are recreated if they don't exist in opensearch.

Expected behavior
1: the Span Index needs to be re-created if it doesn't exist when new events come to dataprepper. (And when they are sent to opensearch).

2: the serviceMap index needs to be re-created if it doesn't exist.

Environment (please complete the following information):
I tried this in dataprepper on kubernetes using the dataprepper helmchart.

Additional context
I tried this using the otel demo apps. It seems pretty consistent with all their traces. If the index is removed, it never gets re-created again unless dataprepper is restarted. Neither the "service-map index" nor the "span index" get recreated.

[KarstenSchnitter]

This might be related to #3342 and maybe #3506. The index setup used for spans is a little complicated. It usually uses a write alias, that points to a concrete span index.

@AdaptiveStep can you elaborate on your setup? Do you use the default index configuration or do you provide a custom config? When you delete the current index, do you keep the write alias if you have that? Can you provide the error log of DataPrepper, that contains the "index missing" message?