[Support finer elasticsearch index storage]

[samz-github]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In my cooperation, we use skywalking to monitor high traffic microservice. Now everyday there are almost elasticsearch 2TB storage size only for the segment index. And it will be grow more storage size as more related microservices are added to Skywalking system. For now, 2TB in only 1 segment elasticsearch index is too big. It will be better if Skywalking can support fine grained segment index for each monitored microservice but not only 1 segment index each day.

Use case

For example, we use skywalking monitor three microservices: account-svc, payment-svc, product-svc. For now, each day there is 1 segment elasticsearch index to store the Skywalking segment records, index name: sw_segment-20221010, 2TB size for this index. It will be good if skywalking can divide this index to three indices: sw_segment-account-svc-20221010, sw_segment-payment-svc-20221010, sw_segment-product-svc-20221010.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

That is a dead end. SkyWalking decreases the number of indices from version to version because too many indices is an over kill for ElasticSearch cluster.
From ElasticSearch's perspective, you should set the number of shards to scale out, rather than put a service-based index.
If you take the math, you could face TTL(N days) x Number of Service(100+ in a distributed system) ~= 800+ indices. Your server would dead.