[Performance] Tuning Vald cluster performance with Pyroscope

[vankichi]

Background:

We are always conscious of performance when developing and improving our services, but we would like to further improve our services by utilizing cloud native solutions.

Describe the solution you'd like:

Identify and improve performance

Describe alternatives you've considered:

Using these tools to Investigate bottlenecks:

• vald-benchmark-operator: load test to the Vald cluster
  • https://vald.vdaas.org/docs/performance/continuous-benchmark/
• Pyroscope ( + Grafana + Jager): search bottlenecks using monitoring/trace tools.
  • https://vald.vdaas.org/docs/user-guides/observability-configuration/