Configure detailed metrics

[erthalion]

Description

Currently, Collector exposes two types of prometheus metrics: regular aggregates and per-syscall aggregates. The latter one is quite verbose and produces more than half of the whole output (181 out of 284 lines), thus it makes sense to make it configurable to be able to disable in the future.

From one side it would enable us to finally expose Collector prometheus metrics, at the same we still would be able to use per-syscall metrics for troubleshooting in development environment.

Checklist

• Investigated and inspected CI test results
• Updated documentation accordingly

Automated testing

• Added unit tests
• Added integration tests
• Added regression tests

Testing Performed

Local testing.

[Molter73]

Wonder if it would make sense to have this as a run-time check instead of compile-time. Thinking from a user of collector, the compile time check would mean a new image needs to be pulled and deployed (maybe even built, since I don't expect us to maintain multiple versions of the image), whereas a run time check could be toggled via an env variable, have the container restart and presto! The metrics are now collected or ignored. We could still have it so these metrics are collected by default. WDYT?

[erthalion]

Wonder if it would make sense to have this as a run-time check instead of compile-time. Thinking from a user of collector, the compile time check would mean a new image needs to be pulled and deployed (maybe even built, since I don't expect us to maintain multiple versions of the image), whereas a run time check could be toggled via an env variable, have the container restart and presto! The metrics are now collected or ignored. We could still have it so these metrics are collected by default. WDYT?

We can, but to be honest I'm not sure there is a use case for switching between short/detailed metrics mode at runtime. The latter one most likely would be used for low level troubleshooting during development (i.e. when some tricky Falco modifications are not playing nice), but how could they be useful in the field?

[Molter73]

but how could they be useful in the field?

I assume they could be used to troubleshoot problems with specific syscalls that might be being dropped, but honestly 🤷🏻‍♂️

[erthalion]

Ok, don't see why not as well, so lets try it.

[Molter73]

LGTM! Left a few comments, but nothing major left IMO.