Allow custom labels for gatekeeper_violations

[grosser]

We want to audit a large list of resources (5k+).
Putting the violations onto the rules will not scale since we'd hit etcd object limit.
So we need gatekeeper_violations to b the source of "what is a violation".
For this we'd like to use "project" label of each resource, so that we can say "hey team foo your bar project has 25 violations".
I understand this can be a memory leak for prometheus and is very bike-sheedy, so this should be opt-in functionality.
Additionally it should track kind either by default or via opt-in.

[maxsmythe]

Clarification: you want the prometheus metric gatekeeper_violations to be the source of truth for violations?

Any thoughts on using logs for this? It should have most of the needed data (no data on what labels the violating object has though):

https://open-policy-agent.github.io/gatekeeper/website/docs/audit#audit-logs

[grosser]

logs would also work since we can get stats from them via log search or logrecycler but we'd still need the extra labels
if adding the extra labels is similar amount of effort then I'd rather have it on the metric so I don't have to parse logs
also a downside of logs would be that summing will be hard whereas violations are a gauge and are instantly usable

[grosser]

the audit logs could be used to fetch the resource and then do custom label metrics from that, but that's also not easy or performant :(

[ritazh]

related #1037

[maxsmythe]

TBH logs seems like a better fit for a few reasons:

• No worries about cardinality (as opposed to Prometheus metrics)
• No worries about PII (whereas prometheus metrics are not necessarily authenticated/encrypted)
• We can write arbitrarily complex outputs

To expand on the last point, we could essentially log the entire violating object if we wanted, letting users take the object labels, annotations, or whatever else was interesting to them.

For Prometheus metrics, labels are flat (i.e. not namespaced). That means if we allowed users to specify custom labels (like project), we would have a hard time adding new labels in the future. Imagine we wanted to add our own project label, that would potentially break your use case. Because we would have no way of knowing what labels users would be injecting, we'd have no "guaranteed safe" label name reserved for us to use.

[grosser]

For labels: they could be prefixed with custom-

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.