Improve observability metrics and dashboards [epic]

[maleck13]

What

Add a specific dashboard that shows important information from an engineering (platform and us) perspective around what the DNS operator is doing.

• Requests to external DNS providers (success, ratelimited etc) (instrumented client)
• State of a given dns record (IE is it ready/ still trying to be validated/ is it in a conflicted state etc)
• Number of reconciles for a given DNSRecord or Managed zone (controller runtime metrics)
• Total number of DNSRecords and Managed Zones in a given cluster

Other important things

https://github.com/kubernetes/kube-state-metrics/tree/main/docs#exposed-metrics

Done

Feel free to split these out into separate issues

Kuadrant/kuadrant-operator#649

• Expose and configure instrumented client metrics #206
• Expose kube state metrics
• Dashboard resource provided to be able to import this dashboard into a grafana instance
• Documentation providing details on the new metrics and how to load up the dashboard

[philbrookes]

related: #206