Create metrics for instaslice

[kannon92]

We don't have any metrics exposed for this operator. We should think of some metrics that operations would find useful for monitoring the health of this service, placement of GPUs, etc.

[kannon92]

@asm582 I think we should consider this a must for release. Can you lead this effort? I don't have a good understanding to know what an admin would want to see from this project for observability.

[tardieu]

• tracking GPU slice allocations. First, which slot is in use vs free on each GPU. Which slices are in use (ie how slots are fused).
• tracking GPU slice requests from pending pods.

[harche]

xref : https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/enable-gpu-monitoring-dashboard.html and https://github.com/NVIDIA/dcgm-exporter

[harche]

xref : https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/enable-gpu-monitoring-dashboard.html and https://github.com/NVIDIA/dcgm-exporter

for coming up with useful prom queries from the existing metrics

[asm582]

• Expose allocations to users present in the InstaSlice object
• Expose configmap info to persona admin