DCGM needs a restart after creating MIG's via instaslice and Container mapping for MIG usage is not exported.

[bharathappali]

When I tried to track the MIG usage (created via instaslice) with DCGM metric DCGM_FI_DEV_FB_USED I couldn't see any data related to that query

Later I have restarted DCGM and I was able to see the MIG usage (but the label's exported_pod, exported_container and exported_namespace are not available). MIG's created via MIG manager have these labels exported along with the container details which was consuming the MIG.

In case of Instaslice we need to query the allocation section in instaslice object and extract the GPU UUID and then query the DCGM metrics which match the label GPU UUID value to record the GPU usage of the container.

Will add more context and my finding shortly in the comments of this issue.

[bharathappali]

Instaslice object:

[bharathappali]

Metrics dashboard shows no data points found for the MIG profile