Datasource Error

[AlHood77]

I'm constantly getting this error popping up in Grafana. I initially thought it was due to changing the Grafana image but even reverting to the initial version when I first set up our managed service, which worked fine, it still shows up.

Other issues I have is when I create a dashboard panel, the panel eventually goes blank after a while. When I re-create the dashboard with the same query it works as expected but will then disappear again.

In addition the metrics browser is broken. It lists all the available metrics but it won't let you select from the list and therefore won't filter any available labels.

We have enabled our Managed Prometheus service via the console, GKE.

[lyanco]

Hi, what version of Grafana are you using? I'll try to reproduce this

[AlHood77]

Also we are currently on GKE 1.21.14-gke.2700

[AlHood77]

Just further context - I have 2 clusters running (prod and non-prod) - Initial set-up of our Managed Prometheus and Grafana worked as expected on both. I tried experimenting with our non-prod instance to see if I could terraform our data source as code and I also changed the Grafana version. This is when the errors starting appearing. Even after reverting back to my initial config and Grafana image I'm still getting these datasource pop-ups errors. Our prod instance is still working perfectly.

[lyanco]

This sounds like a misconfiguration in the Prometheus Frontend Proxy that's required to query Monarch. Can you verify that you have that set up correctly? Can you try going in and reinstalling/reconfiguring that plus the data source in Grafana?

[AlHood77]

I deleted my Grafana instance and it's pvc and now it's working fine. I'm not sure why trying to Terraform the data source managed to break it but I'll just manually add it for now :-)

[lyanco]

Yeah, I don't know... IIRC sometimes TF doesn't do things in order, so maybe this is a situation where it tried to set up the datasource before the service was registered and it got messed up.

Glad to hear it's working!