track "dashboard experience" (load failures, data duration)

[woodsaj]

Issue by Dieterbe
Thursday Aug 27, 2015 at 10:36 GMT
Originally opened as raintank/grafana#433

---

we should track:

• any panels trouble getting the data from the backend
• how long it takes for panels to get the data they need, and also for annotations.

ideally, the metrics should be separated by how long ago are they querying, and how long is the period.
but not a super high accuracy to keep the metrics space sane. I'm thinking something like

panels.load.failures.7d.0d
panels.load_duration.<datasource>.20d.2d

the xd.yd at the end would be how many days ago is the from, and the y would be how many days ago is the until, with per-day resolution.

[woodsaj]

Comment by Dieterbe
Monday Aug 31, 2015 at 05:04 GMT

---

if the clientside js code can post the metrics back to the grafana backend, we can submit them from there through statsd.
alternatively, we can also implement this in graphite-api probably. but in grafana seems nicer because it works with any datasource and gives a true-er insight.

[woodsaj]

Comment by Dieterbe
Monday Aug 31, 2015 at 05:04 GMT

---

@torkelo @woodsaj

[woodsaj]

Comment by torkelo
Monday Aug 31, 2015 at 06:29 GMT

---

Yes, this is absolutist something we need. I would implement this in a generic way for the frontend to take measurements (timings), that are sent to the backend, that are then sent to statsd with a very long interval (like a minute or more) so we can get very good percentiles.

Frontend timing measurements are going to vary by orders of magnitude so it is very important to have something like statsdaemon infront with a flush interval long enough for us to get good percentiles.

I already have some timing/perf measurement code in the frontend (nothing that sends to the backend at the moment, but prints to the console). You can enable this executing this in the js console:

localStorage.setItem('profilingEnabled', 'true')

This will print some dashboard loading times, number of angular scopes, digest timings etc. I have some panel timing stuff in place. all this could be extended and coupled with some component that can send them to the backend as well.

[woodsaj]

Comment by Dieterbe
Monday Aug 31, 2015 at 08:15 GMT

---

that are then sent to statsd with a very long interval (like a minute or more) so we can get very good percentiles.

problem is statsd, statsdaemon, dogstasd just use 1 interval that gets applied to all incoming metrics. currently this is set to 10s. we could also use go-metrics to maintain long running weighted percentiles and get those metrics via expvar. anyway i'll gladly take care of the backend if somebody can do the frontend.

[woodsaj]

Comment by torkelo
Monday Aug 31, 2015 at 08:28 GMT

---

I can do the frontend
grafana/grafana#2624

[woodsaj]

Comment by woodsaj
Wednesday Sep 02, 2015 at 15:48 GMT

---

This is really great as it will also allow us to store more customer-experience metrics as well.

[woodsaj]

Comment by nopzor1200
Saturday Nov 21, 2015 at 01:30 GMT

---

I'd really like to rekindle this ticket and move forward on this.

Big picture wise, I feel like we're flying blind (relatively speaking) with this kind of data.