External data staleness monitoring

[konklone]

We had our first server bug that delayed data updates for a time. We should have something that watches the real-time data feeds and alerts us if it sees stale data.

I suggest performing a HEAD request to https://analytics.usa.gov/data/live/realtime.json and looking for whether its Last-Modified header has become more than 5 or 10 minutes old.

[ArcTanSusan]

Do you have a preference for what tool to use in order to notify folks if data ever is more than 5-10 mins old?

[konklone]

We haven't dived into it too much. Got any ideas?

[ArcTanSusan]

Since the dashboard is entirely front-end w/o a backend, we can try a number of email-sending JS tools such as https://github.com/eleith/emailjs.

[konklone]

That's only appropriate for back-end use, as if it's used in the front-end it would expose the authentication credentials for the SMTP server.

[ArcTanSusan]

Hmm, this seems to be an ops task. Maybe a cron job that calls a python script to make requests to that url and check for the timestamp headers?

[konklone]

Yep, I think it could be relatively self-contained. Ideally, it would be a little general-purpose open source monitoring app that we can host on a Heroku-like system, and which can take enough parameters to do this without much/any custom coding. I'm not sure if something like that exists, but it would be the ideal.

[ArcTanSusan]

@konklone: Thoughts on the nodeJS project https://github.com/iloire/watchmen?

[levinmr]

@smarina04 this is a potential feature. We currently have no way to indicate if data stopped updating for the site. It could be worth alerting somehow if data reports used for the charts has a timestamp older than expected