Healthcheck plugins on dispense, re-instantiating broken ones

[nick-kentik]

Disclaimer: I don't really think this is the correct solution. It's just a first pass at it, with the most important bit being the test that reliably reproduces the problem. Very happy to work on the solution in a different way.

At the moment, if an external plugin that nomad-autoscaler relies on dies for any reason (OOM, for instance, or someone coming along and running kill on it), the nomad-autoscaler process cannot recover. Instead, we see a steady stream of errors of the form rpc error: code = Canceled desc = context canceled for the remaining lifetime of the process.

Attempt to solve this by:

• Running the PluginInfo RPC against plugins when they are being dispensed
• Killing the plugin if that fails, removing it from the plugin instances map
• Re-running dispensePlugins() if a request is made for a plugin not in the instance map

This seems like the minimal approach in terms of moving code around, but I suspect it has a significant performance impact, and probably spams the logs heavily in a range of scenarios. Still, it's a starting point.

Closes #711

[hashicorp-cla-app]

All committers have signed the CLA.

[nick-kentik]

Ping?