Supervisor PID client is None after outage

[jlashner]

We saw some very strange HWP behavior today which is likely related to an outage.

Grafana Logs

The supervisor seems unable to connect to the PID client, with logs like:

Completed with state: ControlState.Error(traceback='Traceback (most recent call last):\n  File "/opt/venv/lib/python3.10/site-packages/socs/agents/hwp_supervisor/agent.py", line 1087, in update\n    self.run_and_validate(clients.pid.declare_freq,\nAttributeError: \'NoneType\' object has no attribute \'declare_freq\'\n', start_time=1729869143.9402528)

After the outage, Bryce says they manually restarted the main process, but not the entire supervisor, putting it in a weird state. The supervisor should be robust to this, and should be sure not to attempt to run any operations if it cannot communicate to all necessary agents.