Review reverse ssh tunnel process

[seanshahkarami]

This is such a critical component that we should take some time to make sure this is designed the way we want now.

To get some discussion started, I'd recommend we think about the following:

1. Do we get enough logging information from this container to say anything useful? If not, we may want to see what we can do to improve that. For example, it would be nice to know that a particular public key is being rejected. We could even combine this with item 2 and have an sshrc post a log message to the logging / analytics pipeline when someone connects and use standard tools like who to see who's connected.

2. We may want to have a per-node home directory for additional security and deployment reasons. We already restrict what commands a particular user is able to run, but it wouldn't hurt to sandbox access even more. Further, this could allow us to do something like put per-node config / parameters in their own directories to allow them to pull new configurations or credentials on their own. It could essentially be a set of synced home directories.

I'd argue that this approach isn't much harder than we have now and could be built out in parallel until it's signed off on.

It's not much harder to have a script which rebuilds multiple directories / authorized_keys files as opposed to a single large one. The only additional step would be ensuring that each user is added to the ssh container. I'd imagine this step is a pretty fast and could be run at container startup or on a schedule.

Just something to think about.

[seanshahkarami]

I went ahead and simplified the sshd server logging. Turns out there's a built-in flag that does essentially the same thing as the run_sshd.py file.