Check timeout on ssh calls (bsc#1213550)

[nadvornik]

What does this PR do?

What issues does this PR fix or reference?

Fixes: https://bugzilla.suse.com/show_bug.cgi?id=1213550

Previous Behavior

When executing ssh or scp command, the timeout was checked only on connection.
When the command got stuck after connecting, it could run forever.

New Behavior

Total execution time is limited.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes/No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

[agraul]

I am a bit concerned about introducing this type of timeout. It's hard to find a good value for it, since we don't know what will be executed via Salt SSH and how long that will take. We don't want to run into a timeout when state.apply is doing its things as expected. But using a value that's too large won't help us much either.

In Uyuni we use SALT_SSH_CONNECT_TIMEOUT = 180 which would time out after 6 minutes with this implementation. That might cause quite a number of false positives.

We probably want to have this discussion upstream. If this is accepted, there are additional changes needed (e.g. updating the documentation and tests).

[agraul]

I just had a discussion with @mcalmer about this and we both think that a general timeout does not make sense. We just can't predict what someone wants to execute over Salt SSH and whether the execution is slow or came to a stall.

If it is possible to detect that nothing is being transferred in the scp case, that would be a better alternative. Maybe ClientAliveInterval helps, but we'd need to test that.