Graceful Tunnel Shutdown

[def324]

I noticed an issue when stopping the container and then restarting it again immedietly after. It seems like the session was still active on the remote server and the port was used up. Therefore when it tried to reconnect it couldn't reuse the same port again. It doesn't seem to happen all the time, so I wasn't able to reproduce it reliably.

It think this could be fixed by a more graceful shutdown, something like this at the beginning of the entrypoint.sh script:

cleanup() {
    pkill -3 autossh
}
trap 'cleanup' SIGTERM

Please let me know your thoughts.

[jnovack]

I've run into this issue, I know what you are talking about it. Do you know how to submit a Pull Request? Please feel free to add it so you get credit.

Please submit the PR against the v2.0 branch.

[jnovack]

Is this still valid in v2.0?

[zrthxn]

@jnovack yes this still happens unfortunately. I've had to reboot my remote every time i have to restart this container to stop the port listening on the remote. kill -9 <pid> doenst work either, it just spawns another process.

[zrthxn]

I tried building a local image with the solution suggested above and it doesn't seem to work. After doing a docker stop autossh-local, the tunnel port on the remote host stays open/listening.

[mrkeuz]

@zrthxn I made little investigation, exactly for your situation there is an ssh option -o StreamLocalBindUnlink=yes that should recreate remote binding if it already used.

  StreamLocalBindUnlink
             Specifies whether to remove an existing Unix-domain socket
             file for local or remote port forwarding before creating a
             new one.  If the socket file already exists and
             StreamLocalBindUnlink is not enabled, ssh will be unable to
             forward the port to the Unix-domain socket file.  This op‐
             tion is only used for port forwarding to a Unix-domain
             socket file.

             The argument must be yes or no (the default).

Note:
there is a some bug report about this option. But hope to your luck and all will be work as expected.
https://openssh-bugs.mindrot.narkive.com/RNGyBguq/bug-2601-new-streamlocalbindunlink-not-working

[zrthxn]

@mrkeuz I tried adding StreamLocalBindUnlink=yes to my docker compose environment like so:

    environment:
    # ...other vars
    - StreamLocalBindUnlink=yes

it doesnt seemt to be an accepted, since when autossh starts it doenst include that option in the command.
Doesn't look like its gonna work unfortunately.

[mrkeuz]

@zrthxn no, it does not work like you using, you need to pass option to autossh and it ssh subprocess. There are two options:

• modify entrypoint.sh of original docker image and add -o StreamLocalBindUnlink=yes to the autossh command
• or create config ssh_config (chmod 600) (see docs about this file) with content like below and mount it to your container to /etc/ssh/ssh_config. Simply speaking this options will be applied for all ssh connections.
  /etc/ssh/ssh_config:

  Host *
    StreamLocalBindUnlink yes
  

[zrthxn]

@mrkeuz
Correction. It does work.
Added it to entrypoint.sh and built a local image, and it closes the target port when the container is stopped.

[dmikushin]

I'm seeing the same issue, just to show the log how it happens:

autossh    | jnovack/autossh v2.0.1 revision 816f453 built 2021-04-01T10:51:38Z
autossh    | Agent pid 9
autossh    | Identity added: (stdin) (autossh)
autossh    | [INFO ] Using autossh 1.4g
autossh    | [INFO ] Tunneling 127.0.0.1:65012  on user@server:port to 172.17.0.1:22
autossh    | [INFO ] # autossh -M 0 -N -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -t -t -R 127.0.0.1:65012:172.17.0.1:22 -p 22 user@server:port
autossh    | Error: remote port forwarding failed for listen port 65012

The 65012 port is reserved on the server, and clearly should work. The reason it does not could be the previous connection was interrupted (for example, due to network problems), and the ssh endpoint on the server is orphaned - it's still working, and holding the 65012 port. New instance of autossh tries to capture the port, and fails.

Generally, this is not a problem of this docker container, and not a problem of autossh. It'a a responsibility of an ssh tunnel to dispose the server endpoint, which needs a solution. Then, the autossh should treat this error as fatal and propagate it to the docker container.

[dmikushin]

StreamLocalBindUnlink=yes should be enabled by default. If you agree, please merge #40