SSH connection drops before sending the command

[rumbu13]

Describe the bug
It seems that it doesn't work through SSH.

To Reproduce
Steps to reproduce the behavior:

• In config, opt for ssh connection (authkey or password doesn't matter, same behaviour)

Additional context
Before each remote command, EOF is received and the response is consequently empty

Logs*

2024-10-15 17:00:36 - INFO - Starting resource allocation process...
2024-10-15 17:00:36 - DEBUG - Inside run_command: use_remote_proxmox = True
2024-10-15 17:00:36 - DEBUG - Executing command remotely.
2024-10-15 17:00:36 - DEBUG - Running remote command: pct list | awk 'NR>1 {print $1}'
2024-10-15 17:00:36 - DEBUG - Attempting to connect to Proxmox host via SSH...
[...]
2024-10-15 17:00:36 - INFO - Connected (version 2.0, client OpenSSH_9.2p1)
[...]
2024-10-15 17:00:37 - DEBUG - userauth is OK
2024-10-15 17:00:37 - INFO - Authentication (password) successful!
2024-10-15 17:00:37 - DEBUG - SSH connection established successfully.
[...]
2024-10-15 17:00:37 - DEBUG - [chan 0] EOF received (0)
2024-10-15 17:00:37 - DEBUG - Remote command 'pct list | awk 'NR>1 {print $1}'' executed successfully. Output:
2024-10-15 17:00:37 - DEBUG - [chan 0] EOF sent (0)
2024-10-15 17:00:37 - DEBUG - Dropping user packet because connection is dead.
2024-10-15 17:00:37 - DEBUG - Dropping user packet because connection is dead.

[fabriziosalmi]

The issue you're encountering seems to stem from the SSH session terminating unexpectedly, leading to an empty response when attempting to execute the command remotely. This could be due to multiple reasons, ranging from session timeout, incorrect command execution, or configuration issues. Here’s how to approach and troubleshoot the issue:

Key Issues from Logs:

1. EOF received immediately after command execution: This indicates that the SSH connection is being closed or terminated unexpectedly after the remote command runs, causing no output to be returned.

2. Dropping user packet because connection is dead: This suggests that the SSH connection may be timing out, disconnecting, or otherwise terminating during the command execution.

Troubleshooting Steps:

1. Verify SSH Configuration:

   • KeepAlive settings: Check if ClientAliveInterval and ClientAliveCountMax are set in your SSH server configuration (/etc/ssh/sshd_config on the Proxmox host). These control how long the server waits before considering the connection as inactive. For instance:

     ClientAliveInterval 60
     ClientAliveCountMax 3

     This ensures that the connection remains alive even during longer commands.

2. Check for Background Processes: Sometimes, the remote command could spawn a background process that causes the session to close. Run the command locally or in a manual SSH session and confirm it works as expected.

3. Add Timeout to the Command: The command being executed (pct list | awk 'NR>1 {print $1}') might take longer than expected to execute. Adding a timeout or debugging it by running it manually on the host could help:

   timeout 30 pct list | awk 'NR>1 {print $1}'

   This will help isolate whether the command itself is taking too long or encountering an issue.

4. Check SSH Logs: Investigate the Proxmox system's SSH logs for additional information on the disconnection. Look at /var/log/auth.log for any SSH-related messages, which may help determine why the connection was dropped.

5. Test with Different Commands: Try simpler commands through SSH to see if the issue is with the particular command or the SSH session in general. For example, use echo "Test" or uptime to see if they return correctly.

6. Inspect Command for Background Process: Ensure that the pct command isn’t unintentionally spawning processes that could lead to session disconnection. Also, try running the command with logging and error redirection to capture any output, e.g.,:

   pct list | awk 'NR>1 {print $1}' > /tmp/pct_output.log 2>&1

   Then check /tmp/pct_output.log for any clues.

7. Verify SSH Version Compatibility: Ensure that the OpenSSH client on your local machine and the SSH server on the Proxmox host are fully compatible, as some versions may exhibit odd behavior with particular configurations.

8. SSH Verbose Output: Enable verbose output for SSH to gather more information on the session:

   ssh -v user@hostname "pct list | awk 'NR>1 {print $1}'"

   This will provide detailed information about each step in the SSH connection process.

Further Diagnostics:

• Test with both password and key-based authentication to rule out any authentication-related issues.
• If the issue only occurs with certain commands, look for any resource limits or restrictions imposed by Proxmox, such as CPU, memory, or disk quota, which might cause the session to drop.

⚠️ If you can then share logs or output here it will be really useful since I failed into reproduce the issue.