Invalid or unknown remote ssh hostkey (Rugged?)

[vytasmk]

A week ago I have updated Redmine to the latest v5.0.4 and redmine_git_hosting v6.0.1. All setup is running on on modified official Docker image. Previously there was Redmine v4.2.x that was made same way and git was working just perfectly.

Today I have my colleague reported that when he tries to checkout repository git asks for password and I know that public keys are not working. Went to Admin->Plugins->Redmine Git Hosting Plugin -> Rescue TAB checked all check boxes and hit save. That was to fast from the last time I have ever used this function. Later checked log file git_hosting.log which showed errors:

2023-02-21 22:16:30 +0200 [ERROR] Invalid Gitolite Admin SSH Keys
2023-02-21 22:16:30 +0200 [ERROR] invalid or unknown remote ssh hostkey

... next several hours was trying different methods: checked permissions, recreated all certificates several times, enabled debug mode on Redmine and also on SSHD process nothing helped. And it was unclear from where this error is coming from. I tried to search for that error using Google and I think I found that it is related to gem named rugged and in the latest version v1.5.1 this error message was added in the code. Rugged 1.5.0.1 -> 1.5.1 file data/vendor/libgit2/src/libgit2/transports/ssh.c line: 743

When I check gem list inside Docker container I see that there is rugged (1.5.1) I tried manually to remove this version and install 1.5.0.1 but if I restart Docker container it automatically installs all gems using bundle install as there are scripts on startup. I will change them later and try to eliminate them.

Has may be anybody sported such problem? That rugged version was released January 21, 2023. So if this is related then all new installations will get in to this problem.

[vytasmk]

Update: it looks like this is related to the libgit2 library which is used by Rugged gem. libgit2 library was updated and that error message (with much more code) was added on 19 Jun, 2023.

By checking that code found that one function is returning wrong data due to wrong return statement location. Reported issue there.

[vytasmk]

Finally got it working. If somebody will get in to the same issue here is the way I have solved this at least for now.

I added Rugged gem with limited version to the Redmine Gemfile gem 'rugged', ["~> 1.5.0", "< 1.5.1"].

Simply execute this command inside Redmine home folder /usr/src/redmine

echo "\ngem 'rugged', [\"~> 1.5.0\", \"< 1.5.1\"]" >> Gemfile

This will prevent Rugged to install 1.5.1 and newer version. Later when problem will be solved this line can be removed.

Finally execute bundle install which will install all required gems.

[sseide]

Thanks - pinning rugged to a version before 1.5.1 fixed our problem too. We had updated our redmine last week to 5.x with git-hosting 6.0.1 (on Linux/Debian) and were seeing similar errors:

2023-03-08 07:35:43 +0100 [INFO] Create Gitolite Admin directory : '/tmp/redmine...'
2023-03-08 07:35:43 +0100 [DEBUG] Accessing gitolite-admin.git at '/tmp/redmine...'
2023-03-08 07:35:43 +0100 [ERROR] Invalid Gitolite Admin SSH Keys
2023-03-08 07:35:43 +0100 [ERROR] failed to set hostkey preference: The requested method(s) are not currently supported

The error message was a bit different, therefore i mention it here for others to find.

The error message can be found at this discussion for libssh2: https://pullanswer.com/questions/ssh-fails-to-fetch-if-not-all-key-types-are-supported
This description is about an error on Windows but this problem affects Linux too it seems.

[Blackbetty312]

I have exactly same problem as above

2023-05-30 09:33:14 +0000 [INFO] Create Gitolite Admin directory : '/tmp/redmine_git_hosting/git/gitolite-admin.git'
2023-05-30 09:33:14 +0000 [DEBUG] Accessing gitolite-admin.git at '/tmp/redmine_git_hosting/git/gitolite-admin.git'
2023-05-30 09:33:14 +0000 [ERROR] Invalid Gitolite Admin SSH Keys
2023-05-30 09:33:14 +0000 [ERROR] failed to set hostkey preference: The requested method(s) are not currently supported

I have tried to fix it in many ways and nothing work.
I run redmine inside docker with that redmine version and redmine git hosting 6.0.0

  Redmine version                5.0.3.stable
  Ruby version                   3.1.2-p20 (2022-04-12) [x86_64-linux]
  Rails version                  6.1.7

@sseide Have you found a solution to this problem?

[sseide]

As i said - the solution from @vytasmk above worked for me too.
The issue he linked in his report is not close so far, therefor no other solution than forcing bundler to not install "[email protected]"

[acosonic]

It looks like you are all wrong :)

And just by adding:
PubkeyAcceptedAlgorithms +ssh-rsa
to
/etc/ssh/sshd_config

then restarting systemctl restart sshd

It worked for me, after hours of unsucessful fixes since I followed wrong leads...

till it came to my mind to take a look at /var/log/auth.log

[pkoevesdi]

I can confirm: changing the version of rugged didn't help on my machine, but adding PubkeyAcceptedAlgorithms +ssh-rsa did.
Thank You so much, it took me 8 hours to find this here.

[prahal]

I tried to generate a new redmine_gitolite_admin_id_rsa keypair.
Then I tried libgit2 and ruby-rugged 1.7.1+ds1 and libssh2 1.11.0-4 from Debian testing (with a tweak to gitolite-rugged gemspec (without tweaking PubkeyAcceptedAlgorithms as it was not working, this probably as I regenerated the redmine_gitolite_admin_id_rsa key, see next).

I refreshed the gitolite setup for the new admin key by running as the gitolite user:

cp /opt/redmine/ssh_keys/redmine_gitolite_admin_id_rsa.pub redmine_gitolite_admin_id_rsa.pub
gitolite setup -pk redmine_gitolite_admin_id_rsa.pub

I still had the same error:

[ERROR] Invalid Gitolite Admin SSH Keys
[ERROR] invalid or unknown remote ssh hostkey

Turns out it came from rugged trying to create git-admin.git in /var/cache/redmine/tmp/redmine_git_hosting/git/
but failing.
I fixed the issue by, as redmine user running:
ssh -P22 -i ~/ssh_keys/redmine_gitolite_admin_id_rsa [email protected]
(as this was the IP and port my debug shown rugged tried to clone gitolite-admin as)
which gave the warning

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
(...)

and asked me to confirm if I wanted to add the key to the know_hosts file.
Once that was done rugged was able to clone the gitolite-admin.git repository and all was fine.

Then I tried to revert to Debian stable libgit2 and ruby-rugged 1.5.1+ds-1, and also libssh2 1.10.0-3+b1
and I now had this error from rugged:

[ERROR] Invalid Gitolite Admin SSH Keys
[ERROR] Failed to retrieve list of SSH authentication methods: Failed getting response

add PubkeyAcceptedAlgorithms +ssh-rsa and restarting sshd fixed it.

So the PubkeyAcceptedAlgorithms will probably not be fixed in a 1.5 patch release but is in 1.7.1.

But even with this issue fixed, it could be that the new GHSA-8643-3wh5-rmjq "Beginning in libgit2 v1.4.5 and v1.5.1, libgit2 will now perform host key checking by default. Users can still override the default behavior using the certificate_check function.", will require a change in redmine_git_hosting to add the 127.0.0.1 localhost host key to the redmine user know_hosts file.

[ilyacherevkov]

I can confirm the actions described above are essential to get the plugin working.

I had the following errors:

2024-03-05 04:13:50 +0000 [INFO] Create Gitolite Admin directory : '/tmp/redmine_git_hosting/git/gitolite-admin.git'
2024-03-05 04:13:50 +0000 [ERROR] Invalid Gitolite Admin SSH Keys
2024-03-05 04:13:50 +0000 [ERROR] failed to set hostkey preference: The requested method(s) are not currently supported

Locking rugged version, adding PubkeyAcceptedKeyTypes +ssh-rsa into sshd_config and restarting redmine helped.

[acosonic]

On the other install with 22.04 and Redmine 5.1 I had to add both

PubkeyAcceptedAlgorithms +ssh-rsa
and

PubkeyAcceptedKeyTypes +ssh-rsa
to
/etc/ssh/sshd_config

then restarting systemctl restart sshd

[acosonic]

Finally got it working. If somebody will get in to the same issue here is the way I have solved this at least for now.

I added Rugged gem with limited version to the Redmine Gemfile gem 'rugged', ["~> 1.5.0", "< 1.5.1"].

Simply execute this command inside Redmine home folder /usr/src/redmine

echo "\ngem 'rugged', [\"~> 1.5.0\", \"< 1.5.1\"]" >> Gemfile

This will prevent Rugged to install 1.5.1 and newer version. Later when problem will be solved this line can be removed.

Finally execute bundle install which will install all required gems.

Had to do this in one Ubuntu 20 but Redmine 5.0.8 and Ruby 3 Instance... On Ubuntu 20, PubKey in sshd is not valid config option...

[renanmarks]

I struggled for the last three days to make this work on a new Redmine install. The tips from @acosonic and @vytasmk combined resolved the communication issues between the plugin and gitolite/ssh.

I just published an updated Docker container that may help other people get a working instance quickly: https://github.com/renanmarks/docker-redmine

[acosonic]

Ah, what I eventually had problem with, is having too weak password for git user... And, it was used to send spam via that server... Because user git user had email enabled, eventually I was able to figure this out...