Cant create SFTP repository. Kopia does not support ed25519 keys

[friki67]

Newbie here. Trying to configure a SFTP repoository using KopiaUI in Windows server 2012r2 I'm getting this error:

Connect Error: INTERNAL: internal server error: unable to open SFTP storage: error establishing connecting: unable to dial [myhost:20021]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"myuser", Auth:[]ssh.AuthMethod{(ssh.passwordCallback)(0x1015520)}, HostKeyCallback:(ssh.HostKeyCallback)(0x10119e0), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch
I've created the know_hosts file using openSSH.

Trying to create the repository using CLI in windows makes the same result:

C:\Users\Administrador\AppData\Local\Programs\KopiaUI\resources\server>kopia repository create sftp --path=kopia/docs --host=myhost --port=20021 --username=myuser --sftp-password=mypass --known-hosts="C:\Users\Administrador\.ssh\known_hosts" 

ERROR can't connect to storage: unable to open SFTP storage: error establishing connecting: unable to dial [myhost:20021]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"myuser", Auth:[]ssh.AuthMethod{(ssh.passwordCallback)(0x1015520)}, HostKeyCallback:(ssh.HostKeyCallback)(0x10119e0), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch
So why this key mismatch? How to get the "right" host key for kopia?

[friki67]

Ok, so I found this https://kopia.discourse.group/t/kopia-and-sftp-repos/183 forum post and used
ssh-keygen -p 20021 myhost >> known_hosts
Using this file fixed my issue with the key and I was able to create my repository.

Thank you for this great software

[friki67]

I'm opening this again because I'm having an issue creating a sftp repository.

Following https://kopia.discourse.group/t/kopia-and-sftp-repos/183 that has worked in the past now throws this error when creating repository:

Connect Error: INTERNAL: internal server error: unable to open SFTP storage: error establishing connecting: unable to getHostKey: mymachine.domain.com : knownhosts: c:\users\administrador\.ssh\known_hosts:1: illegal base64 data at input byte 0

I've created known_host in the same way I did it in the past, using ssh-keygen (openssh for windows).

Could you help me?

[ingul-buz]

I think you are confusing two different things here.

With ssh-keygen you create a key that you use for logging into your host. It actually creates a key pair, a private and a public key, latter of which you should copy and paste into the file ~/.ssh/authorized_keys in your home directory at your host. The file can include several public keys. But if you only use a password, this is not necessary.

The known_hosts file is created automatically when you connect to a previously unknown host for the first time. Since you are using OpenSSH, you can use the simple command ssh your_username@your_host.com. If ssh cannot find a key for this host from known_hosts, it will tell you so and ask if you want to continue. After you answer "yes", the host key will be saved into known_hosts and you can use that from there on. On Windows it's saved into %USERPROFILE%\.ssh\known_hosts. Also this file can include several host keys, even for different hosts.

So, you are confusing your key, which is for authenticating you to the host, with a host key, which is for authenticating the host to you. Keep in mind that the host key (in known_hosts) may change occasionally (generally not that often), after which you have to connect there again using ssh just like above in order to collect the new key.

I know this sounds complicated, and to some extent it is. In your first post it seems there either wasn't a key for this host in your known_hosts, or it had changed (knownhosts: key mismatch). Then in your second post you created a key pair and used that as a known_hosts, which shoudn't work. The error in your latest post indicates that your known_hosts now includes incorrect data (illegal base64 data at input byte 0), which is to be expected. You should now clean your known_hosts. It's a plain text file, so you can open it with a text editor. If your failed attempt is the only data included, you may as well delete the whole file. Normally a known_hosts consists of one or more lines looking like this:

example.com,93.184.216.34 ssh-ed25519 AAAAC3NzaC1lZD...<key truncated>

Then try and connect to your host as I told above, and you should be good to go.

[friki67]

Hello!

I'm trying to get this working again after having problems with another backup solution, and after trying to use WebDAV repository (it is not working for me, see https://kopia.discourse.group/t/invalid-length-when-restoring/1741/3)

I understand your explanation, and I've followed your indications:

• ssh myuser@myhost -p 20021, created known_host file in c:\users\administrator\.ssh
• use this path for "Path to known_host File"

And then I get the "key mismatch" error.
Connect Error: INTERNAL: internal server error: unable to open SFTP storage: error establishing connecting: unable to dial [myhost:20021]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"leleman", Auth:[]ssh.AuthMethod{(ssh.passwordCallback)(0x10b58a0)}, HostKeyCallback:(ssh.HostKeyCallback)(0x10b1d60), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch

Same thing happens when trying to do it from a linux desktop, and from the command line.

I've been searching and I've found this SFTP repository won't work if remote SSH server using non standard 22 port.

Could I fix this in any way? Am I missing something?

[ingul-buz]

Hmmm. I think you may actually have come across a shortcoming in the way Kopia's internal ssh client handles the known_hosts data. This may have been your problem already in the first place.

I tried to figure out why your key would mismatch, and experimented a little. It turned out that Kopia doesn't understand ssh-ed25519 keys at all, and if that's the only key type for your host in known_hosts, Kopia will throw exactly the error you got. Other keys (at least ssh-rsa and ecdsa-sha2-nistp256 keys that I have) seem to work fine.

You could try the command ssh-keyscan -p 20021 <your_host>, which should list all available keys for your host (don't care about the lines beginning with a #). If there's only an ssh-ed25519 key available, you're probably out of luck (regarding the internal ssh client). If the list contains any other keys, copy one or all of those lines to the bottom of your known_hosts file and try to connect again.

You could also use an external ssh client (as you already seem to have OpenSSH), but that requires creating a personal key file for logging into your host. That procedure has its own quirks, so let's leave it as a last resort.

A non-standard port doesn't seem to be a problem. That was the conclusion also in the issue you linked to.
Edit: I noticed that issue actually states that you cannot use non-standard ports with an external ssh client, so there goes our "last resort" above.

As a common observation, I'm getting really tired of this whole mess that is Kopia's mandatory known_hosts data. It's already not easy to understand for many people, and now it seems you may not get it right even if you know what you're doing. Could we please have this thing made optional?

[friki67]

Hello! And thank you.

You could try the command ssh-keyscan -p 20021 <your_host>

And yes ed25519 is the only key available. I'm using this docker container as sftp server: https://github.com/atmoz/sftp

So I have to use the "last resort" path. :(. I'll try to get it working.

[ingul-buz]

Well, that probably won't work either. As I said in my edit above, the issue you mentioned about non-standard ssh ports not working seems to apply only when using an external ssh client. I haven't tried it myself though.

I'm sorry but you seem to be hitting a snag on every front here.

1. External ssh client option doesn't support non-standard ssh ports
2. Internal ssh client doesn't support ed25519 keys in known_hosts
3. Known_hosts is mandatory

That equation surely doesn't leave you with much...

[friki67]

Ok, thank you anyway.

Now I have a clearer vision of how thinks work. Thanks for sharing you knowledge.

What I am going to do is rclone sync and then kopia in my backup server. It is one step more to backup and recover, but I think it could work fine.

[LewisSpring]

Hi all,

I have the same issue and I can't quite get my head around how to fix this issue (other than not using known_hosts)

My error is:

Connect Error: INTERNAL: internal server error: unable to open SFTP storage: error establishing connecting: unable to dial [nas:22]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"backup", Auth:[]ssh.AuthMethod{(ssh.passwordCallback)(0x19c58a0)}, HostKeyCallback:(ssh.HostKeyCallback)(0x19c1d60), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch

I've tried using the openSSH client to connect to the NAS, therefore adding the ed25519 key to known_hosts, and I also tried using the ssh-keyscan -p 22 nas command, which gave me a list of keys, including RSA, but these did not work in the known_hosts textbox either.

Am I missing something, or is there no way around this issue?

Cheers!

[ingul-buz]

Are you sure you copied the whole key? The other keys (especially an rsa one) are much longer than the ed25519 key, and thus the key line is usually wrapped across several lines in a command window. In the known_hosts file each key should be on one long line though.

Maybe it would be easier if you redirected the command output directly to the end of your known_hosts file: ssh-keyscan your_host >> C:\Users\username\.ssh\known_hosts in order to avoid errors in copying. Or if you want to use the known_hosts data textbox instead, redirect the command output to a temporary file and copy the key you want from there.

[LewisSpring]

Maybe it would be easier if you redirected the command output directly to the end of your known_hosts file: ssh-keyscan your_host >> C:\Users\username\.ssh\known_hosts in order to avoid errors in copying

Yes! That was it. That worked perfectly. I knew it would be something like that.
Thank you, @ingul-buz!

[friki67]

Hello again!

So, the understanding of the matter, thanks to @ingul-buz, made me find a solution:

1. Now I am using another SFTP server, using docker, https://github.com/drakkan/sftpgo. It has a web management interface, can do WebDAV,FTP... and a lot of great stuff.
2. Then I did: ssh-keyscan -p 20021 my_host >> C:\Users\username.ssh\known_hosts. This adds ssh-rsa,ecda-sha2-nistp256 and ssh-ed25519 keys to the file.
3. Configure repository in KopiaUI pointing to C:\Users\username.ssh\known_hosts, using my port, 20021

And now it is working.

Thank you!

[ingul-buz]

I'm really glad you both managed to get it working!

Unfortunately the issues I mentioned a few comments back are still there. It shouldn't be necessary to move to another SFTP host just because Kopia doesn't support ed25519 keys. Or because the external ssh option doesn't support non-standard ports. Or, for Pete's sake, because Kopia keeps demanding the known_hosts data even when the docs state it's optional.

Let's hope that one day someone will find the time and inspiration to fix these shortcomings of this great tool. Sadly I still don't have the skills to do it myself.

[andriej]

I can confirm - kopia does not handle ed25519 key.
Luckily my host also supports rsa key and thus I could get it to work.

[ntolia]

This was reported as answered on Kopia Slack. I am closing this out but please feel free to reopen if needed.

[ingul-buz]

I don't use Slack so I don't know what's been answered over there. But the matter of fact is that ssh in Kopia 0.13.0 still doesn't support ed25519 keys. I myself don't need it, since my host has other key options as well, but I think there are others without that luxury.

Just for the heck of it I tried and left only an ed25519 key in my known_hosts file, and this is what I got:

ERROR failed to open repository: cannot open storage: unable to open SFTP storage: error establishing connecting: unable to dial [myhost.com:22]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"ingul-buz", Auth:[]ssh.AuthMethod{(ssh.publicKeyCallback)(0x1f532c0)}, HostKeyCallback:(ssh.HostKeyCallback)(0x1f4f960), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch

I cannot re-open this issue, so @friki67 or @ntolia, feel free to do that if you feel that way...

[ntolia]

@ingul-buz Thank you for your response. Reopening this.

[lupusA]

@ntolia If we open the issue again, can we at least change the title. The defect is that Kopia does not support ed25519 keys. Otherwise, the defect will get lost.

[bernes1]

Any updates on this isssue?

[digtail]

If the known hosts data/file only contains the server's public rsa key and not the ed25519 key, kopia will connect to the sftp server using an ed25519 key (as confirmed by the server's auth log).

[TheManchineel]

This is still a problem on my end. Kopia isn't connecting to my SFTPgo server because of the ed25519 key. Any way to bypass the check?

[Fernando-Checa]

same here.. conecting to truenas

`root@fernandocheca:~/# kopia repository create sftp --path="vps" --host truenas --username=user --known-hosts ~/.ssh/known_hosts --keyfile ~/.ssh/id_rsa

ERROR can't connect to storage: unable to open SFTP storage: error establishing connecting: unable to dial [truenas:22]: &ssh.ClientConfig{Config:ssh.Config{Rand:io.Reader(nil), RekeyThreshold:0x0, KeyExchanges:[]string(nil), Ciphers:[]string(nil), MACs:[]string(nil)}, User:"user", Auth:[]ssh.AuthMethod{(ssh.publicKeyCallback)(0x11e1060)}, HostKeyCallback:(ssh.HostKeyCallback)(0x11dd6e0), BannerCallback:(ssh.BannerCallback)(nil), ClientVersion:"", HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch`

[digtail]

what's even failing here? I'm pretty sure the used libraries support ed25519 (please call me out if I'm wrong), and I don't see anything that disables ed25519 in the kopia source. is this a kopia thing or is this a x/crypto/ssh thing?

[Kemystra]

Hello, new Kopia user here! I also have an SFTP server with ED25519 keys setup. I did a bit of investigation and found a clue.

From the original error, you can see that HostKeyAlgorithms has an empty array:

...HostKeyAlgorithms:[]string(nil), Timeout:0}: ssh: handshake failed: knownhosts: key mismatch

I got an idea to pass the ED25519 algorithm value as defined in the so that it knows what it is dealing with. The function responsible returning the SSH config is the createSSHConfig() function in the file repo/blob/sftp/sftp_storage.go, so I hardcoded the algorithm:

// func createSSHConfig

...

        algorithms := []string{ssh.KeyAlgoED25519} // <- Added constant to slice

	return &ssh.ClientConfig{
		User:            opt.Username,
		Auth:            auth,
		HostKeyCallback: hostKeyCallback,
                HostKeyAlgorithms: algorithms, // <- Added to the parameter
	}, nil
}

I executed this command:

go/bin/kopia repository create sftp --path="backup" --host="laptop-1" --username="sftp_user" --known-hosts=".ssh/known_hosts" --keyfile ".ssh/ameise-1_no_passwd.sec" --log-level=debug

And it prompted me for password (finally! it works):

DEBUG establishing new connection...
DEBUG using internal SSH client
Enter password to create new repository:

It's weird, maybe it was not able to infer the default algorithm to deal with the keys?
Here's the documentation part from ssh.ClientConfig:

// HostKeyAlgorithms lists the public key algorithms that the client will
// accept from the server for host key authentication, in order of
// preference. If empty, a reasonable default is used. Any
// string returned from a PublicKey.Type method may be used, or
// any of the CertAlgo and KeyAlgo constants.
HostKeyAlgorithms [][string](https://pkg.go.dev/builtin#string)

Update:
I suddenly got the stable release (0.15.0) to work without this patch. After a bit of checking, I realized that this bug is now triggered if there is other type of keys than ED21559 inside the host. Maybe it is a case of wrong default algorithm.

[Kemystra]

Update: Apparently crypto/ssh package defaulted to the first key that it found on the host server (see this issue). Someone made a pull request on March 23 to fix this problem, but it has yet to be merged.

In that pull requests, one of the developers posted a trickery that he used to make a thin wrapper for the original crypto/ssh/knownhosts. The wrapper also have the functionality of loose host checking, which would ask the user when connecting to a host for the first time.

[ados8]

Using ssh-keyscan -p 22 IP >> known_hosts was the only thing that worked for me.
Was using Synology NAS as destination and it using the Windows method of ssh username@IP just resulted in ssh-ed25519 for the key which Kopia wouldn't accept.