Struggling to Use SSH Certificates #527

J-Hunter-Hawke · 2021-04-06T17:11:47Z

J-Hunter-Hawke
Apr 6, 2021

Hello,

We've been using our Smallstep CA to reliably issue client certificates to our edge devices, and that project has become invaluable to everyone involved. Now that we're setting up processes to use SSH Certificates, I'm hitting an issue where the SSH session (preauth) on the user side errors out because of a "key_cert_check_authority" issue and then fails to recognize the CA signing key for the host SSH Certificates. I initially followed this guide to set everything up: https://github.com/smallstep/step-ssh-example. Then to troubleshoot, I've tried cross-referencing a couple guides online that more or less reword the step-ssh-example.

I'm about out of ideas to try and troubleshoot these issues. If anyone is willing, could you please review my following implementation and let me know if you see any issues that stick out or if you'd run into these problems before?

Thank you,
Hunter

Hosts Setup:

# Bootstrap the Smallstep config from the Cert Authority.
# This will store the root certificate and some configuration files under the ~/.step directory (/root/.step if running as root). 
step ca bootstrap --fingerprint 9a4...1c4a --ca-url "certificates.domain.com"

# Save the CA's public user SSH key
# We will trust any SSH key signed by the corresponding user private key
echo "@cert-authority * ecdsa-sha2-nistp256 AAAAE...iCfiO/tA5909U=" > $(step path)/certs/ssh_user_key.pub

# Contact the cert authority and generate a short-lived device certificate.
# The Common Name of the certificate will match the host name of the device.
step ssh certificate $HOSTNAME /path/to/ssh_host_ecdsa_key --host --no-password --insecure --force --password-file=$temp_file_path --provisioner "name"

sshd_config:

LogLevel VERBOSE

# Allow root for testing purposes; change to 'no' before going to production.
PermitRootLogin yes

# Only accept PubkeyAuthentication
PubkeyAuthentication yes
HostbasedAuthentication no
IgnoreUserKnownHosts yes
PasswordAuthentication no
ChallengeResponseAuthentication no

# SSH CA Configuration
# The path to the CA public key for authenticating user certificates
TrustedUserCAKeys /root/.step/certs/ssh_user_key.pub # Assumes step ca bootstrap was run as root, UPDATE IF NOT THE CASE!
# Path to the private key and certificate
HostKey /path/to/ssh_host_ecdsa_key
HostCertificate /path/to/ssh_host_ecdsa_key-cert.pub

And then, we restart ssh on the host device.

Inspecting the certificate with step ssh inspect shows a valid certificate, signed by the host CA key, with principal $HOSTNAME.

Users Setup:

# Save the CA's public HOST ssh key
# We will trust any ssh key signed by the corresponding HOST private key
echo "@cert-authority * ecdsa-sha2-nistp256 AAAAE2VH...CMGc5oxF3zQ=" > /etc/ssh/ssh_known_hosts

# Issue a user SSH Certificate
step ssh certificate [email protected] id_ecdsa --principal "root" --no-password --insecure --force

Inspecting the certificate with step ssh inspect shows a valid certificate, signed by the user CA key, with principal 'root.'

id_ecdsa-cert.pub:
        Type: [email protected] user certificate
        Public key: ECDSA-CERT SHA256:5316howqhhKcH0GMDDpvTwTxc1yLhFflY9MRZZzGn2k
        Signing CA: ECDSA SHA256:mXTvR+WRp1lUs1TLmNikYJRVT4lGJVNhFhGzIbu5bn0
        Key ID: "[email protected]"
        Serial: 5765201607071121458
        Valid: from 2021-04-06T16:50:44 to 2021-04-07T08:51:44
        Principals:
                root
        Critical Options: (none)
        Extensions:
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc
                permit-X11-forwarding

SSHing to the host device with ssh [email protected] -i id_ecdsa -v returns:

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.16.0.5 [172.16.0.5] port 22.
debug1: Connection established.
debug1: identity file id_ecdsa type 2
debug1: identity file id_ecdsa-cert type 6
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.3
debug1: match: OpenSSH_8.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.16.0.5:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: [email protected]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: [email protected] SHA256:qhYY72AlxDL9j1hQuocPGeIutEIOJ2FJboy9TZcM/gI, serial 955..23 ID "FR06B11H" CA ecdsa-sha2-nistp256 SHA256:andIe25F58YVnmePe3S7dbGWgPPkoJi9bbxMz6svSPg valid from 2021-04-05T12:51:06 to 2021-05-05T12:52:06
debug1: Host '172.16.0.5' is known and matches the ECDSA-CERT host certificate.
debug1: Found CA key in /etc/ssh/ssh_known_hosts:1
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
debug1: No matching CA found. Retry with plain key
The authenticity of host '172.16.0.5 (172.16.0.5)' can't be established.
ECDSA key fingerprint is SHA256:qhYY72AlxDL9j1hQuocPGeIutEIOJ2FJboy9TZcM/gI.
Are you sure you want to continue connecting (yes/no)?

And responding 'yes' to the TOFU warning continues with:

Warning: Permanently added '172.16.0.5' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@o
penssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: ECDSA SHA256:5316howqhhKcH0GMDDpvTwTxc1yLhFflY9MRZZzGn2k id_ecdsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: ECDSA-CERT SHA256:5316howqhhKcH0GMDDpvTwTxc1yLhFflY9MRZZzGn2k id_ecdsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey,keyboard-interactive).

Then seeing the line "Certificate invalid: name is not a listed principal," I tried adding a line to /etc/hosts with the host info:
172.16.0.5 FR06B11H

SSHing to the hostname with ssh root@FR06B11H -i id_ecdsa -v returns:

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to fr06b11h[172.16.0.5] port 22.
debug1: Connection established.
debug1: identity file id_ecdsa type 2
debug1: identity file id_ecdsa-cert type 6
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.3
debug1: match: OpenSSH_8.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to fr06b11h:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: [email protected]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: [email protected] SHA256:qhYY72AlxDL9j1hQuocPGeIutEIOJ2FJboy9TZcM/gI, serial 955...823 ID "FR06B11H" CA ecdsa-sha2-nistp256 SHA256:andIe25F58YVnmePe3S7dbGWgPPkoJi9bbxMz6svSPg valid from 2021-04-05T12:51:06 to 2021-05-05T12:52:06
debug1: Host 'fr06b11h' is known and matches the ECDSA-CERT host certificate.
debug1: Found CA key in /etc/ssh/ssh_known_hosts:1
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
debug1: No matching CA found. Retry with plain key
The authenticity of host 'fr06b11h(172.16.0.5)' can't be established.
ECDSA key fingerprint is SHA256:qhYY72AlxDL9j1hQuocPGeIutEIOJ2FJboy9TZcM/gI.
Are you sure you want to continue connecting (yes/no)?

And responding 'yes' to the TOFU warning continues with:

Warning: Permanently added 'fr06b11h,172.16.0.5' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@o
penssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: ECDSA SHA256:5316howqhhKcH0GMDDpvTwTxc1yLhFflY9MRZZzGn2k id_ecdsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: ECDSA-CERT SHA256:5316howqhhKcH0GMDDpvTwTxc1yLhFflY9MRZZzGn2k id_ecdsa
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: No more authentication methods to try.
root@fr06b11h: Permission denied (publickey,keyboard-interactive).

Answered by tashian

Apr 6, 2021

Try taking the @cert-authority * out of your /root/.step/certs/ssh_user_key.pub.
The @cert-authority * piece is really only for authorized_keys / known_hosts.

View full answer

tashian · 2021-04-06T17:50:41Z

tashian
Apr 6, 2021
Collaborator

Try taking the @cert-authority * out of your /root/.step/certs/ssh_user_key.pub.
The @cert-authority * piece is really only for authorized_keys / known_hosts.

11 replies

tashian Apr 7, 2021
Collaborator

Interesting! I haven't seen this behavior before — SSH requiring both hostname and IP on the host cert. I've seen plenty of host certs that only have hostnames on them. My hunch is that your case is different because your hostname is not an FQDN. And so, because there is room for ambiguity in a bare hostname, SSH also requires the IP on the cert.

Either that, or you have a less common setting somewhere that's triggering this behavior.

I'd be curious if you still saw the behavior with an FQDN on the cert—if that's feasible for you.

J-Hunter-Hawke Apr 9, 2021
Author

Hi Carl; I've been playing around with this today, and it looks like there's still something similar going on. I wonder if the issue lies in the use of private IP addresses being assigned. Relevant info is as follows (note, this is a different server but same config):

nslookup from the OpenVPN server:

$ nslookup D006B007.subdomain.domain.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   D006B007.subdomain.domain.com
Address: 172.16.0.7

User SSH Cert:

$ step ssh inspect id_ecdsa-cert.pub
id_ecdsa-cert.pub:
        Type: [email protected] user certificate
        Public key: ECDSA-CERT SHA256:cDpju+1LuF1C5x3TQCPE+VHU43OvcgjS2JT9e+2D4G4
        Signing CA: ECDSA SHA256:mXTvR+WRp1lUs1TLmNikYJRVT4lGJVNhFhGzIbu5bn0
        Key ID: "[email protected]"
        Serial: 9791749714461517651
        Valid: from 2021-04-09T13:11:23 to 2021-04-10T05:12:23
        Principals:
                root
        Critical Options: (none)
        Extensions:
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc
                permit-X11-forwarding

SSH Host Cert:

D006B007.subdomain.domain.com# step ssh inspect ssh_host_ecdsa_key-cert.pub
ssh_host_ecdsa_key-cert.pub:
        Type: [email protected] host certificate
        Public key: ECDSA-CERT SHA256:8+NK4rhxC7W8enLD071u0dIknJyGB1MaJu4X1GP4CUw
        Signing CA: ECDSA SHA256:andIe25F58YVnmePe3S7dbGWgPPkoJi9bbxMz6svSPg
        Key ID: "D006B007.subdomain.domain.com"
        Serial: 3638639928653824028
        Valid: from 2021-04-09T18:33:04 to 2021-05-09T18:34:04
        Principals:
                D006B007
                D006B007.subdomain.domain.com
        Critical Options: (none)
        Extensions: (none)

SSH attempt:

$ ssh [email protected] -i id_ecdsa -v

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to d006b007.subdomain.domain.com [172.16.0.7] port 22.
debug1: Connection established.
debug1: identity file id_ecdsa type 2
debug1: identity file id_ecdsa-cert type 6
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.3
debug1: match: OpenSSH_8.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to d006b007.subdomain.domain.com:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: [email protected]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: [email protected] SHA256:8+NK4rhxC7W8enLD071u0dIknJyGB1MaJu4X1GP4CUw, serial 3638639928653824028 ID "D006B007.subdomain.domain.com" CA ecdsa-sha2-nistp256 SHA256:andIe25F58YVnmePe3S7dbGWgPPkoJi9bbxMz6svSPg valid from 2021-04-09T18:33:04 to 2021-05-09T18:34:04
debug1: Host 'd006b007.subdomain.domain.com' is known and matches the ECDSA-CERT host certificate.
debug1: Found CA key in /etc/ssh/ssh_known_hosts:1
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
debug1: No matching CA found. Retry with plain key
The authenticity of host 'd006b007.subdomain.domain.com (172.16.0.7)' can't be established.
ECDSA key fingerprint is SHA256:8+NK4rhxC7W8enLD071u0dIknJyGB1MaJu4X1GP4CUw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'd006b007.subdomain.domain.com,172.16.0.7' (ECDSA) to the list of known hosts.
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected]>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: ECDSA SHA256:cDpju+1LuF1C5x3TQCPE+VHU43OvcgjS2JT9e+2D4G4 id_ecdsa
debug1: Authentications that can continue: publickey
debug1: Offering public key: ECDSA-CERT SHA256:cDpju+1LuF1C5x3TQCPE+VHU43OvcgjS2JT9e+2D4G4 id_ecdsa
debug1: Server accepts key: pkalg [email protected] blen 578
debug1: Authentication succeeded (publickey).
Authenticated to d006b007.subdomain.domain.com ([172.16.0.7]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Remote: cert: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: cert: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
D006B007.subdomain.domain.com#

For now, this isn't that big of an issue since setting the IP address as a principal solves the problem. If needed, I'll set up a cron job on each host device that does an ifconfig and compares the result with the principal in the SSH Certificate to make sure they stay consistent. (They should, but you just never know for certain.)

J-Hunter-Hawke Apr 9, 2021
Author

Correction... I just thought of trying this after hitting "Reply" on my previous response...

Setting the principal to "d006b007.subdomain.domain.com" did the trick. In my previous response, I had it set to "D006B007.subdomain.domain.com."

Looks like I was once again bitten by case-sensitivity in a Unix environment! Thank you for all the help; we really appreciate it!

tashian Apr 14, 2021
Collaborator

I'm glad you figured it out. This seems like a subtle bug in openssh, doesn't it? DNS names aren't case sensitive in other contexts, why should they be when they are certificate principals?

LecrisUT Apr 14, 2021

According to the manpage of ssh_config, the arguments are case sensitive. So it would distinguish between case-sensitive Host configurations.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Struggling to Use SSH Certificates #527

{{title}}

Replies: 1 comment 11 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Struggling to Use SSH Certificates #527

J-Hunter-Hawke Apr 6, 2021

Replies: 1 comment · 11 replies

tashian Apr 6, 2021 Collaborator

tashian Apr 7, 2021 Collaborator

J-Hunter-Hawke Apr 9, 2021 Author

J-Hunter-Hawke Apr 9, 2021 Author

tashian Apr 14, 2021 Collaborator

LecrisUT Apr 14, 2021

J-Hunter-Hawke
Apr 6, 2021

Replies: 1 comment 11 replies

tashian
Apr 6, 2021
Collaborator

tashian Apr 7, 2021
Collaborator

J-Hunter-Hawke Apr 9, 2021
Author

J-Hunter-Hawke Apr 9, 2021
Author

tashian Apr 14, 2021
Collaborator