HPC cluster host for SSH certificate/SSO authentication, possible to setup as a user and not admin?

[mshamash]

Hi,

I’m a new user to Smallstep and all the associated cli tools and am looking into the best way to implement step SSH certificates with SSO amongst the many hosts I connect to regularly.

One of these hosts is particularly unique, however, as it is a high performance computing (HPC) cluster where I am only one of hundreds (or thousands?!) of users who connects regularly via SSH. Ultimately, this means that I don’t have permission to install custom applications or scripts, with rare exceptions, and definitely cannot modify global SSH configs in /etc/ and other “core Linux” directories.
I was looking at the guide Smallstep published on manual host setup (https://smallstep.com/docs/ssh/hosts-step-by-step) and it looks like I ultimately need be sysadmin to enable step integration on a host, is this correct? I was wondering if there were any alternative solutions in this situation, as I’d love to use my newly-built “Tiny CA” in my homelab for SSH certificates everywhere, including the shared HPC cluster (thanks @tashian for the awesome guide you wrote up!).
I feel I may have to just end up going the boring “old-fashioned” route and use an OpenSSL certificate I generate locally for this host in particular...though I’d love to be proven wrong!

Looking forward to hearing your thoughts on this, and best wishes for the New Year to the Smallstep team!

Cheers,

Michael

[maraino]

Hi @mshamash, the docs you link are part of our SaaS offering https://smallstep.com/sso-ssh/, it allows you to link your identity provider (IdP), and give access to the users in your IdP to your servers with some policy.

To be able to use certificates you need to change /etc/ssh/sshd_config, you need three parameters, for example:

TrustedUserCAKeys /etc/ssh/ca.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
HostKey /etc/ssh/ssh_host_ed25519_key

TrustedUserCAKeys defines the file containing the public part of the key used to sign the user certificates.

HostCertificate is the SSH certificate for that host, users will a @cert-authority entry in their known_hosts with the public part of the key used to sign host certificates.

HostKey is the private key used by SSH, there are some defaults, and you can omit it if you use one of the defaults.

Our open-source CA allows you to create the host, and user certificates, the easiest way to do it is to bootstrap your CA using

step ca init --ssh

That will command will create root and intermediate certificates for TLS as well as user and host key for signing SSH certificates. It will also create some templates to automatically configure users using step ssh config and hosts using step ssh config --host. You can create host certificates for an existing key using:

# step ssh certificate -f --sign --host <hostname> <path-to-ssh-public-key>
step ssh certificate -f --sign --host tynyca.local /etc/ssh/ssh_host_ed25519_key.pub

Once you configure your hosts you will need an automatic way to refresh the host certificates, by default they expire after 30 days, you can do it with a cron or a systemd timer. Here are some docs, but they are for X.509 (TLS) certificates instead of SSH ones. At the end of this script, there's an example to configure crond for this.

The default certificate for users is 16 hours, the best way to refresh them is configuring an IdP and using OAuth/OIDC although you can also use the default JWK provisioner created in the bootstrap. The configuration files created using step ssh config can detect if you need a certificate and redirect you to the authentication flow.

The other problem to solve is the user accounts in the hosts, if you already have one, then there is no problem, you should be able to login with your certificate, if not, then you need a way to provision it, that's where our SaaS comes handy. Our open-source version comes with a simple way to do it. When you create your user certificate you need to pass the --add-user flag, (e.g. step ssh login --add-user [email protected]). This automatically creates another certificate for a user provisioner that will run the command sudo useradd -m <principal>; nc -q0 localhost 22 in the server, create the new user, and proxy the connection to you. This is configurable, and of course, the provisioner user must be configured in your host.

In this blog post, you can find some helpful instructions for some of the above, as well as a handy script to bootstrap a server in AWS, but you should be able to adapt to your needs.
https://smallstep.com/blog/diy-single-sign-on-for-ssh/

[mshamash]

Hi @maraino and thanks for the detailed reply. Since editing /etc/ssh/sshd_config is a must for SSH certificates to function properly, it looks like I won't be able to use it on the majority of servers where I'm currently a user (one of many) and not an admin with authority to edit global SSH configs.

Hopefully certificate authentication for SSH becomes more widespread and adopted by our compute cluster provider!

[maraino]

At this point, I'm wondering what the SSH Host key is for then, since it seems like the user key is what's being used throughout authentication? My understanding is the SSH Host key is what signs my 18-hour certificate on my local machine once I pass SSO?

The host key is used to sign host certificates, you don't need it if you are not going to use certificates in hosts. Using certificates on hosts allows you to have just one @cert-authority entry in your known_host and trust automatically hosts with certificates signed by the host key.

Next I'll be looking into ways to automate the step ssh login michael@mydomain .... step so I don't have to run it every day, but can have it run instead upon my first ssh user@hostname command of the day.

There is a way to do this, if you look at our default config template:

certificates/templates/values.go

Lines 81 to 91 in 00c6f08

	"config.tpl": `Match exec "step ssh check-host %h"
	{{- if .User.User }}
	User {{.User.User}}
	{{- end }}
	{{- if or .User.GOOS "none" | eq "windows" }}
	UserKnownHostsFile "{{.User.StepPath}}\ssh\known_hosts"
	ProxyCommand C:\Windows\System32\cmd.exe /c step ssh proxycommand %r %h %p
	{{- else }}
	UserKnownHostsFile "{{.User.StepPath}}/ssh/known_hosts"
	ProxyCommand step ssh proxycommand %r %h %p
	{{- end }}

The step ssh proxycommand can detect if you need a new certificate, looking at the keys in the ssh agent, and do the OIDC flow for you. With more details:

# ssh client rule to match the hosts, you can also use things like `Host *`, `Host 192.168.*`, see `man ssh_client`:
Match exec "step ssh check-host %h"
        # Default username to use, by default ssh use the system username
	User mariano
	# Path the the known_host where @cert-authority leaves (only needed if you use host certificates)
	UserKnownHostsFile "/path/to/ssh/known_hosts"
	# Get the bastion configuration from the ca, and uses `Google` provisioner to start an automatic `step ssh login`, and then proxies the connection to your client.
	ProxyCommand step ssh proxycommand --provisioner "Google" %r %h %p

[mshamash]

I see, I've changed the Match exec "step ssh check-host %h" line to Host * so it'll trigger SSO/cert creation for any host I connect to, and am just realizing that the step ssh proxycommand requires that the remote username be equal to one of the Principals on my certificate (so michael or [email protected]). Is there a way around this with --set=<key=value> perhaps?

[maraino]

The best work around is using certificate templates.

[maraino]

Although there's another one using ssh's AuthorizedPrincipalsCommand or AuthorizedPrincipalsFile and use the email address in your certificate

[mshamash]

I've added the following block to my ca.json file so that I can edit the default SSH cert template (from here)

"options": {
    "ssh": {
        "templateFile": "/etc/step-ca/templates/ssh/cert.tpl"
    }
}

As of now my cert.tpl file looks like this (replicating the default file):

{
        "type": "{{ .Type }}",
        "keyId": "{{ .KeyID }}",
        "principals": ["[email protected]"],
        "extensions": {{ toJson .Extensions }},
        "criticalOptions": {{ toJson .CriticalOptions }}
}

How can I set the principals portion to include the .Principals obtained from OIDC (so michael and [email protected]) and some custom defined ones (ie. other usernames on servers I connect to, like ubuntu, mike, etc.)? I'm not familiar with this format of the cert.tpl file. Thanks!

EDIT: I've been playing around some more, and even if I replace the "principals": {{ toJson .Principals }}, line with "principals": ["ubuntu"], I still get a step-ca server error of authority.SignSSH: ssh certificate principals does not match - got [ubuntu], want [michael [email protected]]. This is with Host * set in my step client SSH config, and me connecting to a host with ssh ubuntu@serverIP

EDIT2 (Jan 10): I added myself as an admin in the ca.json file for my Google OIDC provider by adding the "admins": ["[email protected]"] line. Now I'm able to issue SSH certs for any host/Principal pair I connect to. The only thing I don't like is how I'm forcing the [email protected] for the principals on the cert (see my cert.tpl above). Without it, then the only principal on the SSH cert if the username of the host I connect to, which means my SSH cert will ONLY work on that host. Some hosts have different usernames, so this isn't feasible, that's why I'm forcing a principal of my email address.
Is there any way to set the cert.tpl file Principal to be ONLY my email address as provided by the OIDC provider, instead of hard-coding it in?

Final edit!: Turns out changing the principals line in the template to "principals": ["{{ .Token.email }}"] did the trick! Thanks again for all your help over the past week!
Cheers. Michael