💾 Archived View for colincogle.name › blog › do-it-better › ssh captured on 2024-03-21 at 14:49:16. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-11-14)
-=-=-=-=-=-=-
"Do It Better: SSH" by Colin Cogle
Published November 7, 2023.
SSH is easy to secure, but that doesn't mean there isn't more work to be done.
Let's talk about SSH. It is the quintessential Swiss army knife for system administrators. It can browse and transfer remote files, forward ports, proxy and tunnel traffic, but it's most famous for its eponymous job of giving you a remote shell, securely (hence its name). As processors have gotten faster and cheaper, and encryption has become almost frictionless, it's long since replaced Telnet for configuring embedded devices. In fact, even Microsoft acquiesced and made SSH a first-class citizen starting with Windows Server 2019.
You'd think with SSH everywhere, it would be easy to configure, right? Well, you are right. In fact, the default configuration of OpenSSH is pretty secure, but I guarantee you're still not implementing it as well as you could.
I'm going to assume that your operating system either includes or is supported by a recent version of OpenSSH. We're going to configure things to support only the most modern features, so if you're trying to connect with an ancient version of macOS or PuTTY for Windows Vista, you might get shown the cold shoulder. We're dealing with security. Upgrade your software.
Every major desktop operating system — even Windows — now includes the OpenSSH client (the part you use to connect to other SSH servers). Open up a terminal, PowerShell window, Command Prompt, or what-have-you and try typing in `ssh`. I bet something will happen.
As for the OpenSSH server (the thing you log into), how you get that depends on your operating system. For Windows 10, Windows Server 2019, and newer, the OpenSSH client is usually installed by default, while the OpenSSH server exists as a Feature On Demand. It's built into macOS but dormant until you go into Sharing Preferences and enable Remote Login. Meanwhile, on Linux, you should try installing `openssh-server` and see what happens.
If you're using a network device or some other embedded device, check and see what's available. Enterprise-oriented networking equipment often includes an SSH server. Embedded devices typically use a smaller SSH daemon like Busybox or Dropbear, neither of which I'm going to cover in detail in this article. This article is focused more on permanent installations, such as on a remote server.
Once you've installed an SSH server, you will have to edit text files to get it configured just the way we like it. UNIX and UNIX-like systems will install the configuration files into `/etc/ssh`. (Windows users, it's somewhere under `C:\ProgramData`.)
You could use a reference configuration like Mozilla's "Modern" OpenSSH setup,
or work with me while I create something newer. Open up the server's main configuration file, `sshd_config`. Most of the work we'll be doing will take place in this single file. Leave it open and keep reading.
Let me state the obvious. Just because SSH is secure, why risk it? You are enabling and exposing a remote access method. That's the whole point, but it bears repeating. Consider how much risk you're willing to take, and don't push a half-assed solution into production.
The first thing to consider is where you will be accessing this remote server *from*. Just because we *can* open port 22 to the whole world doesn't mean that we should. Ask yourself:
Well, this makes it easy. Don't expose SSH to the public Internet. Do not add a firewall rule. Do not create a NAT policy. Do not pass Go and do not collect $200. Look, you're already done with this step.
This is the other extreme. Maybe you travel a lot. Maybe you're often on random Wi-Fi networks. Maybe your ISP only offers dynamic IP addresses. Or, you might be lazy, and that's fine, too.
In this case, just configure your firewall to allow SSH connections from anywhere. For example, in this made-up firewall pseudocode, we might make the following changes:
add firewall rule "SSH Server" allow port 22/tcp from anywhere to "MyServer" add firewall nat-policy "SSH Server IPv4" source (any to original) destination (WAN1 to "MyServer") port (22 to original) save
Every network should have some kind of firewall or network control device. Maybe it's a fancy SonicWall or Cisco device. Perhaps it's a simple Azure or AWS security group. Perhaps it's just the modem/router hybrid your ISP gave you. This part is left as an exercise to the reader. (I believe in you.)
This scenario is often used in businesses, and mine is no exception. My company policy is to make remote management interfaces only available to a select few IPv4 addresses and IPv6 subnets. This does mean that remote employees (myself included) need to use a VPN or a remote desktop, but that's a small price to pay for a surprisingly-effective security implementation. As I once explained this, a hacker can't knock on the door if they can't get to it in the first place.
In this scenario, you will expose your SSH server to the Internet, setting up the necessary firewall rules (and IPv4 NAT policies, if you need that older protocol). However, in this case, configure your firewall rules so that access is only allowed from certain IP addresses or subnets. For example:
new address-group "Trusted IP Addresses" edit address-group "Trusted IP Addresses" add address 192.0.2.1/32 edit address-group "Trusted IP Addresses" add address 2001:db8:1234::/48 add firewall rule "SSH Server" allow port 22/tcp from "Trusted IP Addresses" to "MyServer" # for "legacy IP" add nat-policy "SSH Server IPv4" source (any to original) destination (WAN1 to "MyServer") port (22 to original) save
If your remote-side ISP only provides you dynamic IP addresses, maybe you can still configure this scenario. Does your remote firewall offer some kind of VPN or VPN-like feature? If so, set that up, then configure your firewall to allow SSH connections from VPN clients instead of the entire Internet.
Some people will tell you that you should never run SSH on the default port of TCP 22. They'll advise you to pick another number. While that may defer script kiddies and automated scans, it's trivial to find an SSH service. For example, I could just scan all of your ports in the span of a few minutes:
nmap yourserver.yourdomain.com -p 1-65535 -A --open
Or, I can just type your IPv4 address into Shodan and get a report instantly.
In either case, I found that your SSH server is running on port 42069, let's say. I can also see what version of OpenSSH it uses, and begin to see how it might be configured. Don't waste your time with security theater and just leave it on port 22.
SSH relies on public key cryptography to authenticate servers and users. There are four major key types still in use today:
My recommendation is to use EdDSA for everything, falling back to ECDSA or RSA if needed. However, since we're only targeting the newest versions of OpenSSH, let's forget that anything besides Curve25519 exists.
When you start SSH for the first time, you'll notice that it generates many different kinds of host keys: RSA, ECDSA, and Ed25519 (and DSA if you have an older version). There's no need to use anything but the newest, so let's disable and delete the others.
Stop your SSH server, and edit your `sshd_config` file. Comment out all of the other `HostKey` lines.
-HostKey /etc/ssh/ssh_host_rsa_key -HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key +#HostKey /etc/ssh/ssh_host_rsa_key +#HostKey /etc/ssh/ssh_host_ecdsa_key
Now, delete all other host keys and restart OpenSSH:
sudo rm /etc/ssh/ssh_host_*sa_key* sudo systemctl restart sshd
Now, only clients that support Ed25519 host keys can even attempt to log in. This step by itself will break many old IP scanners, botnets, and script kiddies' tools. (I look through my logs sometimes and see a lot of connections failing because `ssh-rsa` is unsupported.)
Now that we have only the best host key type remaining, let's take a look at the next stage of the SSH connection, when the client and server exchange keys and prepare to set up an encrypted session. Just like with host keys, there are a lot of key exchanges supported. We should only support the most secure ones.
And, just like with host keys, older versions of OpenSSH may not be able to connect. But if all our computers are using, let's say, OpenSSH 8, why bother supporting older versions?
You can run the `ssh -Q kex` command to see all key exchanges supported by your version of OpenSSH, but there are three that we'll be using:
(Complaints to @colincogle@mastodon.social)
We're going to leave these off the list for varying reasons:
Go ahead and put the algorithms you want into the `sshd_config` file, like so:
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256@openssh.com,curve25519-sha256
With all of these multiple-choice directives, an SSH client and server will go down the list, from left to right, and use the first thing that they mutually agree on. Always arrange them from most secure to least secure, and omit ones you don't want to support.
Though we've disabled all Diffie-Hellman key exchanges in this topic, you may need to re-enable them if you need to support an older client. Even if you don't, it's still good practice, just in case they get re-enabled for whatever reason. There is a file, `/etc/ssh/moduli`, which needs to be edited. While it can be edited by hand, the fine folks at Mozilla have given us a Bash one-liner:
sudo sh -c "awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.tmp && mv /etc/ssh/moduli.tmp /etc/ssh/moduli"
(Source: "Mozilla Security Guidelines: OpenSSH")
Now, any DH key exchanges will use 3072-bit RSA at a minimum.
Now that both sides can exchange keys securely, we need to look at ciphers. We have far fewer options here, but that's fine because what's offered is quite secure. You can use `ssh -Q cipher` to see what's supported.
Do not use any of these:
Let's add this line to our `sshd_config` file:
Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com
While both provide security that won't be broken anytime soon, I would recommend using AES-256 over AES-128 in most cases. However, I sometimes run graphical apps over SSH by using Waypipe (and the older X forwarding), and I have noticed AES-128 provides a significant speed boost in that case. If you plan to use Waypipe, X forwarding, or you plan to transfer a lot of files via SSH, you may want to consider listing AES-128 first.
My servers prefer AES-256, but my desktop prefers AES-128. If you're unsure, you can also set per-host settings on the client side — more on that later.
Finally, there are MACs — Message Authentication Codes — used to verify that encrypted data has not been modified, intentionally or otherwise, in transit. Again, you can use `ssh -Q MACs` to view what's supported, and we'll promptly cull this long list.
The SHA-2 family is the most secure, and there isn't much of a performance impact on modern hardware. Sort by number of bits, in descending order.
Avoid or disable anything with the following characteristics:
1. Omit anything using 64-bit or 96-bit digests, as those are too small for modern use.
2. Omit anything using broken algorithms like MD5 or SHA-1.
3. Avoid anything using weak algorithms like RIPEMD-160.
4. Avoid anything *not* using ETM (Encrypt-then-MAC mode), as the alternative of hashing before encrypting can lead to weaknesses.
That doesn't leave us with much, but it leaves us with good options. Add this line to your `sshd_config`.
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
Now, we're going to explore the other lines in the `sshd_config` file. Many of these need little explanation, so I'm going to go through this next section quickly.
SSH version 1 was deprecated back in 2006 with RFC 4251,
and all support was removed from OpenSSH 7.6 back in 2017. It's dead. Get on with your life.
If it makes you feel better, try adding this to your `sshd_config` and hope it doesn't complain about an unsupported option:
Protocol 2
Take it from Mister IT Guy over here: **don't use passwords if there's a better option**. Generating SSH keys is a quick and painless procedure. Disable password-based logins and stop credential stuffing before it occurs.
# Disable password-based logins, and only allow key-based logins. AuthenticationMethods publickey
If your server and clients are joined to a Kerberos realm or an Active Directory domain, you can look into setting up OpenSSH's native Kerberos and GSSAPI authentication mechanisms. A Kerberos ticket is arguably even more secure than a stealable key. I haven't worked with OpenSSH and Kerberos in many years, so I won't pretend to know how to configure it.
There is literally no good reason to be working as `root` on a daily basis. Anyone sane should be using `sudo` to elevate their own permissions. However, it's possible you set a root password long ago, or you may have accidentally associated a public key with the root account.
Regardless, logging in as `root` over a network is a pretty terrible idea. Unless you can literally sit down and explain why you should not make this change, you should make this change:
#PermitRootLogin prohibit-password PermitRootLogin no
X forwarding is a great way to run graphical Linux apps over an SSH connection, and have them appear on your remote screen like a native, local app.
However, many Linux distributions have switched Xorg for Wayland. If you use Wayland, you should be using Waypipe to run graphical apps over SSH, as I found it much faster. Thus, if you're using Wayland, your server is headless, or all of this just plain sounds like a dumb idea, comment out the `X11Forwarding` line in `sshd_config`, or set it to `no`:
# We don't need classic X11 forwarding to be supported. #X11Forwarding yes
You can comment out any other options that start with `X11` if you like a nice, clean configuration file; however, they won't have any effect.
If something goes wrong, you're going to want to refer to your logs, so let's make sure we generate some useful data. If you'd like to know which keys are being used to log into your server, bump up the logging level.
SyslogFacility AUTH LogLevel VERBOSE
You can also log what people do via SFTP file transfers.
Subsystem sftp /usr/lib/sftp-server -f AUTHPRIV -l INFO
After a successful login, you should show the user their previous login to make sure it matches what they expect.
PrintLastLog yes UseDNS yes
When you connect to a new SSH server for the first time, you will be asked to verify the SSH fingerprint, so that you can avoid a future man-in-the-middle attack. Most people will blindly accept it. However, this process can be automated by putting the server's host key fingerprint(s) into DNS.
Pick a public name for your server, and then feed it to the `ssh-keygen -r` command, like so:
$ ssh-keygen -r myserver.mydomain.com. myserver.mydomain.com. IN SSHFP 4 1 5919fdbf8b988eced9a8aca480708a791658aa64 myserver.mydomain.com. IN SSHFP 4 2 e166e0335b269cf0b2a0a7cb58e672bce471dafd8b7af634165eed432608acf5
If you get more than two records, then you must have forgotten to delete your unwanted SSH host keys in the previous steps. That's fine. We only want the Ed25519 host key fingerprints, which all begin with the number 4. Ignore the others.
Now, go to wherever you go to edit your DNS zone. This might be Cloudflare, a local BIND server, or your registrar.
If you're unlucky enough to be using Microsoft DNS Server, I have you covered, believe it or not.
Next, add two new SSHFP records, one for each line. If you're unluckier still and your DNS host doesn't support SSHFP records (looking at you, GoDaddy and Network Solutions), then you're out of luck.
Note that this is all for naught if you don't use DNSSEC. If your zones aren't signed, please look at a calendar, see what year it is, then ask yourself why you haven't secured your infrastructure yet. Even GoDaddy realized how important secure DNS is to the Internet ecosystem and stopped charging extra for DNSSEC.
(Even those scummy capitalists can do the right thing once in a while.)
Now, edit your *local* machine's `/etc/ssh/ssh_config` file and add the following two lines:
Host * VerifyHostKeyDNS yes
Now, when you connect to an SSH server for the first time, SSHFP records will be retrieved from DNS, and your computer will help you decide if you're connecting to the right server.
Believe it or not, this is now supported! If using SSH as a native transport for PowerShell remoting sounds like something you want, add this line, substituting the correct path for your particular system:
Subsystem powershell /usr/bin/pwsh -sshs -NoLogo
Now that we've set up OpenSSH, we're ready to expose it to the Internet. However, there is one problem we haven't addressed: we can't log in. Remember, we disabled password-based logins earlier! To make SSH usable and secure, we're going to log in with a keypair.
If you already have one you'd like to use, skip this entire section. On your computer (not the server), generate a new key. Open up a terminal and, like before, we're going to use Ed25519.
mkdir -p ~/.ssh cd ~/.ssh ssh-keygen -t ed25519 -f id_ed25519
You'll be prompted to create a password to protect your key against theft. Pick something strong, or throw your opsec to the wind and use no password at all.
Now, why did I pick the name `id_ed25519`? Keys named like `id_<algorithm>` will be tried automatically when connecting to servers. If you use a different name, you'll need to specify an `IdentityFile` manually for each remote server you connect to.
In the `.ssh` folder in your home folder, you will have two files. `id_ed25519` is your private key. That should never leave your computer (except when being securely backed up), and never fall into the hands of an attacker. This goes double if you decided not to protect your key with a password. The other file is `id_ed25519.pub`, which you can and will share with the world.
Some people have strong philosophical opinions about using one keypair per server. I disagree. I believe a key is more closely associated with a person (or a person's device) and I have no problem using the same keypair for many servers. However, I do create separate ones for personal use and work use.
Now that you have a public key, we need to get it to the server. Copying and pasting is usually the simplest way, so open up `id_ed25519.pub` in a text editor and copy that whole thing.
On the server, navigate to `~/.ssh` and edit the `authorized_keys` file. Paste that public key onto a blank line, then hit save!
Let's give it a shot!
ssh user@myserver.mydomain.com
If your local and remote usernames are the same, you can leave off the `user@`.
If you've set up SSHFP records, you should see "Verified host key found in DNS." Accept the correct fingerprint and you should be logged in!
By this point, we've set up a pretty secure SSH server, but I'm sure my astute and security-conscious readers are getting ready to send me angry toots about me teaching them to expose a remote access method to the Internet with only single-factor authentication enabled. If you're using Kerberos or only allowing certain IP addresses to connect, this may not be necessary. Plus, keys are far less likely to be stolen than passwords. Still, let me show you how to implement MFA via the classic TOTP method.
These steps only apply to Linux systems that use PAM for user authentication, which should be almost all of them these days. On your server, find and install the Google Authenticator PAM module. On Debian-like systems, a simple `apt install libpam-google-authenticator` will do the trick.
(Or grab `google-authenticator-libpam on GitHub.)
Since we're about to mess with the thing that authenticates users, open up another terminal or SSH session on the server, and *keep it open until we're done and we've tested that it works*. Otherwise, a typo could lock you out and give you a very hard problem to fix.
Edit the file `/etc/pam.d/sshd` and add the following line to the top, above the one that ends in `pam_nologin.so`:
auth required pam_google_authenticator.so account required pam_nologin.so
Add ` debug` to the end of that `pam_google_authenticator.so` line to generate more debugging output in your logs.
Next, edit your `/etc/ssh/sshd_config` file, adding this new stuff and making sure these other lines are present, too:
AuthenticationMethods publickey,keyboard-interactive:pam ChallengeResponseAuthentication yes PermitEmptyPasswords no UsePAM yes
Now, on the server, run the `google-authenticator` command to generate your secret (as a QR code) and some backup codes. Scan the QR code with your authenticator app (despite the name, it doesn't need to be Google Authenticator) and be sure to save the backup MFA codes somewhere safe.
Moment of truth, again! In a new window, try to SSH into your server again. If all goes well, you'll be prompted for an MFA code before your SSH login completes.
Note that all SSH users will be required to use MFA. If they haven't set up MFA, then PAM will deny their SSH login every time. To work around this, remove MFA enforcement by adding `nullok` to that PAM.d line we added earlier.
Now that we've configured our servers, we can create profiles for our connections so that we don't need to remember and type in every single detail every time. OpenSSH gives us two files: the system-level `/etc/ssh/ssh_config` and the user-specific `~/.ssh/config`. Both files have the same syntax. Whichever one you choose to edit depends on your personal preferences.
Note that, when you're writing this file, you may be connecting to SSH servers that you do not control. You may want to leave all secure options in here, and let your SSH servers be more restrictive.
# Anything starting with "Host *" will be applied to all outgoing SSH connections. Host * # Let's always attempt to use passwordless authentication. We'll also set a default key to use, too. IdentitiesOnly yes IdentityFile ~/.ssh/id_ed25519 # Always verify host keys via SSHFP records, if present. VerifyHostKeyDNS yes # In this section, we're going to say which host keys we will accept. The "cert" options are for SSH certificates, which I did not cover in this article, but are generally considered to be a good thing. HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256 # The next three options I already covered in detail above. KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256@libssh.org,curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256 MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr # Now, we can set host-specific stuff. In this example, I set a custom hostname and a faster cipher. Host mydesktop HostName 2001:db8:0:1::1 Ciphers aes128-gcm@openssh.com # GitHub has many IP addresses. To avoid having to constantly accept new keys, we can disable OpenSSH's IP-to-DNS mappings. Host github.com CheckHostIP no StrictHostKeyChecking ask User rhymeswithmogul # A hypothetical old piece of networking equipment may require insecure methods to be re-enabled. You can amend your KexAlgorithms, HostKeyAlgorithms, MACs, and Ciphers lists with a plus sign. I like to do it this way, in case a future firmware update offers more secure options. Host oldfirewall HostName 192.0.2.1 User admin IdentityFile ~/.ssh/id_my_old_rsa_key KexAlgorithms +diffie-hellman-group-exchange-sha1
SSH isn't hard. It's actually quite simple to configure and use, certainly more so than a modern email server. However, we've taken the time to adjust some default settings, break some compatibility, and we've hardened things significantly. You can now SSH safely, and that's all that matters.