Harden SSH Access on Ubuntu and Debian: Keys, sshd_config, and fail2ban

If you followed the UFW firewall tutorial, you've closed every inbound port except the one you can't live without: SSH. That single open door is the most attacked service on any internet-facing Linux box — bots hammer port 22 with password guesses around the clock. Hardening it is the highest-value security step you can take on a server.

This tutorial moves SSH from "username + password" to a genuinely locked-down setup: key-only authentication, no root login, no password login at all, a clean drop-in sshd_config, and fail2ban to automatically ban IPs that misbehave. Every change is done in the safe order — you'll always confirm a working key login before removing your fallback, and keep a rescue session open when you reload the daemon.

The golden rule, same as with the firewall: never disable a login method until you've proven the new one works. Get that order wrong on a remote server and you're rebuilding from a console.

What You'll Build

• An Ed25519 SSH key pair, with the public key installed on the server
• Key-only authentication — password and keyboard-interactive logins disabled
• Root SSH login disabled, access restricted to specific users
• A tidy /etc/ssh/sshd_config.d/ drop-in, with the effective config verified via sshd -T
• fail2ban watching the SSH log and banning brute-force sources automatically

Step 1: Make Sure You Have a Non-Root Sudo User

You'll be disabling root SSH, so you need a normal user with sudo to get in. If you already log in as one (e.g. deploy), skip ahead. Otherwise, as root:

adduser deploy
usermod -aG sudo deploy      # 'sudo' on Ubuntu/Debian

Log out and confirm you can SSH in as deploy and run sudo -v before going further.

Step 2: Generate an SSH Key Pair (On Your Local Machine)

Run this on your laptop/workstation, not the server. Ed25519 keys are small, fast, and stronger than the old RSA default:

ssh-keygen -t ed25519 -C "deploy@my-laptop"

Accept the default path (~/.ssh/id_ed25519) and set a passphrase — it encrypts the private key at rest, so a stolen laptop doesn't hand over server access. You now have a private key (id_ed25519, never leaves your machine) and a public key (id_ed25519.pub, safe to share). If you'd rather generate and inspect keys in the browser first, the SSH key generator tool is a quick reference.

Step 3: Install Your Public Key on the Server

The easy way, from your local machine:

ssh-copy-id deploy@your-server-ip

This appends your public key to ~/.ssh/authorized_keys on the server and fixes permissions. If ssh-copy-id isn't available, do it manually — permissions matter, and SSH silently ignores the key if they're too open:

ssh deploy@your-server-ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
cat ~/.ssh/id_ed25519.pub | ssh deploy@your-server-ip "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Step 4: Test Key Login Before Changing Anything

This is the safety gate. Open a new terminal and log in:

ssh deploy@your-server-ip

You should land on the server prompted only for your key passphrase (or nothing, if you loaded it into an agent) — not the server password. If it still asks for the account password, the key isn't being accepted; fix that now (usually a permissions or path issue) before you touch sshd_config. Do not proceed until a key login works.

Step 5: Harden sshd With a Drop-In Config

Modern Ubuntu and Debian include /etc/ssh/sshd_config.d/*.conf from the main config, so the clean approach is a dedicated drop-in rather than editing the big file. Create it:

sudo nano /etc/ssh/sshd_config.d/99-hardening.conf

1# Key-only, no root, restricted users
2PermitRootLogin no
3PasswordAuthentication no
4KbdInteractiveAuthentication no
5PubkeyAuthentication yes
6
7# Reduce the attack surface
8MaxAuthTries 3
9LoginGraceTime 20
10X11Forwarding no
11
12# Only these accounts may SSH in (space-separated)
13AllowUsers deploy

Each line: no direct root logins, no passwords or keyboard-interactive prompts (key only), fewer auth attempts per connection, a short grace window, X11 forwarding off, and an explicit allow-list of users.

⚠️ The cloud-init gotcha. sshd uses the first value it reads for each setting, and drop-ins are read in alphabetical order. Many cloud images ship /etc/ssh/sshd_config.d/50-cloud-init.conf with PasswordAuthentication yes — and because 50- sorts before 99-, it wins, silently overriding your file. Always verify the effective config in the next step, and if password auth is still on, edit 50-cloud-init.conf directly (or rename your file to sort earlier).

Confirm what sshd will actually apply — this reads the merged config exactly as the daemon will:

sudo sshd -T | grep -Ei '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|allowusers)'

You want to see permitrootlogin no, passwordauthentication no, pubkeyauthentication yes. If password auth is still yes, that's the cloud-init file — fix it and re-check.

Step 6: Validate and Reload — Safely

Never restart sshd blind. First, syntax-check:

sudo sshd -t

No output means the config is valid. Now reload the service (which keeps your current connection alive):

sudo systemctl reload ssh    # some systems use 'sshd' as the unit name

Keep your current session open. From a separate terminal, prove a fresh login still works:

ssh deploy@your-server-ip

Only once that new session succeeds should you trust the change. If it fails, you still have the original session to revert 99-hardening.conf and reload again.

Step 7: Install and Configure fail2ban

Key-only auth already defeats password guessing, but fail2ban stops the noise (and protects any other services) by watching logs and banning offending IPs at the firewall. Install it:

sudo apt update
sudo apt install -y fail2ban

Never edit the shipped jail.conf — create a local override that survives package upgrades:

sudo nano /etc/fail2ban/jail.local

[sshd]
enabled  = true
maxretry = 3
findtime = 10m
bantime  = 1h
# bantime  = -1   # uncomment for permanent bans

Enable and start it, then confirm the SSH jail is active:

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

The status output shows currently failed and banned IPs. To lift a ban manually:

sudo fail2ban-client set sshd unbanip 203.0.113.10

Bonus — automatic security updates. A hardened door on an unpatched server is still exposed. Turn on unattended security upgrades:

sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

Common Issues

• Locked out after disabling password auth. This means key login was never actually working and you removed the fallback anyway. Recover through your provider's serial/VNC console (AWS EC2 Serial Console, DigitalOcean/Hetzner console, Proxmox noVNC), log in locally, and set PasswordAuthentication yes in 99-hardening.conf (then reload) to regain a temporary way in while you fix the key. Prevention is Step 4 — never skip the key-login test.
• sudo sshd -T shows passwordauthentication yes despite your drop-in. A lower-numbered drop-in (usually 50-cloud-init.conf) is winning because the first value read wins. Edit that file, or rename yours to sort before it.
• The key is ignored and it still asks for a password. Almost always permissions: ~/.ssh must be 700 and ~/.ssh/authorized_keys 600, both owned by the user. sudo tail -f /var/log/auth.log (or sudo journalctl -u ssh -f on journald-only minimal installs) shows exactly why sshd rejected the key.
• fail2ban banned you while testing. From another allowed IP or the console: sudo fail2ban-client set sshd unbanip <your-ip>, and add your admin IP to ignoreip in jail.local.
• Wrong service name on reload. If systemctl reload ssh errors with "unit not found," try sshd. On very new Ubuntu the daemon is socket-activated (ssh.socket), but reload ssh still applies config changes.

Frequently Asked Questions

Is key-only SSH really safer than a strong password?

Yes, meaningfully. A password can be brute-forced or phished; an Ed25519 private key is 256 bits of entropy that never travels over the network and (with a passphrase) is useless if stolen. Disabling PasswordAuthentication also makes the endless bot password-guessing traffic simply fail at the protocol level, which is why it's the single biggest SSH hardening win.

Do I still need fail2ban if I've disabled password authentication?

It's optional but worth it. With key-only auth, brute-force password attempts can't succeed — but fail2ban still reduces log noise, bans hosts probing for other weaknesses, and protects any additional services you jail later (web logins, mail, etc.). Think of it as defense in depth rather than the primary control.

Should I change the SSH port from 22?

It's security-through-obscurity, not real security — a port scan finds it in seconds — but moving to a high port does cut the volume of automated noise hitting your logs. If you do it, set Port 2222 in your drop-in, allow the new port in your firewall first (sudo ufw allow 2222/tcp), and update your SSH client config. The key/password hardening above matters far more than the port.

Why use a drop-in file instead of editing /etc/ssh/sshd_config directly?

Drop-ins in /etc/ssh/sshd_config.d/ keep your changes isolated, easy to audit, and safe across package upgrades that may rewrite the main file. The one catch is ordering — because sshd uses the first value it reads and files load alphabetically, always confirm the merged result with sudo sshd -T rather than assuming your file wins.

How do I let a whole team in without sharing one key?

Give each person their own key pair and append every public key (one per line) to ~/.ssh/authorized_keys for the shared account, or better, create a per-user account and list them all in AllowUsers (or use AllowGroups ssh-users and add users to that group). Never distribute a single private key — you lose the ability to revoke one person without rotating everyone.

Tear Down

To relax the hardening (for example, to re-enable password login temporarily), remove or edit your drop-in and reload:

sudo rm /etc/ssh/sshd_config.d/99-hardening.conf
sudo sshd -t && sudo systemctl reload ssh

To remove fail2ban and unattended-upgrades:

sudo systemctl disable --now fail2ban
sudo apt remove --purge fail2ban unattended-upgrades

Removing a public key from ~/.ssh/authorized_keys revokes that key's access immediately — no reload needed.

Official References

• OpenSSH sshd_config manual — every directive, authoritative
• Ubuntu Server — OpenSSH documentation — distro-specific setup
• fail2ban documentation — jails, actions, and filters
• man sshd_config, man fail2ban-client — always on the box

Pair this with the firewall you built in Set Up a Firewall with UFW, and generate or validate your client config with the SSH config generator.