fail2ban for Web Services: Protect Nginx from Brute Force and Bad Bots

In the SSH hardening tutorial you gave fail2ban a single job: watch the SSH log and ban password-guessing bots. That's the "hello world" of fail2ban. Your web tier faces a different, noisier threat model — credential stuffing against login forms, vulnerability scanners walking thousands of known-bad URLs, bad bots scraping at machine speed, and floods that tip your app over. fail2ban can defend all of it, but only once you understand how it actually works and how to keep it from banning your own users.

This is the deep dive. You'll learn the real anatomy of a jail, turn on the built-in Nginx protections, write a custom filter and prove it works with fail2ban-regex before deploying it, wire bans into your firewall, and — most importantly — fix the reverse-proxy problem that quietly breaks fail2ban for most real deployments.

What You'll Build

• A working mental model: jail = filter + action + thresholds
• Enabled built-in Nginx jails (nginx-http-auth, nginx-botsearch, nginx-limit-req)
• A custom filter for repeated login failures, validated with fail2ban-regex
• Bans applied through UFW / nftables, with a whitelist so you never ban yourself
• The reverse-proxy / Cloudflare fix so fail2ban bans real client IPs, not your proxy
• Exponential backoff so repeat offenders get progressively longer bans

Step 1: Understand the Anatomy of a Jail

Everything in fail2ban is a jail, and every jail is four things working together:

• Filter — a set of regular expressions (failregex) that match "this was a failure" lines in a log. The special <HOST> token captures the offending IP. Filters live in /etc/fail2ban/filter.d/.
• Log source — the file (or systemd journal) the jail watches, via logpath and backend.
• Thresholds — maxretry failures within findtime earns a ban of length bantime.
• Action — what happens on ban, defined in /etc/fail2ban/action.d/ (default: add a firewall drop rule).

So "ban an IP that fails HTTP basic-auth 5 times in 10 minutes for an hour" is just filter=nginx-http-auth, logpath=error.log, maxretry=5, findtime=10m, bantime=1h, action=<firewall>. Once that clicks, the rest is configuration.

As with SSH, never edit the shipped jail.conf — your overrides go in jail.local, which survives package upgrades.

Step 2: Set Sane Global Defaults

Open the local override and set defaults that every jail inherits:

sudo nano /etc/fail2ban/jail.local

1[DEFAULT]
2# Never ban yourself, localhost, or your monitoring/CDN ranges
3ignoreip = 127.0.0.1/8 ::1 203.0.113.10
4
5# Ban math
6findtime = 10m
7maxretry = 5
8bantime  = 1h
9
10# Repeat offenders get progressively longer bans
11bantime.increment = true
12bantime.factor    = 2
13bantime.maxtime    = 1w
14
15# Use the firewall you already run (UFW users)
16banaction = ufw

ignoreip is your safety net — put your own admin IP there so a bad afternoon of testing can't lock you out. bantime.increment turns a fixed hour into exponential backoff: an IP that keeps coming back gets 1h, 2h, 4h… up to a week. And banaction = ufw routes bans through the UFW firewall you set up earlier instead of raw iptables. (On a host without UFW, leave the packaged default — nftables on modern Debian/Ubuntu, iptables-multiport on older systems.)

Step 3: Enable the Built-in Nginx Jails

fail2ban ships filters for the common Nginx attack patterns. Add these jails to jail.local:

1[nginx-http-auth]
2enabled = true
3# reads /var/log/nginx/error.log by default — catches HTTP basic-auth brute force
4
5[nginx-botsearch]
6enabled  = true
7logpath  = /var/log/nginx/access.log   # override: exercises the filter's 404-scan branch (its default is error.log)
8maxretry = 2
9
10[nginx-limit-req]
11enabled = true
12logpath = /var/log/nginx/error.log

• nginx-http-auth bans IPs failing HTTP Basic authentication (protected admin areas).
• nginx-botsearch catches scanners requesting known-bad paths (/phpmyadmin, /.env, /wp-admin on a non-WordPress site).
• nginx-limit-req works with Nginx's own limit_req directive — when Nginx starts returning 503s to a flooder, fail2ban bans them outright. (This jail only fires if you've configured limit_req in Nginx; the Nginx directives cheat sheet shows how.)

Step 4: Write a Custom Filter for Login Failures

Built-in filters won't know about your app's login endpoint. Say your app returns 401 on a failed POST /api/login. Create a filter:

sudo nano /etc/fail2ban/filter.d/nginx-login.conf

[Definition]
# Matches: 1.2.3.4 - - [..] "POST /api/login HTTP/1.1" 401 ...
failregex = ^<HOST> -.*"(GET|POST) /api/login\b[^"]*" 401\b
ignoreregex =

The <HOST> at the start captures the client IP from the combined access-log format; the rest matches the login path returning 401. Do not enable this yet — first prove the regex actually matches your logs.

Step 5: Test the Filter With fail2ban-regex (Before Deploying)

This is the step that separates a working jail from a silent no-op. fail2ban-regex runs a filter against a real log and reports exactly what it matched:

sudo fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-login.conf

Read the output carefully:

Failregex: 42 total
|-  #) [# of matches] pattern
|   1) [42] ^<HOST> -.*"(GET|POST) /api/login\b[^"]*" 401\b
`-
Lines: 10000 lines, 0 ignored, 42 matched, 9958 missed

If matched is 0, your regex doesn't fit your log format — usually a custom log_format or a different path/status. Adjust and re-run until real failures match and nothing legitimate does. Only a filter that passes here should become a jail. Now enable it:

1[nginx-login]
2enabled  = true
3filter   = nginx-login
4logpath  = /var/log/nginx/access.log
5maxretry = 5
6findtime = 5m
7bantime  = 2h

Step 6: Reload and Verify

Apply the config and confirm your jails are live:

sudo systemctl reload fail2ban
sudo fail2ban-client status

You should see all enabled jails listed. Inspect one:

sudo fail2ban-client status nginx-login

This shows currently failed IPs, total bans, and who's banned right now. Watch fail2ban act in real time:

sudo tail -f /var/log/fail2ban.log

To manually ban or lift a ban:

sudo fail2ban-client set nginx-login banip 198.51.100.5
sudo fail2ban-client set nginx-login unbanip 198.51.100.5

Step 7: Fix the Reverse-Proxy / Cloudflare Trap

This is the single most common way fail2ban silently fails for web services. If your app sits behind a reverse proxy, load balancer, or CDN (Cloudflare, an ALB, another Nginx), the client IP in your access log is the proxy's IP, not the visitor's. fail2ban then either bans the proxy (cutting off all traffic) or, if the proxy IP is in ignoreip, bans nobody.

Fix it by making Nginx log the real client IP. With Cloudflare, trust its ranges and read the CF-Connecting-IP header:

# In your Nginx http {} or server {} block
set_real_ip_from 173.245.48.0/20;   # ...and the rest of Cloudflare's published ranges
real_ip_header   CF-Connecting-IP;

For a generic proxy on your own network:

set_real_ip_from 10.0.0.0/8;
real_ip_header   X-Forwarded-For;

After reloading Nginx, $remote_addr (and therefore your access log and every fail2ban filter) shows the true client IP. Re-run fail2ban-regex to confirm real IPs now match. For a CDN, also consider banning at the edge via the CDN's own firewall/API action so the traffic never reaches your origin — but real-IP logging is the minimum to make host-level fail2ban meaningful.

Common Issues

• fail2ban banned your reverse proxy / CDN. You're logging the proxy IP. Configure set_real_ip_from + real_ip_header as in Step 7 so the log shows the client IP, and keep the proxy ranges in ignoreip as a backstop.
• A jail shows 0 failures no matter what. Either the logpath is wrong, or a custom Nginx log_format breaks the filter's regex. Verify with fail2ban-regex <logfile> <filter> — if it matches there but the jail doesn't fire, check logpath and backend in the jail.
• You banned Googlebot / a legitimate crawler. Aggressive 404/botsearch jails catch well-behaved crawlers too. Raise maxretry, tighten the failregex, or add trusted crawler IPs to ignoreip. Never build a "ban on any 404" jail — legitimate traffic generates 404s.
• Bans don't actually block traffic. The banaction isn't taking effect. If you set banaction = ufw but UFW is inactive, bans no-op — enable UFW, or switch to nftables-multiport. Check sudo iptables -L -n / sudo nft list ruleset for the fail2ban chains.
• IPv6 clients aren't banned. Ensure your banaction and firewall handle IPv6 (modern nftables/ufw do) and that Nginx is logging the IPv6 address the filter's <HOST> can capture.

Frequently Asked Questions

Isn't a Web Application Firewall (WAF) better than fail2ban for web attacks?

They solve overlapping but different problems. A WAF (ModSecurity, Cloudflare WAF) inspects request content to block injection, XSS, and known exploit signatures in real time. fail2ban reacts to behavior over time — repeated failures from one IP — and bans at the network layer. On a small-to-mid deployment, fail2ban is far cheaper to run and catches brute force and scanners well; larger or higher-risk apps often run both, with the WAF at the edge and fail2ban as host-level backup.

How do I stop fail2ban from banning legitimate users behind shared NAT?

Shared IPs (corporate offices, mobile carriers, CGNAT) mean many users look like one address, so an aggressive jail can ban all of them. Mitigate by keeping maxretry reasonable, using shorter findtime windows so transient bursts don't accumulate, whitelisting known corporate ranges in ignoreip, and preferring precise filters (a specific failing endpoint) over broad status-code bans.

Why test with fail2ban-regex instead of just enabling the jail?

Because a jail with a non-matching filter fails silently — it looks enabled, reports zero activity, and provides zero protection while you assume you're covered. fail2ban-regex shows you the exact match count against real log lines, so you deploy only filters you've proven work against your actual log format. It's the difference between security and the illusion of it.

Should bantime be permanent for web attackers?

Usually no. Attacker IPs are frequently transient (botnets, cloud instances, residential proxies), so a permanent ban list grows without bound and can catch a later-legitimate user of a recycled IP. Exponential backoff (bantime.increment) is the better default: persistent offenders climb toward very long bans automatically, while one-off scanners age out. Reserve permanent bans for confirmed, targeted abuse.

Does this work with Apache instead of Nginx?

Yes — the concepts are identical, only the filter names and log paths change. fail2ban ships apache-auth, apache-badbots, apache-noscript, apache-overflows, and more, reading /var/log/apache2/*.log. Your custom-filter and fail2ban-regex workflow is exactly the same; point logpath at the Apache access/error log and adjust the failregex to Apache's log format.

Tear Down

Disable a specific jail by setting enabled = false in jail.local (or removing its block) and reloading:

sudo systemctl reload fail2ban

Unban everyone in a jail at once, or stop fail2ban entirely:

sudo fail2ban-client unban --all
sudo systemctl disable --now fail2ban

Remove custom filters you added (/etc/fail2ban/filter.d/nginx-login.conf) and, if desired, uninstall:

sudo apt remove --purge fail2ban

Official References

• fail2ban Wiki — jails, filters, and actions — the authoritative source
• Nginx ngx_http_realip_module — real client IP behind proxies
• Cloudflare IP ranges — the set_real_ip_from list to trust
• man jail.conf, man fail2ban-regex — always on the box

This completes the server-hardening path: firewall with UFW → SSH hardening → fail2ban for web. Building the Nginx config those jails watch? The Nginx config builder gets you a sane starting point.