How to Block Bad Bots and Scraper Traffic

Not all bots are bad — search engines and uptime monitors are welcome. But bad bots scrape your content, probe for vulnerabilities, brute-force logins, spam forms and waste server resources. Left unchecked, they inflate your bills and slow the site for real users. Here's how to identify and block them at multiple layers.

First, know what you're dealing with

Look at your access logs to see who's hitting you and how often:

# Top user-agents
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Top IPs by request count
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

A single IP or user-agent making thousands of requests, ignoring robots.txt, or hitting /wp-login.php repeatedly is a red flag.

Layer 1: Block by User-Agent (Nginx)

map $http_user_agent $bad_bot {
    default 0;
    ~*(AhrefsBot|SemrushBot|MJ12bot|DotBot|PetalBot) 1;
    "" 1;                       # empty UA is suspicious
}

server {
    if ($bad_bot) { return 403; }
    # ... rest of your server block
}

Layer 1: Block by User-Agent (Apache)

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot|MJ12bot|DotBot) [NC]
    RewriteRule .* - [F,L]
</IfModule>

Layer 2: Rate-limit aggressive clients

User-agents are trivially spoofed, so rate limiting is your real defense. It caps how fast any single IP can hit you:

limit_req_zone $binary_remote_addr zone=perip:10m rate=30r/m;

server {
    location /login {
        limit_req zone=perip burst=5 nodelay;
    }
}

Layer 3: Block IPs and ranges at the firewall

For persistent abusers, drop them before they reach the web server at all:

# Block a single IP
sudo iptables -A INPUT -s 203.0.113.42 -j DROP

# Block a whole range (CIDR)
sudo iptables -A INPUT -s 203.0.113.0/24 -j DROP

Layer 4: Automate with Fail2ban

Manually chasing IPs doesn't scale. Fail2ban watches your logs and bans IPs that trip a rule (too many 404s, failed logins, etc.) automatically:

# /etc/fail2ban/jail.local
[nginx-badbots]
enabled  = true
filter   = nginx-badbots
logpath  = /var/log/nginx/access.log
maxretry = 10
findtime = 60
bantime  = 3600

Don't block the good bots

Be surgical. Allow Googlebot, Bingbot and your uptime monitor, or you'll hurt your SEO and lose visibility into your own uptime. Verify search engine bots by reverse DNS rather than trusting the user-agent, and allowlist your monitoring provider's published IPs.