, 4 min read
Blocking HTTP 404 traffic
1. Problem statement:
There are some bots which requests lots of pages of my web-server, which do not exist.
These bots are blocked on the network level with iptables.
Here is an example of excessive HTTP 404 errors:
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.zip HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tgz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.002|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.xz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.7z HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.rar HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.zst HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.sql.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backup.sql.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backup.sql HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backups.zip HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backups.tar.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
Obviously, this not a human accidentally referencing a wrong page. For this particular IP address the HTTP 404 errors occured more than 14,000 times! Also, if the visitor would want to know all the files on the web-server he could have queried the sitemap in either HTML or XML format.
2. Solution.
In Replacing SSHGuard with 20 Lines of Perl Code
I already block ssh connections, which fail during their login.
So the machinery with iptables and ipset is already in place.
The rules for iptables are:
*raw
:PREROUTING ACCEPT [207:14278]
:OUTPUT ACCEPT [180:113502]
#-A PREROUTING -i eth0 -p tcp --dport 22 -m set --match-set reisbauer src -j DROP
COMMIT
# Empty iptables rule file
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i eth0 -p tcp --dport 443 -m set --match-set https src -j DROP
-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set reisbauerHigh src -j DROP
-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set reisbauerLow src -j DROP
#-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set sshguard4 src -j DROP
COMMIT
Above iptables rule needs an ipset set named https:
create -exist https hash:net family inet hashsize 65536 maxelem 65536 counters
Even after a few hours this filter blocks quite some traffic.
Name: https
Type: hash:net
Revision: 7
Header: family inet hashsize 65536 maxelem 65536 counters bucketsize 12 initval 0x9ae05ab4
Size in memory: 4928
References: 1
Number of entries: 37
Members:
23.239.12.200 packets 0 bytes 0
45.148.10.95 packets 0 bytes 0
5.61.209.33 packets 7 bytes 308
185.177.72.12 packets 0 bytes 0
143.198.202.83 packets 0 bytes 0
2.57.122.173 packets 0 bytes 0
104.208.108.55 packets 0 bytes 0
45.148.10.62 packets 0 bytes 0
35.78.92.79 packets 0 bytes 0
195.178.110.199 packets 0 bytes 0
45.148.10.21 packets 0 bytes 0
172.202.92.73 packets 0 bytes 0
15.223.237.92 packets 0 bytes 0
65.108.72.214 packets 0 bytes 0
195.178.110.31 packets 0 bytes 0
185.177.72.100 packets 0 bytes 0
52.138.34.68 packets 0 bytes 0
89.185.81.112 packets 0 bytes 0
185.177.72.30 packets 0 bytes 0
20.116.109.56 packets 0 bytes 0
34.118.194.53 packets 0 bytes 0
176.34.152.60 packets 0 bytes 0
20.29.72.143 packets 0 bytes 0
213.209.159.175 packets 0 bytes 0
185.177.72.54 packets 0 bytes 0
34.176.114.197 packets 0 bytes 0
185.177.72.22 packets 0 bytes 0
4.204.235.48 packets 0 bytes 0
20.116.59.164 packets 0 bytes 0
185.177.72.58 packets 0 bytes 0
185.177.72.49 packets 0 bytes 0
80.94.95.211 packets 0 bytes 0
23.137.105.178 packets 0 bytes 0
34.165.43.146 packets 299775 bytes 17987150
20.220.151.41 packets 0 bytes 0
45.148.10.200 packets 0 bytes 0
13.212.167.23 packets 0 bytes 0
iptables -nvL --line-numbers shows this:
Chain INPUT (policy ACCEPT 135K packets, 60M bytes)
num pkts bytes target prot opt in out source destination
1 276K 17M DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 match-set https src
2 1944 120K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 match-set reisbauerHigh src
One day later:
Chain INPUT (policy ACCEPT 697K packets, 114M bytes)
num pkts bytes target prot opt in out source destination
1 818K 49M DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 match-set https src
2 7629 511K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 match-set reisbauerHigh src