How to Write a Fail2ban Filter

A step by step Guide

Introduction

The fail2ban program is a great way to defend your web server from those who would subvert or misuse it. A plethora of pages ( like this one, or this video ) around the web will instruct you on how to install it. The program currently ships with configurations to detect and fight over 90 possible attacks, which you can easily enable as you wish following any of the pages I mentioned. If you need to defend against any of those attacks, you're in fine shape.

I needed to write my own Fail2ban jail. My foswiki site (which you are now on) was getting so bombed by AI bots that it was becoming unresponsive. That wouldn't be so bad, except tomshiro.org also forwards my email. So when it stops working, I stop getting email. These bots were making legitimate HTTP and HTTPS requests, so the standard fail2ban configurations weren't helping with something I view as an attack. Although each individual bot's web page promises to abide by the rate limits my robots.txt file contains, when five or six hit my site at once those limits become irrelevant. So I resolved to write a fail2ban jail to slow down this useless traffic. The site is pretty much bog-standard Foswiki, run on Debian 12 (bookworm). The apache version is Apache/2.4.62 (Debian).

The /var/log/apache2/other_vhosts_access.log file was logging hits to my wiki; to show an example of how much traffic I was seeing, here's what an ls showed in that directory:

$ ls -lh other_vhosts_access.log*
-rw-r----- 1 root adm 1.9M Sep  1 20:25 other_vhosts_access.log
-rw-r----- 1 root adm 4.3M Aug 31 23:59 other_vhosts_access.log.1
-rw-r----- 1 root adm 491K Aug 23 00:00 other_vhosts_access.log.10.gz
-rw-r----- 1 root adm 417K Aug 21 23:59 other_vhosts_access.log.11.gz
-rw-r----- 1 root adm 485K Aug 21 00:00 other_vhosts_access.log.12.gz
-rw-r----- 1 root adm 415K Aug 20 00:00 other_vhosts_access.log.13.gz
-rw-r----- 1 root adm 423K Aug 19 00:00 other_vhosts_access.log.14.gz
-rw-r----- 1 root adm 340K Aug 31 00:03 other_vhosts_access.log.2.gz
-rw-r----- 1 root adm 360K Aug 30 00:00 other_vhosts_access.log.3.gz
-rw-r----- 1 root adm 370K Aug 29 00:00 other_vhosts_access.log.4.gz
-rw-r----- 1 root adm 383K Aug 28 00:00 other_vhosts_access.log.5.gz
-rw-r----- 1 root adm 345K Aug 27 00:00 other_vhosts_access.log.6.gz
-rw-r----- 1 root adm 301K Aug 25 16:29 other_vhosts_access.log.7.gz
-rw-r----- 1 root adm 418K Aug 25 00:00 other_vhosts_access.log.8.gz
-rw-r----- 1 root adm 386K Aug 24 00:10 other_vhosts_access.log.9.gz

Without logrotate(8), the log files themselves would choke my system in short order.

Instructions on writing your own fail2ban jail are pretty sparse on the web. I found a few pages with some vague hints, but mostly I had to stumble through the config files on my own until I had a thorough understanding. This page holds my thoughts.

Who should Read This

I assume that you're reasonably literate in Linux system administration. You should be literate in Regular Expression syntax, comfortable with a text editor, and comfortable working with daemons and configuration files. You don't necessarily have to be a regex golf star, but I assume some familiarity beyond searching for literals (like, you might be able to understand what egrep [0-9]+ will find).

I also assume that you're comfortable at the bash command line and have some literacy with the Bourne Again Shell.

Overview

Fail2ban jails run from several configuration files. If you have installed fail2ban and are already running it, these should be familiar. The relevant files for our purposes are:

• /etc/fail2ban/jail.local file, (usually a modified copy of /etc/fail2ban/jail.conf), which contains global configuration directives and directions for specific jails, such as which log file to examine for attacks.
• The files in /etc/fail2ban/filter.d, which contain the regular expressions fail2ban uses to analyze her logfiles.
• The files in /etc/fail2ban/action.d, which contain shell scripts that fail2ban executes when it detects an attack

Fail2ban reads all these configuration files through Python's ConfigParser module, which has its very own configuration file language. It requires section headers surrounded by [square brackets], and does interesting things with %percent% signs. If you're not familiar with ConfigParser, reading up on it will be helpful. Fail2ban also performs some token substitution of its own on the regular expression line it reads; I cover that below.

First, Catch Your Chicken: Development Environment

What?

The ConfigParser python module allows you to include other ConfigParser files inside the current one, among other fun things. This means that you must consider Fail2ban configuration files as part of a whole set of files in a directory, not in isolation.

The simplest way to develop fail2ban jails is to install fail2ban on your development box, carefully disable it from running ('cause Why, on something not serving webpages? ), and then copy the relevant stuff to your $HOME for hax04ing. If you are running a system with systemd installed, you can make sure that Fail2ban isn't running with:

systemctl disable fail2ban

I made a directory under my ${HOME} and copied /etc/fail2ban/ whole into it:

cd ; mkdir other_vhosts ; cd other_vhosts
mkdir fail2ban-config
cd /etc/
tar -cf - fail2ban | (cd ${HOME}/other_vhosts/fail2ban_config ; tar -xvf - )
cd ~/other_vhosts

Then I wrote a short shell script so I could run my filter configuration through fail2ban-regex, the program which ships with fail2ban for developing filters. This passes the appropriate arguments to fail2ban-regex so I can work in my personal copy of fail2ban's configuration directory, as copied over from /etc above. It also sets some sensible defaults for fail2ban-regex arguments; see man fail2ban-regex for details. I named my jail foswikibot, so that forced my filter to be in fail2ban-config/filter.d/foswikibot.conf .

Now I was ready to actually get to the work of developing a fail2ban filter. I copied some other_vhosts_access.log files to my development directory to play with. Log lines in those files look like this:

tomshiro.org:80 144.172.103.95 - - [28/Aug/2025:00:00:21 +0000] "GET / HTTP/1.0" 200 2224 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"
tomshiro.org:443 66.249.66.200 - - [28/Aug/2025:00:00:24 +0000] "GET /foswiki/System/InterWikis HTTP/1.1" 200 10502 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; GoogleOther)"
tomshiro.org:443 5.9.120.41 - - [28/Aug/2025:00:00:52 +0000] "GET /foswiki/Trash/WebNotify?template=backlinksallwebs HTTP/1.1" 200 5720 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; +https://help.seranking.com/en/blex-crawler)"
tomshiro.org:443 146.174.177.122 - - [28/Aug/2025:00:01:02 +0000] "GET /foswiki/TWiki/TWikiCss?rev=2 HTTP/1.1" 200 5383 "-" "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.792.0 Safari/535.1"
tomshiro.org:443 66.249.66.192 - - [28/Aug/2025:00:01:07 +0000] "GET /foswiki/TWiki/TWikiRegistration HTTP/1.1" 200 10784 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
tomshiro.org:443 66.249.66.200 - - [28/Aug/2025:00:01:08 +0000] "GET /foswiki/System/WebSearch?tab=az;recurse=;limit=;nosearch=;search=%5EV;searchletter=V;casesensitive=;web=;SEARCHb2e0c660d1a82ac160a978d79f6646af=3;scope=topic;excludetopic=;type=regex;cover=print HTTP/1.1" 200 12627 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
tomshiro.org:443 66.249.66.200 - - [28/Aug/2025:00:01:16 +0000] "GET /foswiki/System/DataForms HTTP/1.1" 200 17052 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; GoogleOther)"
tomshiro.org:443 162.128.175.136 - - [28/Aug/2025:00:01:23 +0000] "GET /foswiki/System/PageCaching?template=backlinkchildren HTTP/1.1" 200 5222 "https://tomshiro.org/foswiki/System/PageCaching?template=backlinkchildren" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"

Anatomy of a fail2ban filter file

ConfigParser reads filter files. They must contain a [Definition] header. Start comments with a #hash. Configuration items are name-value pairs, with the name on the left, an = equals sign =, and the value on the right. The most important configuration item is in the [Definition] section:

failregex =  The regular expression searched in the log file for lines indicating an attack

Filter files can also have an [INCLUDES] section, which contains a before and after file to read before and after reading the current filter. There are a bunch of other fancy options to allow you to do cool things like parameterize your regular expressions and fine-tune what fail2ban sees. I am mostly going to go for simplicity first.

Step One: Find the date stamp and the sending host in your logfile

The first step in developing a filter is to find the date stamp and the sending host in your log lines. Fail2ban uses tokens surrounded by <angle brackets> for this purpose; see man jail.conf and search for failregex for the details. You must have some identifier for the sending host in your search pattern; without that, you will get:

{^LN-BEG} : Default Detectors
2025-09-01 16:42:24,946 fail2ban.filter         [20349]: ERROR  E: No failure-id group in '^.+$'
ERROR: No failure-id group in '^.+$'

when you attempt to analyze your log file.

If your log file contains actual hostnames, you should use the <HOST> token to find the host; otherwise, you can use <IP4> to save a little time. ipv6 and other matters are beyond the scope of this page, but if necessary fail2ban can handle them also. The jail.conf man page is Your Friend in this. Fail2ban can perform a lookup to find the ip address of a hostname, but that will make it slower. It's probably best to do that in your action.d file only with actual Bad Guys, instead of looking up every hostname in your logfile. YMMV of course.

fail2ban finds the date in a logline with date.strptime . This ordinarily Just Works; I groveled through some other apache-*.conf files to copy some hints.

I was able to find the timestamp and host in my logfile with the simplest possible regular expression:

failregex = ^.+ <HOST> .+$

When I run this through test_fail2ban.sh, I get:

2025-09-01 17:15:07,303 fail2ban.filter         [21413]: TRACE  T: Working on line 'tomshiro.org:80 43.157.147.3 - - [28/Aug/2025:15:17:42 +0000] "GET / HTTP/1.1" 200 1309 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"'
2025-09-01 17:15:07,303 fail2ban.datedetector   [21413]: HEAVY  H:   matched last time template #00
2025-09-01 17:15:07,303 fail2ban.datedetector   [21413]: HEAVY  H:   got time 1756394262.000000 for '28/Aug/2025:15:17:42 +0000' using template ^[^\[]*\[(Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: HEAVY  H: Looking for match of [('tomshiro.org:80 43.157.147.3 - - [', '28/Aug/2025:15:17:42 +0000', '] "GET / HTTP/1.1" 200 1309 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"')]
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: HEAVY  H:   Looking for failregex 0 - '^.+ (?:\\[?(?:(?:::f{4,6}:)?(?P<ip4>(?:\\d{1,3}\\.){3}\\d{1,3})|(?P<ip6>(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?<=:):)))\\]?|(?P<dns>[\\w\\-.^_]*\\w)) .+$'
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: TRACE  T:   Matched failregex 0: {'ip4': None, 'ip6': None, 'dns': 'like'}
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: TRACE  T: Working on line 'tomshiro.org:443 66.249.66.192 - - [28/Aug/2025:15:17:04 +0000] "GET /foswiki/System/PerlDoc?module=Foswiki::Query::Parser HTTP/1.1" 200 8457 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; GoogleOther)"'
2025-09-01 17:15:07,303 fail2ban.datedetector   [21413]: HEAVY  H:   matched last time template #00
2025-09-01 17:15:07,303 fail2ban.datedetector   [21413]: HEAVY  H:   got time 1756394224.000000 for '28/Aug/2025:15:17:04 +0000' using template ^[^\[]*\[(Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: HEAVY  H: Looking for match of [('tomshiro.org:443 66.249.66.192 - - [', '28/Aug/2025:15:17:04 +0000', '] "GET /foswiki/System/PerlDoc?module=Foswiki::Query::Parser HTTP/1.1" 200 8457 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; GoogleOther)"')]
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: HEAVY  H:   Looking for failregex 0 - '^.+ (?:\\[?(?:(?:::f{4,6}:)?(?P<ip4>(?:\\d{1,3}\\.){3}\\d{1,3})|(?P<ip6>(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?<=:):)))\\]?|(?P<dns>[\\w\\-.^_]*\\w)) .+$'
2025-09-01 17:15:07,303 fail2ban.filter         [21413]: TRACE  T:   Matched failregex 0: {'ip4': None, 'ip6': None, 'dns': 'Mobile'}

Results
=======

Failregex: 2945 total
|-  #) [# of hits] regular expression
|   1) [2945] ^.+ <HOST> .+$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [9190] ^[^\[]*\[(Day(?P<;_sep>;[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
`-

Lines: 9190 lines, 0 ignored, 2945 matched, 6245 missed
[processed in 1.22 sec]

You can see here that although only 2945 lines matched the failregex, fail2ban found a date in all of them. Hurray.

Step Two: Write a regular expression which catches all the lines in your logfile

Now look at the failregex. It is only matching a few of the lines in your log file. Your goal is to write a failregex which matches all the lines in your logfile, with the <HOST> and timestamp tokens also read correctly. If fail2ban finds a timestamp, it will remove it from the log line before trying to match (not search -- see re.match ). So the log line you're really trying to match will read something like:

tomshiro.org:443 66.249.66.199 - - [] "GET /foswiki/TWiki/VarACTIVATEDPLUGINS HTTP/1.1" 200 7574 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.183 Mobile Safari/537.36 (compatible; GoogleOther)" 

I have found sites like debuggex and regex101 useful for this task. Regex has lots of different dialects; be sure you're on the Python one when using sites like these. Regular expressions read left to right, just as people do, so start with the first characters in your log line (in my case, the name of my server).

Regular expressions are complex enough that you can write a regex which is nonsense. Python will sensibly abend the compilation of such a thing, and the end result will be that you'll get an error like the following one:

+ fail2ban-regex -c ./fail2ban-config -l HEAVYDEBUG --print-no-missed ./other_vhosts_access.log foswikibot.conf

Running tests
=============

2025-09-02 16:26:33,242 fail2ban.filter         [11068]: DEBUG  D: Setting usedns = warn for Filter(None)
2025-09-02 16:26:33,242 fail2ban.filter         [11068]: DEBUG  D: Created Filter(None)
Use   failregex filter file : foswikibot, basedir: ./fail2ban-config
2025-09-02 16:26:33,242 fail2ban.configreader   [11068]: INFO   I: Loading configs for filter.d/foswikibot under ./fail2ban-config 
2025-09-02 16:26:33,242 fail2ban.configreader   [11068]: DEBUG  D: Reading configs for filter.d/foswikibot under ./fail2ban-config 
2025-09-02 16:26:33,243 fail2ban.configreader   [11068]: DEBUG  D: Reading config files: ./fail2ban-config/filter.d/foswikibot.conf
2025-09-02 16:26:33,243 fail2ban.configparserinc[11068]: INFO   I:   Loading files: ['./fail2ban-config/filter.d/foswikibot.conf']
2025-09-02 16:26:33,243 fail2ban.configparserinc[11068]: TRACE  T:     Reading file: ./fail2ban-config/filter.d/foswikibot.conf
2025-09-02 16:26:33,243 fail2ban.configparserinc[11068]: INFO   I:   Loading files: ['./fail2ban-config/filter.d/foswikibot.conf']
2025-09-02 16:26:33,243 fail2ban.configparserinc[11068]: TRACE  T:     Shared file: ./fail2ban-config/filter.d/foswikibot.conf
Real  filter options : {'logtype': 'file', 'datepattern': '^[^\\[]*\\[({DATE})\n{^LN-BEG}'}
Use      datepattern : ^[^\[]*\[({DATE})
{^LN-BEG} : Default Detectors
2025-09-02 16:26:33,244 fail2ban.filter         [11068]: ERROR  E: Unable to compile regular expression '^tomshiro.org:(?z80|443) (?:::f{4,6}:)?(?P(?:\d{1,3}\.){3}\d{1,3}) - - \[\] "GET /foswiki.+ HTTP/[0-9]+\.[0-9]+" [0-9]{3} [0-9]{3,5} .+(Bytespider|ClaudeBot|iboubot).+$':
unknown extension ?z at position 15
ERROR: Unable to compile regular expression '^tomshiro.org:(?z80|443) (?:::f{4,6}:)?(?P(?:\d{1,3}\.){3}\d{1,3}) - - \[\] "GET /foswiki.+ HTTP/[0-9]+\.[0-9]+" [0-9]{3} [0-9]{3,5} .+(Bytespider|ClaudeBot|iboubot).+$':
unknown extension ?z at position 15

You can also cause ConfigParser to fail to read your config file. Remember that all '%' signs must be doubled unless they are specific instructions to ConfigParser, for example variable interpolations. This failure looks like:

+ fail2ban-regex -c ./fail2ban-config -l HEAVYDEBUG --print-no-missed ./other_vhosts_access.log foswikibot.conf

Running tests
=============

2025-09-02 16:29:18,298 fail2ban.filter         [11208]: DEBUG  D: Setting usedns = warn for Filter(None)
2025-09-02 16:29:18,298 fail2ban.filter         [11208]: DEBUG  D: Created Filter(None)
Use   failregex filter file : foswikibot, basedir: ./fail2ban-config
2025-09-02 16:29:18,298 fail2ban.configreader   [11208]: INFO   I: Loading configs for filter.d/foswikibot under ./fail2ban-config 
2025-09-02 16:29:18,298 fail2ban.configreader   [11208]: DEBUG  D: Reading configs for filter.d/foswikibot under ./fail2ban-config 
2025-09-02 16:29:18,298 fail2ban.configreader   [11208]: DEBUG  D: Reading config files: ./fail2ban-config/filter.d/foswikibot.conf
2025-09-02 16:29:18,298 fail2ban.configparserinc[11208]: INFO   I:   Loading files: ['./fail2ban-config/filter.d/foswikibot.conf']
2025-09-02 16:29:18,299 fail2ban.configparserinc[11208]: TRACE  T:     Reading file: ./fail2ban-config/filter.d/foswikibot.conf
2025-09-02 16:29:18,299 fail2ban.configparserinc[11208]: INFO   I:   Loading files: ['./fail2ban-config/filter.d/foswikibot.conf']
2025-09-02 16:29:18,299 fail2ban.configparserinc[11208]: TRACE  T:     Shared file: ./fail2ban-config/filter.d/foswikibot.conf
Traceback (most recent call last):
  File "/usr/bin/fail2ban-regex", line 34, in 
    exec_command_line()
  File "/usr/lib/python3/dist-packages/fail2ban/client/fail2banregex.py", line 844, in exec_command_line
    if not fail2banRegex.start(args):
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/fail2ban/client/fail2banregex.py", line 735, in start
    if not self.readRegex(cmd_regex, 'fail'): # pragma: no cover
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/fail2ban/client/fail2banregex.py", line 394, in readRegex
    reader.getOptions(None)
  File "/usr/lib/python3/dist-packages/fail2ban/client/configreader.py", line 343, in getOptions
    self._opts = ConfigReader.getOptions(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/fail2ban/client/configreader.py", line 178, in getOptions
    return self._cfg.getOptions(section, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/fail2ban/client/configreader.py", line 264, in getOptions
    v = self.get(sec, optname, vars=pOptions)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/configparser.py", line 815, in get
    return self._interpolation.before_get(self, section, option, value,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/configparser.py", line 396, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "/usr/lib/python3/dist-packages/fail2ban/client/configparserinc.py", line 61, in _interpolate_some
    return super(BasicInterpolationWithName, self)._interpolate_some(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/configparser.py", line 443, in _interpolate_some
    raise InterpolationSyntaxError(
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '% %(yourhost)s:(80|443)  - - \\[\\] "GET %(foswikiroot)s.+ HTTP/[0-9]+\\.[0-9]+" [0-9]{3} [0-9]{3,5} .+%(bots)s.+$'

For me, the first winning regex was:

failregex = ^tomshiro.org:(80|443) <IP4> - - \[\] .+$

This gave me a report like this:

Results
=======

Failregex: 9190 total
|-  #) [# of hits] regular expression
|   1) [9190] ^tomshiro.org:(80|443) <IP4> - - \[\] .+$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [9190] ^[^\[]*\[(Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
`-

Lines: 9190 lines, 0 ignored, 9190 matched, 0 missed
[processed in 0.68 sec]

Now you can extend your regular expression to the right, still catching every line or figuring out what lines it doesn't catch. About a hundred lines in my log file didn't match a "GET .+" request; I grep(1) ed them to find that they were actually when I and others were legitimately editing pages on the wiki, which naturally uses a POST command. So the longest regular expression I could write that still matched everything was:

^tomshiro.org:(80|443)  - - \[\] "(GET|HEAD|POST).+$

Step Three: Modify your failregex to catch the lines showing attacks

In my case, I wanted to exclude known bots. Here's what I came up with:

bots = (Bytespider|ClaudeBot|iboubot)
yourhost = tomshiro.org
foswikiroot = /foswiki

failregex=^%(yourhost)s:(80|443) <IP4> - - \[\] "GET %(foswikiroot)s.+ HTTP/[0-9]+\.[0-9]+" 200 [0-9]{4,5} "-" ".+; %(bots)s.+\)\
"

I am using a bit of ConfigParser language here to set variables inside my regular expression; the %(bots)s string puts in the regular expression for the bots I wish to block, and the other two work in similar fashion.

This reports both hits and misses when I use fail2ban-regex:

Results
=======

Failregex: 214 total
|-  #) [# of hits] regular expression
|   1) [214] ^tomshiro.org:(80|443) <IP4> - - \[\] "GET /foswiki.+ HTTP/[0-9]+\.[0-9]+" 200 [0-9]{4,5} "-" ".+ (Bytespider|ClaudeBot|iboubot).+"
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [9190] ^[^\[]*\[(Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?)
`-

Lines: 9190 lines, 0 ignored, 214 matched, 8976 missed
[processed in 0.56 sec]

Check Your Results

Now the Real Fun begins. I got a plausible number of hits on my failregex, above. But I can't know from just that number if fail2ban saw all and only the bots hitting my foswiki server. If she saw fewer bots than were actually in the log file, bots which I don't detect are hitting my site. If she saw more bots than were actually in the log file, legitimate users are being mistaken for bots and banned. Both of those outcomes are Bad.

I used egrep(1) to strain out and count just the bot lines from my log file:

egrep -c '(Bytespider|ClaudeBot|iboubot)' other_vhosts_access.log
358

In my case, the number of matches I got with egrep(1) (358) was larger than the number of matches fail2ban gave me (214). I had to investigate this. I created a file containing only bot hits:

egrep '(Bytespider|ClaudeBot|iboubot)' other_vhosts_access.log > bytespider.txt

I was then able to run test_fail2ban.sh against this file with the -p print-all-missed flag:

test_fail2ban.sh -f bytespider.txt -p print-all-missed > test.out

That gave me the same report as above, but after the Lines: line I got a |- Missed line(s): report with all the lines which fail2ban-regex missed. By careful squinting and some more narrowing down of possible matches, I was able to see that my original failregex had these flaws:

• Not all log lines contained a "200" response code
• Not all log lines had the "-" string after the two numbers
• Bots were hitting my site outside of my wiki. I didn't care about those hits, they are mostly static pages which are cheap to serve.

So now I have a reasonably trustworthy filter.d file. The next step is action.d.

--30--

This is an old silly joke about a chicken soup recipe which starts "First, catch your chicken". Back