This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Configuration

The controller's tree is an agent's tree plus two things. Every key, with its default, is in the configuration reference; this page is the shape of it.

/etc/shieldlist/controller/
  controller.yaml         the service itself
  parsers/<source>/ rules/ policies/ reports/   the LIBRARY: every parser, rule, policy, preset — once
  packs/<name>.yaml       a PACK: a named list of rules — what a server is given
  agents/<hostname>.yaml  one machine: its packs, the rules it adds, what it changes, its logs
  trust.yaml exemptions.yaml exemptions.d/      the fleet's, to every machine
  exemptions-cli.yaml     what the Web UI, the API and shield exempt
  secrets.yaml            the controller's own provider credentials

Packs: which rules run where

A pack is a list of rules for a type of server — plesk, infra, docker, mysql — and the unit a server is given: packs: [plesk, infra] in its declaration. A rule is never copied, only listed: a server runs the union of its packs (and the rules its declaration names on their own), each rule once, and receives only what those rules need — their parsers, their policies, their presets. A server whose declaration names no packs: at all runs the packs controller.yaml gives every machine (the shipped ones, on a fresh install); packs: [] runs none. In the Web UI: Rules → Packs is a grid of packs × servers to tick, and a server's Rules tab has its packs (each on or test, at the server's level or its own), the rules added on their own, and everything it runs with the threshold it crosses on there.

Where addresses are from

geoip:
  source: dbip            # the free DB-IP lite country + ASN databases, fetched monthly under <state>/geoip
mail:                     # (beside smtp/user/password/from/security) the same account read back:
  imap: imap.example.net:993
  imap_user: abuse@example.net
  imap_password: "…"
  imap_security: tls      # or starttls; empty: by the port
reporter:                 # who signs the abuse reports, exported or mailed (Settings → Reporting → Abuse reports)
  org: Example Hosting    # empty parts fall back on console_url's host and mail.from
  contact: abuse@example.net
  domain: example.net
owner_reports:            # mails to the networks' abuse desks, one per banned address (off by default) — see the Abuse reports page
  enabled: off
  default: off            # networks without a setting of their own (Lists → Networks sets one per network)
  format: xarf            # words + X-ARF document | text
  daily_limit: 200
  per_network: 20
  lines: 3
  copy_to: abuse@example.net   # a blind copy of every mail, header X-Shieldlist: owner-report
  subnets: on                  # a prefix an offenders list blocks is told to its desk: one mail per desk per pass, the subnets that entered with every address banned in each (header X-Shieldlist: subnet-report; on by default under the switches above; never twice for a prefix within a day)
  subnet_subject: "Abuse report: {what} blocked after repeated attacks - {date_rfc}"   # {what}: the prefix, or "N subnets of AS…"
  subnet_template: |           # that mail's text (empty: as shipped): {prefixes} {count} {asn} {as_name} {date} {date_rfc} {window} {subnets} {org} {domain} {contact} {acknowledgement} — {subnets} is each subnet's account, written by the controller
    Network: AS{asn} {as_name}
  bounces: on                  # read the mailbox (mail.imap): the delivery failures, and the desks' answers sorted by patterns
  bounces_folder: INBOX        # your own patterns: reply-patterns.yaml beside controller.yaml (see docs/CONFIG.md)
  keep_sent: on                # file a copy of every report in the mailbox (instead of copy_to)
  sent_folder: Sent
  every: 10m                   # the ceiling between two passes (5s at least); IMAP IDLE reads an answer within seconds anyway
  delay: 15s                   # a ban waits this long before its report leaves (late log lines)
  registries: on               # keep the RIPE and AFRINIC dumps locally (~270 MB) and find the desks there before RDAP
  registries_every: 7d
  purge: 7d                    # settled answers and sent copies older than this leave the mailbox (empty: kept)
  rules:                       # which rules' bans are mailed: every rule unless `except` names it (patterns: ssh-*)
    except: [redirect-crawler, bad-bot]
  rule_texts:                  # what the mail says of a rule, over the rule's own report words
    redirect-crawler: "A crawler that ignores a 301 is not after the content…"
  template: |                  # the mail's text, with placeholders (empty: as shipped; Settings → Abuse reports shows and previews it)
    Source IP: {ip}
# or files of your own (GeoLite2 or any MaxMind-DB provider):
#  country: /var/lib/geoip/GeoLite2-Country.mmdb
#  asn: /var/lib/geoip/GeoLite2-ASN.mmdb

Off unless configured — a fetch is outbound traffic. Sanctions are enriched as they arrive (country, AS number and name) and rows from before are backfilled; the Web UI shows a flag and the network, filters and counts by them; the API answers country, asn, as_name. DB-IP's licence (CC BY 4.0) asks for attribution: the Web UI shows "IP Geolocation by DB-IP".

controller.yaml

listen:
  agents: ":17453"     # the agent transport
  api: 127.0.0.1:17454      # the API and the Web UI
  public: 127.0.0.1:17455   # the public unblock page alone — see below; unset: it answers on the API listener
state: /var/lib/shieldlist/controller
log_file: /var/log/shieldlist-controller/controller.log   # its own log (the unit's LogsDirectory), as well as the journal; notices logrotate by itself
tls:                        # absent: self-signed, pinned by agents through the join token
  cert: /path/fullchain.pem #   the operator's own certificate…
  key: /path/privkey.pem
  off: false                # …or plain HTTP behind a proxy on a private address
level: standard             # the fleet's default level
packs: [openssh, bind, mariadb, postfix-dovecot, web-essentials]   # what a machine without a declaration runs — the shipped packs, on a fresh install
agent_state: test           # and what it does with them until told otherwise: test | on — arming is a person's act
reporting: agent            # who reports a machine's sanctions: agent | controller | off
lift: owners                # who may lift a machine's bans: owners | operator
sources: [googlebot, bingbot, cloudflare]   # published exemption lists to fetch
sources_refresh: 24h        # on this schedule; unset = on command only
ipv6:                       # how far a ban follows an offender through its allocation
  widen_to: 56              #   off | 56 | 48
  widen_after: 4/24h        #   that many distinct /64 of one allocation, inside a day
recidivism:                 # the long memory — the addresses that keep coming back
  bans: 5                   #   banned this many times within the window…
  machines: 4               #   …or by this many machines
  within: 30d
  duration: 30d             #   the ban, the first time
  again: 365d               #   and when it happens again within this long
  off: false

Everything is optional; an absent file is a controller on one machine.

Recidivism follows the sharing. Every ten minutes the controller looks for the addresses banned again and again, and judges each machine on what that machine sees: its own bans, and the ones it applies from the servers sharing its packs (fleet: offer/enforce below). Where the count reaches bans (or machines), that machine is ordered to ban the address — "recidivism" is the order's author, in Banned by hand. A machine that shares nothing counts its own bans alone, and none of them reaches another machine this way. A person's lift of one of the bans counted stands. The fleet's copies and the bans by hand never count.

A ban by hand on the whole fleet reaches every machine, the ones enrolled later too — except the machines the ban itself names (except in the ban dialog, except in POST /v1/bans). The sharing switches govern what the machines decide, not what a person decides for the fleet.

Deleted yours, or want the shipped one back? shieldlist-controller init writes it when there is none — with the free rule library beside it. It never replaces what is already there, so running it again after an upgrade only adds what is new.

listen.public: the public page on a name of its own

The unblock page faces the world; the Web UI faces its operator. Served on one address they cannot be separated — an operator who wants their Web UI behind an address filter has to take the page down with it.

Give listen.public an address — in the file, or from the Web UI (Settings → General → Unblock page, Listener of its own, bound on save, no restart) — and the page moves:

listen:
  api: 127.0.0.1:17454      # the Web UI, which you may now close
  public: 127.0.0.1:17455   # the page, and only the page
public:
  page_url: https://unban.example.net/    # where it answers

Two vhosts on your proxy, one per listener — and the public one may live on another machine entirely, since it needs nothing but that port. The second one is the first with another name and another port:

server {
    server_name unban.example.net;
    listen 80;
    listen [::]:80;
    location / {
        proxy_pass http://127.0.0.1:17455;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

ln -s /etc/nginx/sites-available/unban.example.net.conf /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx
certbot --nginx -d unban.example.net --redirect

What this buys is not only convenience: on the public listener /v1, the Web UI and the sign-in form do not exist, so a mistake in the public vhost cannot reach them. Its bare address redirects to the page.

page_url matters because operators hand the old address out: the Web UI links there, and the API listener's own /unblock answers a 301 to it. Set both together, or people follow a link to a 404.

ipv6: following an offender through its allocation

An IPv6 offender is counted and banned at /64, because that is one network's standard allocation. But a customer is given a /56, which holds 256 of them, and an attacker who holds one walks through its /64 for nothing: the same client returns as a new address every time and is banned afresh every time.

widen_to says how far a ban may follow them, and widen_after how many distinct /64 of the same allocation must have been banned inside the window before it does. The offender being judged counts as one of them, so 4/24h means it and three siblings.

widen_to	what happens
absent, or off	every ban stays at the /64 it is counted at
56	the usual customer allocation — the sensible choice
48	an organisation, and often a whole hosting region

48 is offered because an operator may deliberately want it. Nothing reaches that far on its own: the escalation is one step, from the /64 to whatever widen_to names.

Settings → Bans in the Web UI sets all of this — never, /56 or /48, and after how many. It writes the block below and reloads; there is nothing to edit by hand.

This is one setting for the whole fleet. The controller sends it to every machine, so there is nothing to configure per host and no rule to edit; an agent too old to know it simply leaves its bans where they are counted. A widened ban is still refused inside an origin whose trust.yaml entry says range_ban: never.

agents/.yaml

The file is named after the machine's hostname, its identity — the hostname as the machine reports it (hostname there: web1.example.net or web1; the Web UI's Servers page shows it). A machine without a file — or whose file leaves packs: out — runs the packs controller.yaml names, in the state agent_state says (test).

agent: web01
name: web01.example.net          # what the Web UI, the exports and the desks' mails call it — default: the hostname as the machine reports it; give a machine that reports a short one its full name
description: the shared-hosting machine
level: strict                    # scales every rule here: instant | strict | standard | lenient
packs: [openssh, plesk, custom]   # the packs it runs — or, per pack, how:
# packs:
#   plesk: {}
#   infra: {state: test, level: strict}
state: on                        # on | test — relaxes the machine, never arms it
fleet:                           # the pack is the unit of sharing: a ban travels
                                 # only to servers running a pack that holds its rule
  offer: no                      # its bans go to the servers sharing its packs (default yes)
  enforce: no                    # it applies the bans of the servers sharing its packs (default yes; lifts and bans by hand always arrive)
lift: operator                   # only the operator may lift its bans
reporting: agent                 # agent | controller | off
rules:                           # what this machine changes about a rule — or adds
  ssh-bruteforce: {level: lenient}
  http-bad-ua: {threshold: [3/1m, 10/1h]}
  wp-scan: {state: off}
  mail-auth: {ban: mail-hardcore}
  root-login:                    # a rule outside its packs: runs here as written
logs:                            # where its logs are (pushed; empty: its own agent.yaml says)
  - {path: /var/log/nginx/*.log, log: http, ports: [80, 443]}
  - {path: "/var/www/vhosts/system/{target}/logs/attribution.log", attribution: '$time_iso8601 $remote_addr $remote_port $host "$request"'}

The second entry is an attribution log: a second access_log the web server writes with the TCP source port, declared by its nginx log_format string and read by no parser — the agent attaches the port (src_port) to each line of evidence when a ban is decided, and the reports carry it (the X-ARF payload, the desk's mail, the console). What a network behind a carrier-grade NAT needs to find the subscriber (RFC 6302). The agent's Configuration page has the nginx side. Agents from build 317 of 0.1.0 read it; an older agent is given the declaration without it, and the assembly says so.

An entry under rules: changes what one machine runs of one rule — its level, its threshold outright, its state, its policy — and, for a rule outside the machine's packs, adds it. Parsers, policies and presets are shared, never overridden: a machine that needs a different regex needs a different rule.

Who reports

reporting: — the fleet's default in controller.yaml, each machine's own in its declaration — says who submits a machine's enforced sanctions to the abuse provider: agent (the machine itself, with the key in its own secrets.yaml), controller (the controller on its behalf, with the key and daily_limit in the tree's secrets.yaml — one account and one reporting address for the fleet), or off. A machine told controller pauses its own sender; the controller then reports each of its sanctions once, as it arrives, composed from the rule's report preset and the fields the convicting lines named — the same words the agent would have used — and keeps the outcome on the sanction (the Web UI says "reported by the controller"). What was not enforced is never submitted, whoever would report; a report the provider refused or the allowance could not cover is not retried; a lift "with withdraw" takes back the controller's own report where it was the controller that reported.

# secrets.yaml — beside controller.yaml, 0600
abuseipdb:
  key: your-api-key
  daily_limit: auto           # your plan, or a number below it; auto learns it from the provider
  # enabled: false            # keep the key, send nothing (Settings → Reporting → AbuseIPDB writes this)

Levels

A rule writes its numbers once; the machine's level scales them, the same way for every rule: instant — the first hit convicts; strict — half the hits; standard — as written; lenient — twice the hits. Set for the fleet, per server, per pack on a server, per rule on a server. See the agent wiki's Writing rules.

Assembly

For each machine the controller assembles the tree it will run: the rules of its packs and the ones it adds, the parsers they score, the policies and presets they name, the fleet's exemptions — the rules that differ rewritten in place (comments kept) with the resolved threshold, state and policy — validated with the agent's own loader before it leaves. An agent older than 0.1.0 (the format's first release) still syncs but is refused a bundle by name and keeps running what it last received. shieldlist-controller -t shows every machine's assembly and what differs. A tree written in the 0.0.x format is named by the loader: shieldlist-controller -config DIR -convert NEWDIR rewrites it, comments kept.

Lists: block or exempt a network, a country, prefixes

lists/<name>.yaml names a set of addresses and says what it does where:

list: as398781
description: a proxy network — flood of last month
asn: 398781              # or country: CN, or prefixes: [203.0.113.0/24, 198.51.100.7]
block: fleet             # a preventive firewall ban of every prefix — the fleet, or [web01, web02]
exempt: [ns1, pack:web]  # never sanctioned on these machines and on every machine running the web pack (the key was `allow` once; still read)
duration: permanent      # the blocks' duration: permanent (default), 30d, 1y…

A list may instead hold offending subnets — offenders, with asn for one network's:

list: as6079-offenders
asn: 6079
offenders:
  within: 7d        # the window looked back: 24h … 90d (default 7d)
  addresses: 5      # at least this many banned addresses in one of the network's prefixes…
  share: 30         # …or at least this % of its addresses (IPv6 in /64s) — either, or both, whatever its size
block: fleet

…or without asn for every network's, narrowed by the banned addresses' countries and networks:

list: offenders
offenders:
  within: 7d
  addresses: 20
  share: 10
  except_countries: [FR, BE, CH, DE]   # the addresses of these countries never count (or countries: [CN, RU] — those alone)
  except_asns: [3215]                  # nor these networks (or asns: [6079, 16509] — those alone)
block: {packs: [plesk]}

The prefixes are the ASN database's own entries — the ones a network's page lists — each address counted in the entry filed for it. They are worked out again at every pass of the lists (ten minutes) from the bans the fleet applied over the window — its own convictions, never the copies nor a detection in test: a prefix enters when it starts attacking and leaves when its bans age out, the rest of the network stays reachable. Several lists may judge one network, each with its own criteria and its own scope; a network with a list of its own is not left out of a global one (two lists may block the same prefix, each where it applies — except_asns leaves a network out). The dialog (Ban its offending subnets… on a network's page, Ban offending subnets… on Lists) lists every prefix the criteria would block, with its banned addresses and share, before the list is written. An offenders list blocks; it exempts nothing.

The list keeps a ledger of its parts: when each prefix entered, when it left, what was convicted in it when last counted, whether its desk was told (GET /v1/lists/{name} → parts, past; the network's page says Blocked by … since … on each prefix). A prefix entering is told to its network's abuse desk when owner reports are on (owner_reports.subnets, below).

A list may instead name a published list:

list: malicious-ip
url: https://raw.githubusercontent.com/romainmarcoux/malicious-ip/main/full-40k.txt   # or urls: [a, b]
format: lines            # lines | cidr-lines | json-prefixes | ips-anywhere
refresh: 1h              # 10m at the least; 24h when absent
attribution: Aggregated by Romain Marcoux — MIT licence
block: [pack:web]

It is fetched over https only, on its cadence and on Refresh now, read into prefixes and nothing else, bounded (a million entries, nothing wider than a /8 or an IPv6 /20, a sixteenth of the internet in all, public addresses only — a line that is not an address refuses the whole fetch, and a failed fetch keeps the last good file under <state>/lists/), then pushed to the machines in its block scope as data — lists/<name>.set in their bundle, held in sets of their own on the agent (see the agent's Enforcement page): no orders, no sanctions, a diff at every refresh. With an exempt scope it is an allow list (its prefixes travel as exemptions; at most 20 000). A catalogue ships — preset: malicious-ip in the file, or the Web UI's Published blocklists — each entry with its attribution.

A scope is fleet, or a list mixing machine names, pack:<name> items — the machines running that pack, the ones given it later too — and group:<path> items — the machines whose declaration puts them in that group or one below it (group:clients covers clients/acme), the ones moved there later too, and none of the ones moved out: a group is read from the declarations at every reload and every apply, never turned into machine names. Search engines exempted on the web hosts and never on the DNS servers is exempt: [pack:web]; the CloudFlare ranges exempted on the shared hosts alone is exempt: [group:shared]. except:<machine> carves one machine out of any of these.

Block travels as DATA, never as orders: the list's resolved prefixes are published as one nftables set (lists/<name>.set in the bundle) that each machine in the scope loads whole — seconds for tens of thousands of prefixes, one named drop counter per list, no sanction, no expiry; removing the block (or the list) withdraws the set. The machine's own exemptions are carved out of the set before it loads, so an exemption always wins. An autonomous system is resolved to its prefixes from the ASN database, a country from the country database (geoip in controller.yaml), at most 100 000 prefixes per list. Exempt travels in the bundle as exemptions. A machine's own exemptions always win over a block. The Web UI's Lists page is the remote control; any network or country opened from the rail offers Block…. The feed leaves the list-driven bans out (copies-like: GET /v1/sanctions?list_blocks=include shows them).

Published allow lists (sources:)

Search engines and CDNs publish their address ranges. Naming a source in controller.yaml lets the controller fetch it, on the schedule and on command (shield refresh cloudflare, the Web UI's Refresh now). The catalogue is compiled in — googlebot, google-special-crawlers, bingbot, applebot, duckduckbot, duckassistbot, openai-searchbot, openai-chatgpt-user, openai-gptbot, anthropic, perplexity-user, perplexitybot, mistral-user, mistral-index, google-user-triggered-fetchers, prestashop-cloudsync, cloudflare, fastly, jetpack, ahrefs, uptimerobot, pingdom, stripe-webhooks, and the presets anthropic-outbound (the calls Claude makes to a site's MCP server, a fixed allocation), private-ipv4, private-ipv6, carrier-nat — nothing is fetched unless named, the result is a generated file under exemptions.d/ (never hand-edit it), a list that looks like nothing a crawler publishes is refused, and the last good file stays when a fetch fails. geoip: {source: dbip, city: on} adds DB-IP City Lite beside the country and ASN editions (Settings → About → Cities): the city and region under every address and in the search — a reading aid, never a decision. Costs about 60 MB downloaded a month and 130 MB of memory on the controller; the sanctions on file are completed in the background the first time. With files of your own, city: is the path of a city database. console_session: 24h is how long a Web UI session lasts after the last activity (12h unless set; 30 days since sign-in at most). A source applies everywhere unless sources_on narrows it, by source, to a scope:

sources: [googlebot, bingbot, cloudflare, uptimerobot]
sources_on:
  googlebot: [pack:web]        # the servers running the web pack — not the DNS servers
  uptimerobot: [web01, web02]

An allow list of your own (custom_sources) may use the format ips-anywhere: every address or range found in the document, whatever its shape (JSON of any layout, text, HTML).

Retention (retention: in controller.yaml)

What the store keeps, applied once a day; Settings → Maintenance shows the sizes and can flush sooner.

retention:
  evidence: 180d     # the convicting lines (and what they named)
  samples: 400d      # the agents' hits/bans-in-force minute lines (the charts)
  # sanctions: 730d  # unset: the sanctions themselves are kept for good

Bans in force are never removed, whatever the spans say.

Update checks (updates: in controller.yaml)

updates: off stops the daily look at the forge's releases; on (the default), Settings → About says what the latest controller and agent releases are, the topbar shows an Update available pill, and GET /v1/updates answers the same.

Reporting switch (secrets.yaml)

abuseipdb.enabled: false keeps the key and sends nothing — the Web UI's Settings → Reporting → AbuseIPDB switch writes it; absent means on. The same key works in the agent's own secrets.yaml for machines that report with their own key (reporting: agent).