Shieldlist/shieldlist-agent
Watch
1
0
Fork
You've already forked shieldlist-agent
 0
Code Issues Pull requests Projects Releases 4 Packages Wiki Activity Actions
4 releases 4 tags
  • v0.0.9 0b05b0fbcd

    v0.0.9 Stable

    Robin released this 2026-08-26 21:35:03 +02:00 | 1 commits to main since this release

    Shieldlist Agent 0.0.9

    What it is

    The agent is what actually protects a machine: it tails its logs, scores what it reads against your rules, and blocks the offenders in the kernel — by itself, with no database and no daemon beside it. It works alone with local files, and it can be enrolled with a controller to be driven with the rest of a fleet.

    What you gain over 0.0.3

    Mostly speed and steadiness: a large ban set no longer costs you the loop, a flood no longer stalls detection while the firewall is being written, and a restart's restore takes seconds instead of minutes.

    Then what came with the fleet: enrolled mode, a simpler configuration format, rules that count by site or by origin, lists that block whole networks, and a test mode that decides like production and bans nobody.

    Everything that changed

    Faster, and steadier under attack

    • Single addresses live in plain firewall sets and only ranges in interval sets: listing an interval set dumps all of it before any change, so with a hundred thousand bans every call cost the best part of a second, stalled each round and turned a restore into minutes.
    • A refused batch is retried without the element the kernel names, then bisected — never one process per element — and the reason it was refused is logged.
    • Orders and lifts from a controller are applied as one pass: one transaction, one firewall batch, one log line. Thousands applied one at a time used to freeze evaluation for minutes.
    • The sync sends what changed instead of scanning the store, and the machine's own counters are published rather than recomputed.
    • A query that walked the whole sanctions table on every sync was split into two indexed halves merged in order.
    • The bans in force come from the published aggregates, not a table scan on every sync.
    • WatchKeep in two passes: count every group, build lines and progress only for the survivors.
    • The control loop's questions are answered the moment they arrive, not on the next snapshot.
    • A permanent ban's timeout is written in days — the kernel refuses nine digits of seconds, and long bans were silently refused.
    • Lifting keeps a kernel element another live sanction still needs; every value entering the store is canonical, so two rows can never fight over one element.
    • The unit asks for a CPU weight instead of a hard quota: a quota rationed the restore and the enforcement exactly under attack.
    • The restore says where its time went — store, kernel, waiting for the loop.
    • One named drop counter read by reset, instead of listings that dump the sets.
    • Aho-Corasick prefilters for the rules whose patterns are long lists of literals.

    Detection

    • Hits survive a restart: the rules' windows are snapshotted to disk and come back, and they cross an in-place reload.
    • {target} in a log input's path names the site every line of that file is about — a field, a filter and a facet, without allocating; a fixed file can declare its target.
    • Rules count by origin: countries, asns and their exceptions, with the geo databases pulled from the controller or named locally.
    • targets / except_targets: count only what was aimed at a site, or everything but.
    • A severer rule's conviction extends and renames an existing sanction instead of stacking a second one.
    • tighten: a returning offender needs less evidence per past episode.
    • Evidence lines keep what the parser captured, by name — the username tried, the path asked for, the site aimed at.
    • Lines two web servers write for one request are counted once.
    • A hit is a line some rule scores against the offender: a successful login weighed against a ban is not counted as one.
    • Several thresholds per rule, and a verdict restarts every threshold's counter for that offender.
    • The machine is never an offender: loopback and its own interface addresses are exempt in the core, and a non-routable address is never reported.

    Enrolled mode

    • shield enrol <token> attaches the machine: the bundle is validated before it is swapped in and reloaded in place — the pipeline is rebuilt between rounds, the store and the kernel untouched.
    • A received configuration that no longer loads is set aside at start; the machine runs its own files until a good one arrives.
    • The sync carries what it decided, who it is counting and how close they are, its running view (memory, lines and hits per minute, pending verdicts, shed, errors) and its clock.
    • It is nudged the moment a round bans or hits, so the console shows this second's decisions, and falls back to its interval on a quiet machine.
    • Suspects travel as a delta, with their lines; a suspect new to the list sends its last lines whole, so the controller always holds lines for what it shows.
    • Bans the operator orders are applied like any sanction — same checks, same enforcement, recorded as manual with the reason as evidence, reported when a preset is named.
    • Blocked lists arrive as data and load into their own firewall sets off the main loop, carved around the exemptions; shield why and shield status say when an address is covered by one.
    • The controller's log inputs replace the local ones; exemptions by country or AS are assembled for the machine.
    • Detect-only from the controller is remembered across a restart, and arming says when a restart is needed.
    • Nothing executable ever comes down that link: a controller sends configuration, never code.

    The configuration made simple

    • One rule, one file, five blocks; parsers named or written inline; thresholds as 5/10m, as a list, or scaled by level (instant, strict, standard, lenient).
    • Packs group rules; -convert rewrites a 0.0.x tree once.
    • state: test on a rule or a machine: decides like production, bans nothing.
    • share: no keeps a rule's bans off the fleet.
    • An old-format file is named once and dropped, not half-validated.

    Reporting

    • The comment ends with the episode's minute, first report and renewal alike, so a repeat offence is actually stored by the provider.
    • Sends may run in parallel; an unreachable provider is no longer read as a spent daily allowance; reporting can be switched off without losing the key.
    • The rule is unchanged: one chance at the moment of the decision, and shield report -last <duration> is the manual catch-up.

    Commands and diagnostics

    • shield status <key> prints one value alone; new_today counts from midnight in the machine's own zone.
    • shield tail follows decisions and their evidence live; shield why explains one address; shield ranges shows the busiest buckets.
    • SHIELDLIST_DEBUG=1 writes a debug log with each sync's cost; SHIELDLIST_PPROF=<addr> opens Go's profiler on a loopback address.
    • Per-parser and per-rule cost sampling, published in health and shown by shield stats — a rule at zero hits is visible.

    Operations

    • Debian packages for amd64, arm64, armhf and riscv64 from our APT repository; a hardened unit; logrotate; man pages; third-party licence notices.
    • The unit never wants nftables.service: Debian's own unit flushes the whole ruleset when it starts, taking every other firewall's rules with it.

    Security

    • A hardening pass on the geolocation database reader against crafted files.
    • https:// required for a controller address.
    • The packaged unit no longer blocks shield exempt.

    Early days

    Formats can still move between releases and an upgrade can ask for a re-enrolment. Start on test servers, keep what protects you today, and tell us what breaks.

    Credits

    Geolocation: DB-IP Lite (CC BY 4.0). Go modules: THIRD_PARTY_LICENSES. Detection rules translated from public sources keep their attribution in the rule files.

    Downloads
  • v0.0.3 8c441d5a51

    v0.0.3 Pre-release

    Robin released this 2026-08-15 16:07:48 +02:00 | 121 commits to main since this release

    Shieldlist agent v0.0.3 — the reporting release. The agent can now tell the world about the offenders it bans, watch itself do it, and post its decisions to your own endpoints, while several sharp edges found by running it armed on a production host were filed off.

    Abuse reporting (AbuseIPDB)

    • The agent submits the offenders it bans to AbuseIPDB under your API key. Off unless you arm it: reporting.provider in agent.yaml is the only switch, the key and your daily_limit live in secrets.yaml (0600), and what each report says is a preset file you own — reports/*.yaml, referenced by a rule's report_as:. An agent in detect-only never reports anybody.
    • Redaction is default-deny: a preset's include: list is the only way a parsed field can leave the machine, the comment is composed the moment the ban is decided and stored with it (shield why shows the exact text), and the validator refuses a preset naming a field its rule never captures. A comment describes the offender's behaviour — one sample per included field with a distinct count — and never your thresholds, your server's configuration, or anyone else's usernames and mailboxes.
    • A report gets one chance, at the moment of the detection, and it leaves on the ban's heels: the decision loop nudges the sender as soon as a round's bans are durable. Whatever stops a report — allowance spent, provider down, key refused — the reason is one log line and the sanction stays on record, still owed. Nothing is retried on a cadence and no backlog drains by itself.
    • shield report -last 6h sends the past on purpose: it counts what is owed in the window you name, shows the estimate, asks, then submits freshest first at full speed, stopping cleanly when the provider says the allowance is spent. The allowance is watched from both ends — your daily_limit binds absolutely, the provider's rate-limit headers are read on every response, the reset is midnight UTC, and a bounded probe notices an upgraded plan or another machine freeing room.
    • Addresses that are not publicly routable are never submitted, and the machine's own addresses are never offenders in the first place (see below).

    Outbound hooks

    • hooks: in agent.yaml POSTs one JSON document per enforced ban to any URL you name — a chat relay, a SIEM, your own dashboard — carrying the ban's facts and the same default-deny comment a report would, never a raw log line. Hooks are local configuration by invariant: a controller can never push one. One delivery attempt each, right behind the ban, outcome logged, nothing in detect-only.

    Watching it: shield tail, and a store that counts returns

    • shield tail follows decisions live, each with the log lines that convicted it, plus again lines when a sanctioned offender keeps coming back, reported lines when the report leaves (with the provider's abuse score), and the provider's own words when it refuses. One terminal shows the whole story.
    • Sanctions now carry hits and last seen (schema 11, migrated automatically): every absorbed return is counted, evidenced (bounded per sanction), and said — an address that keeps coming back is visible coming back. shield bans grows HITS, LAST and REPORTED columns; shield why tells each sanction's full story including its report's fate; last seen is the log's clock, not the verdict's.
    • shield status answers more without costing more: reporting counters (owed, sent, duplicate, refused) and the exemption count in the default output, published by the agent's own aggregates pass — plus every loaded rule listed, the zero-ban ones included, because a rule that never fires should be visibly at zero, not absent. Reports piling past ten thousand owed is one edge-triggered WARN.

    The machine itself is never an offender

    • Loopback and the machine's own interface addresses cannot be sanctioned by any rule, built into the core rather than left to an exemption file — no configuration of this program can firewall the machine it runs on. Loopback events are dropped at evaluation, before any state exists: a local service authenticating against another (a webmail against its IMAP server) is plumbing, not an attack, and the real client is in the web application's own log.

    Faster list parsers

    • prefilter_any: — a parser may now gate on a list of tokens instead of a single literal, answered by a multi-literal automaton in under a microsecond. This is what makes big list parsers affordable: a 683-entry User-Agent list that cost ~920µs per line ungated now costs ~1µs on the lines it ignores, a measured ~3300× improvement. Exactly one gate per parser, and the gate only ever admits lines the pattern then judges.

    Also

    • Log lines about reports read like the decision lines they follow — rule, action, bare copyable address — and every egress outcome is one line with the provider's exact words where they matter.
    • The systemd-less quick start, examples and wiki are updated; docs/CONFIG.md gains the full reporting, hooks and prefilter_any reference.

    Upgrading from v0.0.2: replace the binaries and restart; the store migrates itself (schema 9 → 11). The reporting block, presets and hooks are opt-in — nothing changes until you write them.

    Downloads
  • v0.0.2 5ac56e5d2d

    v0.0.2 Pre-release

    Robin released this 2026-08-15 00:30:20 +02:00 | 151 commits to main since this release

    Three fixes from putting v0.0.1 on a real, busy production machine — a
    hosting node with a database, twenty PHP pools and twenty-three sites.
    Every one of them was invisible on a quiet test box.

    The agent's memory ceiling no longer follows the machine

    The agent used to subtract a reserve from the machine's currently
    available     memory, so that it would "give room back" as the protected
    service grew. On a machine that genuinely uses its memory it did the
    opposite of its intent: a nightly job pushed available memory under the
    reserve, and the agent's budget collapsed from 929 446 buffered events
    to 16 384 — while the process was holding 16 MB. It shed a capacity it
    had never taken, because of what something else was doing at midnight.

    limits.memory is now a fixed ceiling and the whole of the policy.
    The agent reads MemTotal — enough to resolve 25% — and nothing else.
    It does not watch free memory and does not shrink when something else
    grows.

    A program whose behaviour depends on the rest of the box is one whose
    behaviour cannot be reproduced, and one that competes with every other
    program for the same slack. Tell this one what it may have; dividing the
    rest is the administrator's job, and MemoryMax in the unit is the tool
    for it.

    Exemptions were being silently discarded

    loadList decoded straight into the destination slice, and yaml.v3
    replaces a slice rather than appending to it. So the second
    exemption file read discarded the first: the moment shield exempt
    created exemptions-cli.yaml, every exemption written by hand stopped
    existing — an operator's own infrastructure included — and nothing said
    so. The count in shieldlist-agent -t was the only trace, and only if
    you knew what it should have been.

    If you have ever run shield exempt, check the count:

    shieldlist-agent -t

    exemptions.d/

    The useful exemption lists are generated: the ranges Google and Bing
    publish run to a couple of thousand entries and are refreshed from a
    source. Pasted into exemptions.yaml they bury the dozen a person
    maintains, and the next refresh has to preserve them.

    /etc/shieldlist/
  exemptions.yaml         yours
  exemptions.d/*.yaml     one file per generated source
  exemptions-cli.yaml     what `shield exempt` wrote

    All of them load, and each entry remembers which file it came from.

    The agent writes its own log

    Under systemd the log goes to the journal, which is right — but
    /var/log is the first place anyone looks after starting a security
    daemon, and finding nothing there reads as "it did not start".

    log_file: /var/log/shieldlist/agent.log

    Set in the shipped configuration, with the same lines as journalctl -u shieldlist-agent. The lines written before the configuration was read
    are replayed into it, so it holds the whole run. No logrotate snippet
    ships with it: the file is identified by (device, inode) like every
    other file the agent follows, so a rename is noticed and the file
    reopened. The unit gains LogsDirectory=shieldlist, without which
    ProtectSystem=strict leaves /var/log unwritable.

    Upgrading

    install.sh again — it replaces the binaries and touches no
    configuration. Which means an existing install needs log_file added by
    hand if you want the file.

    Still pre-1.0 and still unsigned: the checksums prove the download was
    not corrupted and nothing more.

    Downloads
  • v0.0.1 79e7025fae

    v0.0.1 Pre-release

    Robin released this 2026-08-14 19:11:16 +02:00 | 153 commits to main since this release

    The first real release of the standalone agent.

    It tails the log inputs you declare, evaluates your rules, applies
    sanctions to the machine's own nftables, and keeps every decision in a
    local store that survives restarts. One binary, plain YAML, no database
    daemon, no external service on the machine it protects. shield ships
    alongside for status, bans, why, unban, exempt and replay.

    Install

    curl -fsSLO https://git.lrob.net/Shieldlist/shieldlist-agent/releases/download/v0.0.1/install.sh
less install.sh          # one file of sh — read it before running it
sh install.sh

    It enforces nothing until you say so

    A fresh install writes a starter configuration in detect-only: every
    rule records what it would have done, and nftables is never touched.
    Run it beside fail2ban, CrowdSec or whatever already guards the machine,
    read a day of it —

    shield status               # what it would be holding
shield bans                 # which addresses, by which rule
shield why 203.0.113.9      # the log lines that convicted one of them

    — then set state: on in /etc/shieldlist/agent.yaml and restart.

    Exempt yourself first, in /etc/shieldlist/exemptions.yaml.

    What is in it

    • Detection: literal-prefiltered RE2 parsers, rules that score
      weighted evidence across parsers within a window, negative weights so
      good traffic can exculpate, distinct counting, group_by ip and
      range:/n.
    • Sanctions: duration ladders that climb only when an offender
      returns after a ban ran out, port scopes per rung, decay measured from
      a ban's end, exemptions (with until:), trust weights on origins.
    • Enforcement: one inet table, prerouting, ct direction original
      so machine-initiated replies are spared; persist-first, so losing the
      agent never unbans; a restart rebuilds the data plane from the store
      in the background while the agent already watches.
    • Reading the past: shieldlist-agent -dry-run 24h [-rule NAME]
      judges by window and ban duration — a crossing counts only if the
      sanction it earns would still be in force now.
    • Bounds: a memory ceiling the agent holds itself to (half the
      machine by default), measured per-rule and per-parser cost, ordered
      shedding, and figures for all of it in shield stats.

    Proven on a 2-core VPS: ~68k lines/s at one core, and 5.1M distinct
    addresses banned without losing one. What nothing can bound — nftables
    set memory is kernel memory — is written down in docs/RESOURCES.md.

    It checks the kernel before trusting it

    At startup the agent builds its own table under a throwaway name, with
    no hook — so nothing in it is ever evaluated against a packet — puts
    elements in it with timeouts and concatenated address·port keys, and
    removes it again. What it proves is what the agent will actually ask
    for, because the probe renders the same script.

    A kernel that refuses gc-interval gets a warning and a working agent;
    an expired element stops matching traffic whether or not it has been
    collected. A kernel that refuses anything else stops the agent at
    startup with one sentence naming what it refused, rather than a parse
    error out of a child process in the middle of an attack.
    shieldlist-agent -t runs the probe too — that is where an operator
    asks whether a machine can enforce, and the only place an agent in
    detect-only would ever find out.

    Verified on Debian 12 (kernel 6.1, nft 1.0.6) and Debian 13 (kernel
    6.12, nft 1.1.3).

    Its log is where you will look for it

    Under systemd the log goes to the journal, which is right — but
    /var/log is the first place anyone looks after starting a security
    daemon, and fail2ban and CrowdSec both write a file there. So does this
    now: log_file: /var/log/shieldlist/agent.log, set in the shipped
    configuration, with the same lines as journalctl -u shieldlist-agent.

    No logrotate snippet ships with it. The agent identifies the file by
    (device, inode) like every other file it follows, notices when logrotate
    has renamed it away, and reopens.

    Pre-1.0, and honest about it

    • Releases are not signed. The checksums are served from the same
      host as the binaries: they prove the download was not corrupted, and
      nothing more. Signing is an open decision (SPEC §11).
    • The configuration format may still change.
    • Shipped rules cover sshd only. Rules for web, mail and panels are
      being written; docs/CONFIG.md and the wiki's Writing rules page
      are how to write your own meanwhile.
    • The central controller does not exist yet. Standalone is a mode, not a
      fork: the same files carry over when it does.

    Requires Linux with nftables. Builds for amd64, arm64, armv7 and
    riscv64; no cgo.

    Downloads