Shieldlist/shieldlist-controller
Watch
1
0
Fork
You've already forked shieldlist-controller
 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions
2 releases 2 tags
  • v0.0.9 80e8d6303f

    v0.0.9 Stable

    Robin released this 2026-08-26 21:34:58 +02:00 | 1 commits to main since this release

    Shieldlist Controller 0.0.9

    What it is

    The controller is the brain of a Shieldlist fleet: your servers report what they see, it keeps the history, decides what each machine runs, and gives you one console to run all of it. You host it — there is no cloud service in the middle, and every protected machine keeps defending itself if the controller is down.

    What you gain over 0.0.1

    One console for the whole fleet instead of one shell per server: what each machine decides, what it is counting right now, and why an address is blocked.

    A rule language you can read and edit from a screen, with a diff before anything is pushed, and a test mode that decides exactly like production and bans nobody.

    Lists: block an autonomous system, a country, a set of prefixes or a public blocklist as one firewall set, carved around your exemptions.

    Abuse reporting done for you, repeat offenders included, and a public page that takes unblock requests off your inbox.

    A store that answers in milliseconds on a year of history, and a controller that starts serving in about a second.

    Everything that changed

    The fleet

    • Enrolment with a join token; the tree is assembled per machine and per agent version, only the rules that differ are rewritten, and the result is validated by the agent's own loader before it travels.
    • Packs are the unit of assignment: a pack is a list of rules, a server names its packs, nothing else reaches it.
    • Sharing by pack: machines running a common pack accept each other's decisions; a ban lifted at its origin takes its copies down; a machine joining the sharing is served the still-active backlog.
    • A ban by hand on the fleet is ONE order — every machine applies it at its next sync, every machine enrolled later at its first, with what is left of the duration; a fleet lift cancels it everywhere.
    • A ban given from a detection carries that detection's log lines, so the manual sanction holds its evidence on every machine and reports with those facts.
    • Re-ask a machine for its history (POST /v1/agents/{host}/resync); a re-enrolled machine's fresh numbering no longer overwrites its past.
    • Per-server display names; a declared but not yet enrolled server is marked as such.
    • The controller reports on its agents' behalf, or leaves it to each machine — per fleet and per server.
    • Clock skew measured on every sync, with warnings per agent.
    • shieldlist-controller status, token, agents, why, unban, api-token, account.

    The rule language, and editing it

    • The configuration made simple: one rule, one file, five blocks — what it recognises, from how much, what it bans with, who it notifies, how it is reported.
    • Levels scale the numbers: instant, strict, standard, lenient.
    • Editors in the console for rules, patterns, policies, report presets and packs, with a pattern tester, a diff preview, and "edit as file" for the raw YAML.
    • Every edit is a file edit, validated whole and reverted if it does not load.
    • Rules can count by origin (countries, asns, and their exceptions) and by what was aimed at (targets, except_targets).
    • A {target} segment in a log path names the site every line of that file is about; a fixed file can declare its target.
    • Servers → Logs shows a machine's declared log inputs.
    • Prod and test are two axes: a rule or a server in test decides like production and bans nothing; the console says «would ban» where it means it.

    The feed and the tickets

    • Sanctions and hits as tickets, not tables: a live feed filterable by state, server, rule, country, network, target, username and dates, refreshed by the second, pausable, with the requests behind each decision on demand.
    • Hits — the offenders being counted but not convicted — with their progress toward each threshold, their log lines, and the ones that came to nothing kept an hour and shown greyed.
    • One kind at a time or all at once: All, Bans, Hits, Detection-only (test), Expired, Exempted.
    • Every ban says how long: in force with what is left, ticking to the second; over, with its length; lifted; a detection with what it would have been.
    • Cards laid out by areas — flag and address left, the moment on top, the facts beside — with the network name, the target, the username, and the AbuseIPDB link.
    • A returning offender says so; a new sanction flashes as an overlay that fades over the ticket's own colours.
    • Copy buttons on the address and on the log lines; log boxes keep where you scrolled and open at their end.
    • Evidence lines carry what the parser captured, by name — statistics on any field, a filter on any value.

    The map

    • A world map of the sanctions under the current filters, drawn in-house from Natural Earth outlines.
    • Attacks ripple out of their country, then out of their city when a city database is loaded; your own servers sit on it at their own place.
    • Rays from the attacker's city to the server that was hit, carrying the volume — one head per departure, in file, landing on the server's point.
    • A dot for a country too small to see; countries with no outline get a spot.
    • Bans or hits, your choice; arcs can be turned off; full screen; the height follows the chart's.

    The chart

    • Bans, detections and hits stacked over the window you choose, from one-second steps to a year.
    • The line of bans in force — distinct addresses banned at each step — over the whole history.
    • Sixty round bars per window, a soft log scale with a ruler, hover counts, and a click that narrows the whole page to that moment, seconds included.
    • An incremental tail: the live edge is merged in rather than reloading the series.

    The side panel

    • Servers, countries, networks, targets, rules and usernames tallied under the current filters, sanctions and hits side by side.
    • Pins keep what you watch first; an eye sets an entry aside as a lens, not a setting; multi-select with ctrl/⌘ and with checkboxes on a phone.
    • Sortable by bans or hits, compact thousands, and the panel slides over the page on narrow screens.

    Windows and dates

    • The window is a wheel — from the last minute to everything on record — turned by mouse, wheel or finger, and it drives the feed, the map, the chart and the panel at once.
    • Custom dates with times and presets (today, 7 days, this month) that keep running to now.
    • Saved views per account: a set of filters under a name, following you between browsers.

    Lists, exemptions, published blocklists

    • Lists are named sets of addresses — an AS, a country, prefixes, or a published URL — with a role (blocked or exempted) and a scope (the fleet, a pack's servers, or named servers).
    • A resolved list travels as data and becomes one nftables set on each covered machine, with its own counter, carved around the exemptions — no ban orders, no history buried under thousands of rows.
    • Published blocklists can be followed by URL, fetched and bounded, with a catalogue to start from (Romain Marcoux's malicious-ip list, MIT, credited).
    • Exemption sources for the crawlers everyone keeps: Google, Bing, Apple, DuckDuckGo, Cloudflare, Fastly, Ahrefs, Stripe, UptimeRobot, Pingdom, OpenAI, Jetpack, and the private ranges — each on or off, with its own scope and refresh.
    • Networks and countries picked by name or number from the ASN database; any origin dialog can block or exempt what it shows.
    • Blocked and Exempted as two tabs, with exact figures, sortable, dated to the minute; "Banned by hand" grouped one row per address.
    • Check addresses: paste up to a thousand and see, for each, the sanctions, every list that blocks or exempts it with the entry that covers it, and its network — the answer shield why gives on a machine, for a batch.
    • The address dialog says which list covers an address, so an address inside a blocked prefix is no longer reported as "nothing on record".

    The network view

    • What each prefix of a network did over a window — bans and hits apart, share in addresses — and the prefix an address belongs to.
    • Ban the address, its prefix, or the whole network, from the rows that say which one attacks.

    Abuse reporting and reputation

    • The controller runs the same one-chance sender on its agents' behalf, eight in flight, resuming behind its last report after a restart.
    • A report's comment ends with the episode's minute: without it, a provider answers 200 and stores nothing for a repeat, and every recidivist was lost.
    • A return after a report is reported again once the renew interval has passed, with the latest hit as its time.
    • Settings → AbuseIPDB: the key, the daily plan, the allowance left, who reports for whom, and a switch that stops reporting without losing the key.
    • An address's reputation on demand; your servers' own addresses checked daily.
    • A manual ban is reported with the facts of the detection it was given from.

    The public unblock page

    • An optional page where a blocked visitor lifts their own address, asks for another, or asks to be whitelisted; requests are approved or refused from a Requests menu with a badge.
    • The visitor's address found on both families through your own resolvers, their reverse name, network and country, and — if you allow it — the lines that earned the block.
    • Branding, contact, language (en/fr), built-in captcha or Turnstile; refusals logged so a rule can act on abuse of the page.
    • Mail: one SMTP relay configured once, each account choosing what it is notified about, with a test mail.

    Where addresses are from

    • An in-house MaxMind-DB reader and the free DB-IP Lite databases (CC BY 4.0): country, AS, and optionally city, fetched monthly or pointed at your own files.
    • The databases are served to the agents, so rules and exemptions by country or AS run on the machines themselves.
    • Origins resolved at ingest and backfilled; they drive the filters, the facets, the map and the search.

    Statistics, history, maintenance

    • A statistics page: countries, networks, rules, servers and targets over 7, 30, 90 or 365 days or any range, exportable as CSV, with a clicked bar filtering the feed to that window.
    • Below-threshold suspects kept as episodes with their raw lines, so a past window shows who was being counted then — not who is being counted now.
    • Retention set per kind from the console (evidence, samples, hits, hit lines, sanctions), swept daily; Maintenance says what the store holds and can flush or compact it.
    • Counters maintained by database triggers at ingestion rather than scanned on demand.

    Accounts and the panel itself

    • Sign in with a name and a password, a one-time code if you want one, or an API token; sessions slide with activity and their length is a setting.
    • Rename yourself, change your password and e-mail, turn two-factor on or off from the account dialog.
    • Theme toggle in the topbar (auto, dark, light) remembered with the account, alongside the time zone and the display choices.
    • The ban colour means one thing everywhere — the rail, the counters, the map, the chart, the headers — and hits keep their own.
    • A ✕ empties every search and filter field; ⓘ explains a setting where it is; hints wrap and stay inside the box.
    • The console notices when the controller under it was updated and offers to reload; a daily check tells you when a newer release exists.
    • It works on a phone: stacked map and chart, menus as centred sheets, dialogs inside the screen, a film over the page while the panel is open.

    Operations

    • Debian packages for amd64, arm64, armhf and riscv64, from our APT repository; a service user and a starter tree created on install; a hardened systemd unit; its own log directory and logrotate; man pages; third-party licence notices.
    • GET /v1/vitals, Settings → Vitals and shieldlist-controller status read the same pulse: CPU, memory, sync cost, queue, errors — and the agents' own errors are one click away.
    • SHIELDLIST_DEBUG=1 writes a debug log with each sync's cost; SHIELDLIST_PPROF=<addr> opens Go's profiler on a loopback address.
    • Boot no longer waits on the heavy loads: the geo databases and the counters load in the background, each saying when it started, how long it took, or what is degraded if it failed.

    Performance

    • Write transactions queued in Go, WAL with synchronous=NORMAL, page cache and mmap sized from the machine — a large batch no longer loses the lock to short writers until it gives up and is resent every second.
    • Queries pinned to their indexes on every periodic path: the sender's cursor, the changed-rows scan, facets grouped so the start index is kept, field facets behind a covering index.
    • Bulk work in one transaction, never row by row on the big table.
    • What the console polls is memoised; one round at a time; heavy views at most every fifteen seconds.

    Security

    • Two read-only audit passes: constant-time sign-in, configuration files written 0600, a bounded join token, the SSRF guard extended to carrier-NAT space, a captcha whose digits are not in the markup, documented guidance for running behind a reverse proxy.
    • The console renders text attackers wrote: everything escaped, a strict CSP, cookies HttpOnly and SameSite=Strict, and a write needs a header a cross-site form cannot send.
    • Accounts are never locked out by failed sign-ins — a lockout is a way to lock you out. Failures are logged so a rule can ban whoever is trying.

    Early days

    Formats can still move between releases, an upgrade can ask for a re-enrolment, and one controller serves one fleet on SQLite. Start on test servers, keep what protects you today, and tell us what breaks. Support is best-effort; known issues are on the forge.

    Credits

    Geolocation: DB-IP Lite (CC BY 4.0). Map outlines: Natural Earth. Blocklist catalogue: Romain Marcoux's malicious-ip list (MIT). Exemption sources: the ranges published by each operator. Go modules: THIRD_PARTY_LICENSES. The full list is under Settings → About.

    Downloads
  • v0.0.1 38a67021d4

    v0.0.1 Pre-release

    Robin released this 2026-08-16 03:50:37 +02:00 | 320 commits to main since this release

    First prerelease of the controller: the control plane for a fleet of Shieldlist agents. Much of it will move; the protocol is versioned and additive from here on.

    What it does. Assembles each machine's configuration from a shared tree of parsers, rules, policies, presets and exemptions plus one declaration per machine (level, state, who may lift, who reports, fleet switches, per-rule overrides), validates it with the agent's own loader, and pushes it to enrolled agents the moment it is loaded — over a long-lived, agent-initiated connection, with polling as the fallback. Records every sanction the agents decide, redistributes them across the fleet according to each machine's switches, and carries the operator's lifts back to the machines within seconds.

    Operator's faces. A JSON API on its own listener with scoped tokens; a console served by the same binary — fleet, machine settings and overrides, bans with lift and exempt from any row, rules, address lookup, exemptions and published sources, audit, people and tokens, configuration; local accounts with a password and an optional one-time code; and shield as a client of the API (-controller URL -token T) with the same verbs on the whole fleet. Every write through the API or the console is a write to the configuration files, reloaded whole or reverted with the loader's errors as the answer.

    Published exemption sources. Googlebot, Google's special crawlers, Bingbot, Applebot and Cloudflare, fetched on command or on a schedule when named in controller.yaml, materialised as generated files under exemptions.d, refused when a list looks like nothing a crawler publishes.

    Transport. HTTP with protobuf bodies (proto/shieldlist/v1/agent.proto), TLS self-signed by default with the fingerprint carried by the join token and pinned by the agent, or the operator's certificate, or plain HTTP behind a proxy that terminates TLS. Both sides exchange versions on every request; a refusal names both and says which side is behind. Requires an agent that speaks the protocol — the next agent release; agents 0.0.3 and older cannot enrol.

    Not yet. Log ingestion and the event store, controller-placed rules and fleet replay, controller-side abuse reporting, owner scoping, packages. State is SQLite; the schema is versioned and migrated in place.

    Assets: static binaries for linux amd64, arm64, armv7 and riscv64, the systemd unit, SHA256SUMS. Docs: docs/SPEC.md, docs/CONFIG.md ("The controller"), packaging/README.md.

    Downloads