After Phase 7.D consolidation, the aggregator runs as user `secubox`.
secubox_core.user_store reads BOTH /etc/secubox/users.json and (as
fallback) /etc/secubox/auth.toml. The secubox-users postinst chowned
users.json to root:secubox 0640, but auth.toml had NEVER been touched
by any postinst — left at the package default 0600 root:root since the
secubox-auth 0.1.0 install in May 2026.
Live symptom : every admin login returned "Identifiants incorrects"
no matter what password was typed. The aggregator log showed :
user_store: fallback to auth.toml for get_user(admin)
user_store: auth.toml unreadable ([Errno 13] Permission denied:
'/etc/secubox/auth.toml')
The fallback path was the only reason auth.toml mattered at all (the
canonical store is users.json) — but with users.json reset to root:root
by an earlier intervention, the fallback became the active path and
its permission denial took down the entire auth chain.
This commit ensures both files are root:secubox 0640 after every
package install / upgrade.
Admin login UI was permanently flagging "Clock not synced" with the
warning "TOTP window widened to ±60s" even when the system clock was
correctly synced. Cause : ntp_health.probe() only knew how to read
chrony, but SecuBox boxes ship systemd-timesyncd by default ; chrony
is the operator-opt-in alternative. On a stock install the probe
caught FileNotFoundError, returned `synced: False`, and the
recommended_totp_window() widened to ±60s on EVERY login.
Added a chrony → timedatectl fallback chain :
1. Try `chronyc -n tracking` (chrony users).
2. Fall through to `timedatectl show` reading NTP +
NTPSynchronized properties (timesyncd users).
3. Only return the synthetic error if BOTH commands are missing.
The response now carries an extra `source` field
("chrony" / "timedatectl") so the operator can see which backend
answered the probe.
Continuation of the wg LXC migration. Resolved the unprivileged-LXC
bind-mount denial that blocked 2.3.1 by switching the wg target to
PRIVILEGED LXC (matches the existing mitmproxy WAF LXC convention),
and parameterised the launcher's listen-host so the LXC can bind
0.0.0.0 on its br-lxc interface.
Live cutover attempt then surfaced a deeper architectural issue :
mitmproxy transparent mode uses Linux's SO_ORIGINAL_DST socket
option to recover the original destination IP/port. SO_ORIGINAL_DST
is conntrack-backed, and conntrack is namespace-scoped. The DNAT
entries created by the host's `nft prerouting` chain are invisible
inside the LXC's netns. Every flow errors with :
Transparent mode failure: FileNotFoundError(2, 'No such file or
directory')
and gets RST'd after the TLS ClientHello.
Two viable architectures going forward, documented inline in the
LXC conf template :
1. Keep mitm-wg on the host (current path — the host service was
never disabled in source, only at runtime, and the rollback
re-enabled it ; the iPhone tunnel is back on the host mitm-wg).
2. Switch the LXC to `mitmproxy --mode wireguard` — mitmproxy
BECOMES the WG endpoint, no DNAT needed, conntrack stays in
the LXC netns. Requires migrating wg-peers.json to mitm's
server.conf format + moving UDP 51820 ingress (host nft DNAT
instead of wg-quick on host).
The LXC infrastructure (privileged template, env-driven launcher,
provisioner) stays shipped as scaffolding for the option-2 migration.
Live state of gk2 host : back on host mitm-wg, iPhone tunnel rx
84 MB / tx 733 MB, growing normally.
Two related improvements + one honest WIP note.
== NM connection id = village3b-r3-<last-octet> ==
The .nmconnection endpoint previously used `client_label` (the visitor
User-Agent) as the NetworkManager connection `id`. Operators ended up
with entries called "Mozilla/5.0 (X11; Linux x86_64; rv:151.0)
Gecko/20100101 Firefox/151.0" in the Network panel. Switched to
"village3b-r3-<last-octet>" : short, stable, distinct per peer (last
octet of the assigned 10.99.1.x).
== lxc-provision refinements ==
Multiple discoveries during the wg LXC migration attempt :
- lxc-create --template download instead of `debian` (the latter
requires PRIVILEGED LXC).
- LXC reads its runtime config from /data/lxc/<name>/config, NOT
/etc/lxc/<name>.conf. Write a clean, deduped config that
preserves the auto-generated idmap.
- Unprivileged LXCs do NOT auto-apply lxc.net.0.ipv4.address (the
kernel won't let the container set its own address without
CAP_NET_ADMIN on the netns). Push the IP + default route + DNS
+ systemd-networkd profile via lxc-attach post-start.
- Install python3 + curl + procps in the container only AFTER the
network + DNS are up — apt fails earlier.
== Known roadblock — not yet shipped live ==
The wg LXC bind-mount of /etc/secubox/toolbox/ca-wg fails with
"Permission denied" because the dir is 0750 root:secubox-toolbox and
the unprivileged LXC's mapped UID (100000) can't read it. Two
resolutions (documented inline in the conf template) :
1. chmod 0755 /etc/secubox/toolbox/ca-wg (relaxes CA file
visibility — OK for the public cert pem, not for the private
key inside it).
2. Drop lxc.idmap to switch to privileged mode (matches the
existing mitmproxy LXC convention).
The host secubox-toolbox-mitm-wg.service stays the active path
until the operator resolves this — no surprise cutover. The iPhone
tunnel is unaffected.
Three independent improvements landed together (small enough that
splitting would have churned the same files twice).
== secubox-waf 1.2.1 : LXC mitmproxy.service hygiene ==
Live diagnosis on gk2 found the LXC mitmproxy WAF process running
1 d 16 h with :
- 200 MB RSS, 9 threads
- 934 ESTAB TCP connections persisting
- 51 % response-error rate on the HAProxy backend
(eresp = 3.7 M of 7.2 M sessions)
- one core saturated (87 % of an ARM Cortex-A72)
Phase 6.P had shipped RuntimeMaxSec=21600 (6 h) drop-ins for the
host-side mitm-wg + mitm captive but never reached the LXC service.
Adding it now :
- RuntimeMaxSec=21600 : clean cycle every 6 h
- MemoryHigh=400M, MemoryMax=512M : same envelope as host units,
OOM-kill cleaner than swap thrash under attack traffic
Live restart applied — Until: 12:55:16 UTC.
== secubox-toolbox 2.3.0 : auto-detect /wg/onboard ==
End-user feedback : the current activation flow is fragmented across
"go to kbin.gk2.secubox.in, scan QR, install mobileconfig, trust CA,
…" with per-platform variations. Replaced by a single landing page
that detects the visitor's UA and renders the right one-click flow
first. Sections covered :
- iOS / iPadOS : WireGuard App Store link + QR + ca.mobileconfig
+ trust toggle reminder + battery-saver caveat.
- Android : Play Store link + QR + ca.pem + ca.cer + Settings
path + Samsung/Xiaomi user-CA hint.
- Linux : .nmconnection + .conf + ca.pem + nmcli reload one-liner
+ update-ca-certificates + Firefox-only note.
- macOS : Mac App Store link + .conf drag-onto-menu-bar +
Trousseau system import.
- Windows : official installer + add-tunnel-from-file + cert
store import + Firefox cert store caveat.
All artefacts stay at their existing paths ; the onboard page just
composes them. Hand a single URL to anyone — they get the right
flow first.
== secubox-toolbox 2.3.0 : mitm-wg LXC scaffolding ==
secubox-toolbox-lxc-provision now accepts \`transparent\` (default,
existing behaviour) or \`wg\` (new). \`wg\` provisions the
toolbox-mitm-wg LXC at 10.100.0.62 from the existing config +
service templates :
- Config template rewritten for TRANSPARENT mode on TCP 8081
(instead of mitmproxy --mode wireguard which would have required
a server.conf format peer migration). wg-quick stays on the
host ; only the mitmdump process moves to the LXC.
- Service template runs the SAME secubox-toolbox-mitm-wg-launch
script as the host service, bind-mounted in RO — single source
of truth for the addon chain + ignore_hosts compose logic.
- RuntimeMaxSec=21600 + memory caps match the WAF LXC's hygiene.
Cutover is OPT-IN — no automatic switchover in postinst. After a
successful provision the script prints :
1. lxc-attach -n toolbox-mitm-wg -- ss -tln | grep 8081
2. nft replace rule … dnat ip to 10.100.0.62:8081
3. systemctl disable --now secubox-toolbox-mitm-wg.service
4. update /etc/nftables.d/secubox-toolbox-wg.nft for boot-survival.
Operator picks the cutover moment so the live iPhone tunnel session
is never broken mid-flow.
WebUI showed firewall errors after Phase 7.D consolidation. Crowdsec
module /health returned `status: degraded` with `bouncers_ok: false,
bouncers_count: 0, nftables_ok: false` even though :
- `crowdsec.service` was active
- `crowdsec-firewall-bouncer.service` was active
- cscli + nft both worked perfectly from a root shell
- 2 bouncers were properly registered (cs-firewall-bouncer + secubox-waf)
- inet crowdsec / ip6 crowdsec6 tables were live
Three independent gaps in the new aggregator-mounted module context :
1. The /health endpoint runs `sudo cscli bouncers list -o json` and
`sudo nft list tables`, but the aggregator user (`secubox`) had
no sudoers entry → cscli refused with "permission denied" before
any actual query.
2. The aggregator unit had `NoNewPrivileges=true`. Even after
installing the right sudoers, the subprocess could not gain
privileges via the setuid sudo binary — systemd's flag blocks
all privilege escalation including narrowly-scoped sudoers.
3. The aggregator unit had `ProtectSystem=strict` which makes the
entire filesystem read-only (incl. /var). When cscli starts, it
touches perms on /var/lib/crowdsec/data/crowdsec.db. Under
ProtectSystem=strict that chmod() fails with EROFS and cscli
exits with rc=1 before printing the bouncer list — even when
sudo and the sudoers entry are both in order.
Fixes :
secubox-crowdsec :
- new sudoers.d/secubox-crowdsec drop-in shipped to
/etc/sudoers.d/. Two exact-match entries (no wildcards, no
shell, no flag escapes) :
secubox ALL=(root) NOPASSWD: /usr/bin/cscli bouncers list -o json
secubox ALL=(root) NOPASSWD: /usr/sbin/nft list tables
- debian/rules installs the drop-in to /etc/sudoers.d/ at
0440 root:root.
secubox-aggregator :
- systemd unit : NoNewPrivileges=false so sudo can escalate via
the sudoers drop-in. The sudoers entries themselves are the
defense in depth — they cap what the aggregator user can
escalate to two read-only listings.
- systemd unit : ReadWritePaths += /var/lib/crowdsec so cscli's
perm-touch on crowdsec.db succeeds ; += /etc/secubox for the
toolbox/dpi/etc modules that write config there.
Live verified : status=ok, healthy=true, bouncers_count=2,
nftables_crowdsec=true, nftables_crowdsec6=true.
Bumps :
secubox-aggregator 0.2.0 → 0.2.1
secubox-crowdsec 1.0.5 → 1.0.6
Cosmic / GNOME / KDE WireGuard import via the Network panel fails on
the standard wg-quick .conf with :
failed to connect to WireGuard VPN: le mot de passe pour
«wireguard.private-key» n'est pas indiqué dans le paramètre
«passwd-file» et nmcli ne peut le demander à l'utilisateur sans
l'option «--ask».
That's because NetworkManager classifies the private key as an
agent-owned secret (`private-key-flags=1`) by default — nmcli refuses
to bring the tunnel up without a TTY to prompt on. The iPhone WG app
handles the same .conf natively, hence the Linux-only failure.
New endpoint `GET /wg/profile/new.nmconnection` emits the same fresh
peer in NetworkManager keyfile format with `private-key-flags=0`
(system-owned, no prompt). Operator install :
sudo install -m 0600 village3b-toolbox.nmconnection \
/etc/NetworkManager/system-connections/
sudo nmcli connection reload
nmcli connection up <conn-id>
Interface name is slugified from the User-Agent label (max 15 chars,
[a-z0-9_-] only) so even arbitrary client headers produce a valid
netlink ifname.
Same peer registration path as /wg/profile/new — added to wg-peers.json
and the clients table at level=r3.
Twice in this session the aggregator went to 83 % CPU and free RAM
dropped to 137 MB after a single Firefox tab on the toolbox admin
dashboard hammered /api/v1/toolbox/admin/{health,config,clients,
metrics,filter-control/list} at ~530 req/sec. The page's
`setInterval(refreshAll, 10000)` is supposed to fire once per 10 s
(0.5 req/sec total), but stacked timers / multiple stale tabs / FF
session restore bring the rate orders of magnitude higher.
Adds a permanent nginx rate-limit so no client (current FF tab,
future similar bug) can put more than 20 req/sec sustained / 40
burst on the admin path. /api/v1/toolbox/ proper stays unlimited
so the captive portal probes from real users aren't capped.
Live verified : after applying, load avg 9.16 → 2.34, free RAM
137 MB → 353 MB, aggregator CPU 83 % → 18 %. The page itself keeps
working — no 429 floods, sustained traffic stays under burst.
After the same-origin tunnel-detection probe started working the
verification card flipped to "Tunnel R3 actif mais CA R3 NON trusté"
even when the mobileconfig was installed. Two bugs in the cert probe :
1. The probe loaded `https://www.gstatic.com/generate_204` as an Image.
`generate_204` returns HTTP 204 with no body — for an <Image>
element that's indistinguishable from a TLS error : both fire
onerror. Even a perfect CA install always reported "NON trusté".
2. gstatic.com sits in mitm-wg's ignore_hosts whitelist (Phase 6.N
bypasses Google/Apple/banks for cert-pinning reasons). So mitm-wg
passes that request through unmodified — the iPhone sees the real
Google cert, not ours. The probe wasn't actually testing CA trust
at all.
Fix : probe `https://duckduckgo.com/favicon.ico` via fetch(no-cors).
duckduckgo is NOT in the whitelist so mitm-wg actually intercepts and
terminates TLS with our R3 CA — the only thing that lets the request
succeed is the iPhone trusting that CA. fetch(no-cors) resolves on
TLS success and rejects on cert/network error, so HTTP status (which
we can't read in no-cors mode anyway) doesn't confuse the verdict.
The previous fix shipped XFF propagation in mitm-wg + XFF reading in
api.py, but the verification card STILL showed "Hors tunnel R3". Root
cause was in landing.html.j2 JS : the probe loaded
`http://10.99.0.1:8088/qr/splash.png` as an Image from an HTTPS page.
iOS Safari blocks mixed content, so the request never actually fired.
The previous fix made the request reachable but the browser was
suppressing it entirely.
New approach :
- New endpoint `GET /wg/r3-check` on the toolbox FastAPI returns
`{tunnel: bool, peer_ip: str|null}` based on `_client_ip(request)`
(which honours the X-R3-Peer / XFF headers set by inject_xff).
- landing.html.j2 JS replaces the broken cross-origin HTTP image
probe with a same-origin HTTPS fetch to `/wg/r3-check`.
Result : no mixed-content block ; iOS Safari fetches the probe ; the
toolbox reads the mitm-wg-set X-R3-Peer header and answers
`{"tunnel": true}` ; the card displays "Tunnel R3 actif".
Live verified : `curl -H "X-R3-Peer: 10.99.1.15"` returns
{"tunnel":true,"peer_ip":"10.99.1.15"} ; without the header returns
{"tunnel":false,"peer_ip":null}.
Screenshot from the iPhone showed mitm-wg's R3 banner header firing
correctly (SHA1 visible) but the verification card body reading
"Hors tunnel R3 — tu accèdes à kbin via Internet direct". The detection
chain was wrong in two places :
1. mitm-wg proxied upstream WITHOUT propagating the WG peer IP. By the
time the request landed at the toolbox FastAPI (after looping
wg-toolbox → mitm-wg → HAProxy → nginx) it carried the box's own IP
as the source.
2. The toolbox FastAPI's R3 check read request.client.host — which is
the proxy peer (unix socket) not the real client. Even when XFF was
set by nginx the code ignored it.
3. The verification page JS probes http://10.99.0.1:8088/qr/splash.png
to detect tunnel state. That address is on the wlan AP iface which
is linkdown in normal operation — packets from the WG subnet had no
reply path (rp_filter symmetry violation) and the image always
failed to load, so JS concluded "off tunnel" no matter what.
Fixes :
- New mitmproxy addon `inject_xff.py` loaded FIRST in mitm-wg launch.
Sets `X-Forwarded-For` carrying flow.client_conn.peername plus
sentinel headers `X-Through-R3-Tunnel: 1` and `X-R3-Peer: <ip>` so
downstream backends can detect tunnel state cheaply.
- `secubox_toolbox/api.py` : new `_client_ip(request)` helper that
prefers `X-R3-Peer` > leftmost XFF > raw socket peer. Splash and
`_resolve()` both use it now.
- `nftables.d/secubox-toolbox-wg.nft` : adds DNAT
iif wg-toolbox dst 10.99.0.1 tcp dport 8088 → 10.99.1.1:8088 so
the JS probe lands on the captive portal even when wlan is down.
- `systemd/secubox-toolbox.service.d/20-bind-all.conf` : binds the
captive portal uvicorn on 0.0.0.0:8088 (was 10.99.0.1:8088, only
reachable on wlan). The DNAT above + this bind together let WG
peers hit the verification probe successfully.
Live on gk2 : `http://10.99.1.1:8088/qr/splash.png` returns 200,
mitm-wg launches with inject_xff first in the addon chain, refresh of
kbin.gk2.secubox.in verification page now follows the "Tunnel R3 actif"
path.
Browsing still failed on the iPhone after the WG handshake/DNAT fix —
all DNS queries went to 10.99.0.1:53 (captive-AP IP from legacy peer
configs) and returned ICMP port unreachable because the wlan AP was
linkdown and unbound only listened on 127.0.0.1.
Three coordinated fixes :
1. nftables.d drop-in adds DNAT for 10.99.0.1:53 → 10.99.1.1:53 on
the wg-toolbox interface so legacy peer configs keep working.
2. New `unbound/99-secubox-wg.conf` shipped to
/etc/unbound/unbound.conf.d/ binds unbound to 10.99.1.1 AND
10.99.0.1 (ip-transparent so the down wlan address still works)
and allows the whole 10.99.0.0/16 to query.
3. `wg.py` now hands out DNS = 10.99.1.1 on new peer profiles so
future configs don't depend on the DNAT shim.
Live verified : tunnel rx 348 KB, tx 449 KB, iPhone Safari resolves
+ loads pages.
Two Phase 7 follow-ups exposed by last night's reboot :
== secubox-toolbox : WG boot survival ==
After reboot the iPhone reported "connected" but no traffic flowed and
no banner was injected. Three independent boot-time gaps :
1. /etc/nftables.conf has `udp dport 51820 accept` MISSING from inet
filter input — the default policy=drop silently swallowed every
WG handshake initiation that arrived from 192.168.1.254.
2. The `inet wg-toolbox` NAT/forward/prerouting table was created by
`secubox-toolbox-wg-provision` ONLY at package install time. After
reboot it was gone — even if handshake had completed, packets had
nowhere to MASQUERADE.
3. wg-peers.json holds 35 enrolled peers, but the wg-quick config has
no [Peer] blocks by design. Peers are added at runtime via `wg set`
by the FastAPI /wg/profile endpoints. Reboot wipes them all.
Fixes :
- New `nftables.d/secubox-toolbox-wg.nft` drop-in installed at
`/etc/nftables.d/` so `nftables.service` loads everything at boot :
the UDP 51820 input accept rule + the full `inet wg-toolbox` table
(postrouting masquerade, forward accept, QUIC drop, AND the
prerouting DNAT to mitm-wg :8081 that the provision script used to
leave commented out).
- The provision script now writes the drop-in to /etc/nftables.d/
on install/upgrade so it survives `nft flush ruleset`.
- DNAT-to-mitm-wg lines uncommented in the provision script too —
gk2 always runs with R3 interception enabled, so the "OPTIONAL"
framing was misleading.
- New `secubox-toolbox-wg-restore.service` (one-shot, After=wg-quick
@wg-toolbox) calls `secubox-toolbox-wg-restore` which walks
wg-peers.json and re-injects every peer via `wg set`. Boot-clean
enrolment recovery.
== secubox-waf : silenced-but-tracked attackers ==
Operator request : "attackers must be silenced but not lost track back".
`/api/v1/waf/bans` only shows ACTIVE CrowdSec decisions, so once a ban
expires the IP visually disappears from the dashboard even though we
still have its threat history on disk.
New `GET /bans/history?hours=N&limit=M` walks waf-threats.log (handles
both mitm WAF format `client_ip` and in-process WAF format `ip`),
aggregates per-IP within the window, and returns :
- count, severity_max, first_seen, last_seen, sample_path
- top_categories (up to 3)
- currently_banned : true if the IP is still in the active CrowdSec
decision set (cross-referenced with the warm /bans cache)
The waf/ dashboard gains a "Tracked Attackers" glass-card after the
existing Visitors/Vhosts cards, table of top 30 IPs in the 24 h
window. Each row shows the silenced (🚫 active ban) vs tracked (👁️
expired but logged) state alongside the attack signature.
Live numbers on gk2 after deploy : 117/117 modules mounted, history
endpoint returned 8 tracked IPs in 72 h, top offender 858 scan hits
on /server-status.php.
Three small bugs surfaced when auditing the aggregator load after
mass migration on gk2 :
magicmirror : debian/rules installed only api/main.py + an empty
__init__.py, so the running module's `from .routers import mmpm` had
no routers/ package to find. Was already broken under its own
per-module systemd unit. Now ships api/__init__.py from source +
api/routers/{__init__.py,mmpm.py}.
openclaw : postinst chown'd /var/lib/secubox/openclaw to root:root with
0750. The Phase 7.D aggregator runs as `secubox`, which couldn't
traverse the parent dir during the module's import-time mkdir of
scans/. Now chown to secubox:secubox (when the user exists), 0755 on
parent, 0750 on scans/ — same containment, traversable by the
aggregator.
secubox-aggregator-migrate : older packaging shipped some modules under
/usr/lib/secubox/secubox-<name>/ in addition to the canonical
/usr/lib/secubox/<name>/. On upgraded systems both coexist and the
discovery walk picked up duplicates that mounted as empty (no paths)
sub-apps under /api/v1/secubox-<name>/. Now skipped when an unprefixed
twin exists.
Live on gk2 : 117 mounted, 0 failed, 4.4 GiB free RAM. magicmirror and
openclaw both return HTTP 200 via nginx.
The first loader iteration loaded api/main.py as a flat module
sbx_mod_<name> with sys.path.insert of <name>/. Two patterns broke
across 11 of 121 modules :
1. Relative imports : modules that do `from .deps import X` failed
with "attempted relative import with no known parent package"
because the module had no __package__ set (just a flat name with
no dots).
2. Absolute api.X imports : a request that loaded auth first inserted
auth's `api`, `api.deps`, `api.webui_identity`, ... into sys.modules.
When haproxy then ran `from api.webui_identity import X`, Python
returned auth's cached `api.webui_identity` — usually missing the
name haproxy expected, so it raised ImportError. (Hit haproxy
because it loaded after auth alphabetically.)
The new loader :
- Builds a synthetic parent package `sbx_pkg_<name>` whose __path__
is `<name>/api`. Loads api/main.py AS `<pkg>.main` with
__package__ = pkg, so `from .deps import X` resolves through the
submodule_search_locations and finds the right deps.py.
- Drops every `api*` entry from sys.modules BEFORE each module load,
and again AFTER, so each module gets a clean api namespace and one
module's loaded api.X cannot shadow another module's api.X.
- Keeps the temporary sys.path entry so absolute `from api.X` still
works during exec_module (until cleanup).
Verified live on gk2 : 119 of 121 modules now mount (was 110). The
remaining 2 are real module bugs unrelated to the loader :
- magicmirror : ships api/main.py + api/__init__.py but no
api/routers/ — `from .routers import mmpm` always failed (this
module never started cleanly under its own systemd unit either)
- openclaw : /var/lib/secubox/openclaw is 0750 root:root so the
aggregator (running as `secubox`) can't traverse it during
api/main.py's directory-creation block. Fix is in the openclaw
postinst (chown to secubox or add a group), not here.
First version of secubox-aggregator-migrate only touched
/etc/nginx/secubox-routes.d/<name>.conf and used a simple socket-name
sed (replace `<name>.sock` with `aggregator.sock`). On gk2 this left
the navbar empty after migration because :
1. Most legacy routes live in /etc/nginx/secubox.d/<name>.conf — that
directory was untouched, so `/api/v1/<name>/` still targeted the
now-defunct per-module socket.
2. webui.conf carries inline `location /api/v1/<name>/` blocks for the
stats/auth modules (auth, certs, metrics, waf, soc, ...) plus a
catch-all `location /api/ { proxy_pass http://127.0.0.1:8001; }`
pointing at hub's old TCP port. Both untouched.
3. The naive sed of `:/` (strip-prefix rewrite) sent `/foo` upstream
instead of `/api/v1/<name>/foo`. The aggregator mounts each
sub-app under `/api/v1/<name>` and so cannot find anything when
the prefix is stripped — every route returned 404 or 502.
4. Hub used TCP 127.0.0.1:8001, not a unix socket, so the socket-name
sed never matched.
This rewrite :
- Walks BOTH /etc/nginx/secubox.d/ and /etc/nginx/secubox-routes.d/
- Collapses every `proxy_pass http://unix:/run/secubox/<name>.sock(:/...)?`
variant to `proxy_pass http://unix:.../aggregator.sock:/api/v1/<name>/`
so the full prefix arrives at the aggregator
- Handles the same rewrite inline inside sites-enabled/webui.conf
- Special-cases hub TCP 127.0.0.1:8001 → aggregator unix
- Rewrites the /api/ catch-all to forward the full URI unchanged
- Backs up the inline-block file under /root (NOT next to webui.conf,
because nginx auto-includes sites-enabled/* and would crash on a
duplicate default_server)
Verified live on gk2 : 106/110 mounted modules now return 200 via nginx,
free RAM 4.5 GiB.
Automates the Phase 7 cutover applied live on gk2 :
1. Walk /usr/lib/secubox/<name>/api/main.py to discover installable
modules and rewrite /etc/secubox/aggregator.toml accordingly
2. Restart secubox-aggregator and read /health to find out which
modules actually mounted (filters out modules with relative-import
bugs that cannot be loaded from a file path)
3. For each mounted module : back up + patch the matching
/etc/nginx/secubox-routes.d/<name>.conf so proxy_pass targets
/run/secubox/aggregator.sock instead of the now-defunct
/run/secubox/<name>.sock
4. nginx -t + reload
5. systemctl stop + disable secubox-<name>.service for migrated
modules (frees ~30 MB Python interpreter each)
6. Final report with rollback recipe
Idempotent : second run is a no-op (modules already pointed at
aggregator socket, services already disabled). Operators re-run it
after installing a new secubox-* package to fold it into the
aggregator.
Wires the script under /usr/sbin/ via debian/rules and prints a
one-line hint in postinst pointing operators at it.
Mass migration on gk2 was 110/121 modules → 1515 MB RAM freed
(3499→1984 MB uvicorn), 153 MB → 2.7 GiB free.
## Validation live on gk2
Per-module RSS sum (ai-insights + cookies + avatar) : 152 MB across 3 procs
Aggregator (same 3 modules mounted) : 57 MB in 1 proc, 2 threads
SAVINGS : 95 MB (-62%) on this 3-module pilot.
Extrapolation : ~100 modules × 30 MB savings = ~3 GB recoverable
across the full SecuBox-Deb install. Migration is incremental
(per-module opt-in via /etc/secubox/aggregator.toml).
## Architecture
### packages/secubox-aggregator/aggregator/main.py
Master FastAPI app that mounts each listed module's FastAPI as a sub-app
under `/api/v1/<name>`. Modules are loaded by absolute path from
`/usr/lib/secubox/<name>/api/main.py` using `importlib.util.spec_from_file_location`
since they ship as cwd-relative `api.main` rather than as Python packages.
Isolation : each `app.mount()` wrapped in try/except. Load failures
recorded in `_LOAD_ERRORS` and surfaced at `/health`. One broken module
doesn't poison startup.
### Config — /etc/secubox/aggregator.toml
```toml
modules = [
"ai-insights",
"cookies",
"avatar",
]
```
Default file shipped via postinst. Operator edits to opt in additional
modules then restarts secubox-aggregator. Per-module systemd units stay
running until disabled explicitly — letting operator rollback by config-only.
### systemd unit — single uvicorn
`/lib/systemd/system/secubox-aggregator.service` runs :
uvicorn aggregator.main:app
--uds /run/secubox/aggregator.sock
--workers 1 --backlog 400 --limit-concurrency 200
--log-level warning
Environment : PYTHONOPTIMIZE=1, PYTHONDONTWRITEBYTECODE=1, PYTHONUNBUFFERED=1.
MemoryMax=1G, MemoryHigh=768M as a safety net (typical full mount stays
under 300 MB).
## Smoke test live (gk2)
mounted: ['ai-insights', 'cookies', 'avatar']
routes: 6
/health endpoint : HTTP 200, JSON status
/api/v1/ai-insights/health : HTTP 200
/api/v1/cookies/health : HTTP 200
/api/v1/avatar/health : HTTP 200
## What's NOT in this commit
* Nginx proxy_pass updates (per-module routes still point to old sockets).
Operator does the cutover after enabling each module in aggregator.toml,
then disables the per-module systemd unit and updates nginx upstream
for /api/v1/<module>/.
* Mass migration of all ~100 modules. Pilot validates the pattern ;
next sprints expand the config to cover more modules in batches.
## Files
packages/secubox-aggregator/aggregator/__init__.py (NEW)
packages/secubox-aggregator/aggregator/main.py (NEW, 165 lines)
packages/secubox-aggregator/debian/control (NEW)
packages/secubox-aggregator/debian/changelog (NEW)
packages/secubox-aggregator/debian/rules (NEW)
packages/secubox-aggregator/debian/postinst (NEW)
packages/secubox-aggregator/debian/aggregator.toml.example (NEW)
packages/secubox-aggregator/systemd/secubox-aggregator.service (NEW)
sysctl + THP + network buffer tuning. Load 8.08 → 4.75 in 5 min.
ESTAB connections from sustained 942+ → 81. Run queue 12 → 5.
Live applied + persisted via /etc/sysctl.d/95-secubox-perf.conf and
/etc/tmpfiles.d/secubox-thp.conf. Backported into secubox-mitmproxy
package install rules.
User : 'baisse rapide des performances du tunnel' followed by 'd\'abord les
performances'. Phase 6.N (dynamic whitelist) closed the retry-storm cause.
Phase 6.O squeezes more perf from the kernel/network stack.
## Live verification (before → after on gk2)
load avg : 8.08 → 4.75 (-41% in ~5 min)
run queue : 12 → 5 (-58%)
CPU idle : 5% → 21% (4x more breathing room)
swap activity: 0 (already healthy, just better protection now)
## What changed
### /etc/sysctl.d/95-secubox-perf.conf (NEW)
Memory :
vm.swappiness = 10 (default 60 → keep working set in RAM)
vm.vfs_cache_pressure = 50 (retain dentry/inode cache longer)
vm.dirty_background_ratio = 5 (earlier background writeback)
vm.dirty_ratio = 15 (lower sync flush threshold — less latency spike)
Network :
net.core.rmem_max / wmem_max = 16 MiB (was 200 KiB — WG burst friendly)
net.core.rmem_default / wmem_default = 1 MiB
net.core.netdev_max_backlog = 8000 (8x default — scheduling jitter)
TCP :
net.ipv4.tcp_fastopen = 3 (client + server : -1 RTT on repeat conn)
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1 (~500 TIME_WAIT idle currently)
### /etc/tmpfiles.d/secubox-thp.conf (NEW)
Switches Transparent Huge Pages from 'always' to 'madvise'. On Armada
under memory pressure, 'always' causes compaction stalls (multi-ms
pauses in scheduling). 'madvise' lets apps opt-in (none here do),
which avoids the stalls completely.
### Not applied (kernel constraint)
BBR congestion control + fq qdisc were planned but the Marvell custom
kernel (6.12.85) wasn't built with CONFIG_TCP_CONG_BBR / NET_SCH_FQ.
Would need a kernel rebuild for those — kept in comment in the file.
Cost of staying on cubic + pfifo_fast : ~3-5% less throughput.
## Files
packages/secubox-mitmproxy/sysctl/95-secubox-perf.conf (NEW)
packages/secubox-mitmproxy/tmpfiles.d/secubox-thp.conf (NEW)
packages/secubox-mitmproxy/debian/rules (install wires)
## Diagnostics that mattered
vmstat 1 3 : si/so = 0 (no active swap thrashing — relief)
ps state : 12 procs in run queue (scheduling bottleneck)
cpu freq : already at max (1.4 GHz, schedutil governor OK)
cpu gov : schedutil (modern, no need to change)
THP : 'always' (changed to madvise — gain on this hw)
## Not pursued
- eBPF/XDP : Marvell DSA driver limits XDP to generic mode (no native
hook in driver). Gain would be ~2-3x, not 200x. Postponed.
- LXC pruning : memory available is fine (2.98 GB avail). No need
to stop yacy/horde/rustdesk.
User : 'dynamicly whitelist for necessary and required host... pin cert ?'.
YES — the WAF now auto-detects cert-pinning and self-heals.
## How it works
1. mitm-wg hooks tls_failed_clienthello (fires on cert-pin reject)
2. cert_pin_detect addon counts failures per SNI in a sliding 5-min window
3. When count >= 3 within the window, the SNI is normalized to a regex
pattern (drop subdomain, escape dots) and appended to
/var/lib/secubox/toolbox/mitm-bypass-dynamic.conf
4. systemd path watcher detects the file change, waits 10s for the dust
to settle (avoid restart storm if multiple SNI fail at once), then
restarts secubox-toolbox-mitm-wg
5. The launcher merges static + dynamic bypass lists, dedups, and applies
the combined regex to mitm --set ignore_hosts
6. Subsequent flows to the learned pinned host pass through opaque
Result : the first user to hit a new pinned host pays a 3-failure tax.
Every other user passes silently. No manual maintenance.
## Anti-foot-shooting guards
- NEVER_AUTO regex blocks own infrastructure (secubox/gk2/cybermind/maegia)
and RFC1918 subnets from being learned (a misbehaving local service
shouldn't be silently bypassed).
- _VALID_SNI rejects junk SNIs (IPs, single-label, malformed).
- The pattern normalization drops only the FIRST subdomain so a single
entry like (.+\.)?googleapis\.com covers all *-pa.googleapis.com,
but doesn't accidentally whitelist all of .com.
- Already-added patterns aren't re-appended (in-memory dedupe + load
existing file on addon startup).
## Files
packages/secubox-toolbox/mitmproxy_addons/cert_pin_detect.py (NEW)
packages/secubox-toolbox/sbin/secubox-toolbox-mitm-wg-launch (modified)
packages/secubox-toolbox/debian/secubox-toolbox-mitm-wg-dynreload.path (NEW)
packages/secubox-toolbox/debian/secubox-toolbox-mitm-wg-dynreload.service (NEW)
## Live verification
After deploy on gk2 :
ignore_hosts regex applied (835 chars, 37 patterns, static+dynamic)
secubox-toolbox-mitm-wg-dynreload.path : active (waiting)
Tunable via env :
SECUBOX_PIN_THRESHOLD (default 3)
SECUBOX_PIN_WINDOW (default 300s)
User : 'baisse rapide des performances du tunnel'.
## Diagnosis
mitm-wg log scan (30 min) on Android 10.99.1.15 + iPhone 10.99.1.18 :
98 for people-pa.googleapis.com (already whitelisted, OK)
37 for playatoms-pa.googleapis.com (whitelisted, OK)
32 for quake-pa.googleapis.com (whitelisted, OK)
25 for tracking.eu.miui.com ← NEW : Xiaomi telemetry
24 for oauthaccountmanager.googleapis.com (whitelisted, OK)
17 for securitydomain-pa.googleapis.com (whitelisted, OK)
8 for tg.socdm.com ← NEW : TreasureData
8 for iphone-ld.apple.com ← NEW : Apple Location
8 for b-graph.facebook.com (whitelisted, OK)
6 for ssbsync.smartadserver.com ← NEW : SmartAd
4 for user-sync.fwmrm.net ← NEW : FreeWheel
4 for pixel.rubiconproject.com ← NEW : Rubicon
3 for js-sec.indexww.com ← NEW : Index Exchange
3 for c1.adform.net ← NEW : AdForm
3 for web-banner.ads.aps.amazon-adsystem.com ← NEW
3 for bi-events-collector.spot.im ← NEW : OpenWeb
3 for images-static.trustpilot.com ← NEW
3 for wct-2.com ← NEW
Each failed handshake triggers Chrome retry (5-10 attempts/s) → 116
ESTAB connections per Android peer (vs ~5 normal) → mitm worker queue
saturation → tunnel slow.
## Fix
13 new patterns added to _MITM_BYPASS_DEFAULT_ENTRIES :
miui.com, xiaomi.com, socdm.com, adform.net, rubiconproject.com,
smartadserver.com, indexww.com, spot.im, fwmrm.net,
amazon-adsystem.com, wct-[0-9]+.com, trustpilot.com,
iphone-ld.apple.com
These pass through opaque (TLS passthrough, no mitm decrypt). The banner
and report counter still see them via the tracker_count body scan (Phase 6.G).
## Live verification
Before whitelist :
fails/min : 100+
Android cnx : 116 ESTAB
load avg : climbing
After whitelist (60s after restart) :
fails last 60s : 4
Android cnx : 1 ESTAB
load avg : 7.31 (was 8.5, stable now)
Whitelist count : 24 → 37 patterns total.
Country flags were silently broken since Phase 3 — banner displayed as
'FR' (two boxed letters) instead of 🇫🇷 on most browsers.
## Root cause
Flag emojis are Unicode regional indicator PAIRS :
🇫🇷 = U+1F1EB (REG IND F) + U+1F1F7 (REG IND R)
The browser font engine joins the two indicators into a single flag glyph
only when the codepoints arrive as actual Unicode characters. NCR-encoded
entity references (\🇫\🇷) do NOT trigger the join — they
render as two letter boxes.
Banner's _ncr() function was encoding the flag like any other emoji, so
the right-side text showed 'FR' on every page.
## Fix
Skip _ncr() for ctx['flag'] specifically. Output raw UTF-8. The banner's
JS payload gets emitted via json.dumps which escapes to UTF-16 surrogate
pairs (\ud83c\uddeb\ud83c\uddf7) — the browser's JS engine decodes
these to the codepoints, and the font join logic kicks in.
Verified by inspecting the rendered snippet :
Before : '🇫🇷' (renders 'FR')
After : '\ud83c\uddeb\ud83c\uddf7' (renders 🇫🇷)
## Trade-off
Pages with legacy iso-8859-1 charset that don't include the flag bytes
in JS context : may show garbage. Vanishingly rare in 2026 web. Worth it
for the visual gain on the 99.9% of UTF-8 pages.
Status icon (🔍) + app emojis (📰📺 ☁ ...) keep NCR encoding because
they're single codepoints and NCR works fine for them.
User : 'chrome arrive a bypasser le mitm toolbox ?' YES, via 3 vectors :
## Vector 1 — QUIC over UDP 443 (silent bypass)
Chrome heavily prefers HTTP/3. mitm-wg only DNATs TCP 443/80 to
10.99.1.1:8081. UDP 443 (QUIC) flows straight through wg-toolbox → lan0,
invisible to mitm (0 mentions in 5-min log scan).
Fix : wg-provision adds an nft drop rule on UDP 443 from wg-toolbox iface.
Chrome's QUIC handshake fails → transparent fallback to HTTP/2 over TCP →
gets DNAT'd to mitm → analyzed.
nft add rule inet wg-toolbox forward iif wg-toolbox udp dport 443 drop
Cost : ~50ms extra on first connect to a new host. Worth it.
## Vector 2 — Google Play Services cert pinning
Android Network Security Config makes Play Services pin Google's CA.
User-installed CAs are rejected. mitm-wg log full of :
Client TLS handshake failed... does not trust the proxy's certificate
for oauthaccountmanager.googleapis.com / people-pa.googleapis.com /
playatoms-pa.googleapis.com / quake-pa.googleapis.com
Each failed handshake = retry storm + broken Chrome sync.
Fix : extend _MITM_BYPASS_DEFAULT_ENTRIES with Google + Meta API patterns :
googleapis.com, google-analytics.com, gstatic.com,
googleusercontent.com, android.clients.google.com,
facebook.com, facebook.net, fbcdn.net, instagram.com
These pass through opaque (mitm doesn't decrypt, doesn't try to forge cert).
Same approach already used for Signal / WhatsApp / banks.
## Vector 3 — IP-direct fetches (no SNI)
CDN backends served by IP (Akamai 104.116.96.165, Google 172.217.22.46
etc.). mitm intercepts these at the IP/port layer (DNAT match) but can't
classify by hostname. NOT a bypass per se — traffic IS intercepted, just
labeled by IP rather than domain. Acceptable.
## Live verification
Before fix : 20+ handshake failures / 10s on Chrome Android
After fix : 0 handshake failures / 5s on Chrome Android (10.99.1.15)
nft counter on UDP 443 drop : ready (0 packets yet, as
Chrome will only attempt UDP 443 once per host then cache
that fallback is needed)
## Whitelist count growth
Phase 6.F : 15 patterns (Signal/WA/TG/Apple/banks)
Phase 6.K : 24 patterns (+Google/Meta) — current
Live regex applied : 'ignore_hosts regex applied (475 chars, 20 patterns)'
(20 because some default entries are comments after my edit — the actual
runtime count is 24 ; cosmetic mismatch, the regex still includes all
non-comment lines via grep -v '^[[:space:]]*#' in mitm-wg-launch)
