A present-but-stale localStorage Bearer was used exclusively, ignoring a valid
secubox_session cookie, so every webui 401'd into a /login.html loop. Now both
sources are tried (Bearer first, then cookie); 401 only if neither validates.
Strictly more permissive — no regression for working Bearer clients.
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
30s was too short for slow module operations behind the reverse proxy — a
podcaster audiobook upload (server-side ZIP extraction of ~120MB) exceeded it,
so nginx returned 502 on an upload that actually succeeded. 300s matches the
per-vhost timeouts already used elsewhere (lyrion, yacy). Applies to every
module served through secubox.d.
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
* feat(sbxmitm): capture HLS segments as kind=segment (Phase 2, ref #812)
Add segmentKind(path, ctype) to classify HLS/DASH segment chunks (.ts,
.m4s, .m4v paths or video/mp2t, video/iso.segment content types), and
give it precedence over mediaKind in Capture so a .ts served as
video/mp2t is tagged kind="segment" instead of "video". Whole-file
.mp4/video/* stays "video" and .m3u8/.mpd manifests stay "manifest".
IsMedia now also matches segmentKind so ambiguous-ctype segments (e.g.
.m4s over application/octet-stream) are still captured.
* feat(core): HLS media-playlist parser + segment-URI rewriter (Phase 2, ref #812)
* feat(dpi): HLS manifest replay — rewrite segment URIs to captured segments (Phase 2, ref #812)
media_replay() branches on record kind: a captured HLS manifest is parsed
with secubox_core.hls and its segment URIs are rewritten to the matching
captured kind=="segment" records' replay URLs (read-time join by absolute
URL, scoped to the same mac_hash+host). Master/ABR and AES-128 encrypted
playlists are detected and returned raw with X-SecuBox-Media:
unsupported-variant instead of a broken rewrite. Every other kind keeps the
unchanged Phase-1 FileResponse path; any manifest parse/read error falls
back to the raw FileResponse (never a 500).
* fix(dpi): log HLS manifest/segment/rewrite truncation + pin manifest replay fail-safe (Phase 2 review, ref #812)
* feat(dpi): Media tab — hide HLS segments, manifest playback/download + unsupported-variant note (Phase 2, ref #812)
* test(#812): HLS reassembly end-to-end + changelog 0.1.28 (Phase 2)
Cross-layer integration test seeding a temp media-buffer exactly as the Go
tee + janitor produce on disk (metatag JSONL + per-session object files),
then driving the real DPI media_replay handler: manifest rewrite joins each
segment URI to its captured segment by resolved URL, every rewritten URL's
id replays to byte-identical captured bytes, and a janitor-style append-only
eviction flip leaves the evicted segment unrewritten (410 on direct replay)
while its siblings keep replaying. This is the whole-branch-review-flagged
integration coverage Phase 1 was missing.
Bumps secubox-toolbox-ng changelog to 0.1.28-1~bookworm1 with the Phase 2
HLS reassembly summary (segment capture, manifest rewrite,
unsupported-variant detection for master/ABR + AES + DASH).
* chore(dpi): changelog 1.2.0 — media buffer API + HLS reassembly (#812 Phase 1+2)
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* docs(media-buffer): design spec — dashcam capture + replay + metatag lifecycle (ref #812)
* docs(media-buffer): phase-1 implementation plan (capture+janitor+API+DPI tab) (ref #812)
* feat(sbxmitm): media buffer store + metatag writer (ref #812)
* fix(sbxmitm): media-buffer review — 4096 atomic-append cap + *string buffer_ref (ref #812)
Task 1 review (Important): mediaBufferLineMax 8192→4096 to match mediacatch.go's
PIPE_BUF atomic-append guarantee across 4 workers; BufferRef string→*string so the
janitor (Task 3) can flip it to JSON null on eviction.
* feat(sbxmitm): tee media up/download bodies into the buffer (ref #812)
Wire the R4 media buffer tees into mitmPipeline: the download hook wraps
resp.Body and the upload path wraps req.Body via teeReadCloser, capturing
whole-file media into the rolling /data buffer. Add the mbuf field + flags
(--media-buffer default OFF, --media-buffer-root, --media-buffer-per-object).
Design upgrade over the plan sketch: ObjectWriter.Write is now NON-BLOCKING.
Instead of io.TeeReader driving a synchronous os.File.Write on the client read
path (which would add latency / could stall large media on ARM eMMC), Write
copies each chunk into a bounded channel and returns len(p),nil immediately; a
background goroutine started at Capture drains the channel to disk. A full
channel drops the chunk and flags truncated — never blocks, never errors the
proxied flow (design 4.2/7). Close signals the drainer, waits for it to flush
queued chunks + close the file, then appends the metatag; it is idempotent,
deadlock-free (sole closer of ch) and safe if Write is never called or races
Close.
* fix(sbxmitm): media download tee — defer closure so w.Close runs (ref #812)
defer resp.Body.Close() bound the ORIGINAL upstream body at the defer
statement, before the media-buffer tee reassigned resp.Body — so the
tee's Close() (and w.Close, which flushes the sink/metatag/drain
goroutine) never ran on any download. Wrap the close in a closure so
it reads resp.Body at return time instead.
Adds TestTeeDeferOrderingMatchesMainGo, which replicates main.go's
exact defer-then-reassign ordering and fails against the old plain
defer form (verified manually, then restored the fix).
Also: drop the dead write-only ObjectWriter.dropped field (truncated
already covers it), document Capture's reserved contentLen param, and
note that ObjectWriter.Close is allowed to block (post-stream
finalisation), unlike Write.
* feat(sbxmitm): media buffer janitor — time + LRU eviction, metatag survives (ref #812)
SweepOnce evicts a session's bytes (RemoveAll, idempotent) once
first_ts+retentionSecs elapses, then falls back to oldest-first LRU
eviction if the buffer root (media bytes only, log excluded) still
exceeds sizeCeilBytes. Eviction never rewrites the append-only
metatag log in place — it appends a corrected line with the same id
(expired:true, buffer_ref:null); readers dedup by id, last line wins,
so the flip is durable and safe across the 4 concurrent sbxmitm
worker processes each running their own RunJanitor (idempotent
RemoveAll + harmless duplicate flip lines).
RunJanitor ticks SweepOnce every 30s (test-overridable period) until
its context is cancelled; wired at sbxmitm startup as a
process-lifetime goroutine when --media-buffer is enabled.
NewMediaBuffer's signature is unchanged; retentionSecs (1200),
sizeCeilBytes (24 GiB) and nowFn now default inside the constructor.
* feat(core): media-buffer metatag reader (dedup by id, fail-empty) (ref #812)
* test(core): media-buffer reader — tail-read truncation + malformed-field regressions (ref #812)
* chore(#812): keep SDD scratch reports out of the feature diff (restore to master)
* feat(dpi): media-buffer list/replay/thumb API (admin/owner, audited, 410-on-evict, traversal-safe) (ref #812)
* harden(dpi): media API — \Z-anchored id regex + role short-circuit security note (ref #812)
* feat(sbxmitm): media-buffer retention/size-ceil flags + /data tmpfiles + opt-in worker env toggle (ref #812)
- main.go: --media-buffer-retention / --media-buffer-size-ceil flags, applied
onto the already-constructed mbuf's unexported retentionSecs/sizeCeilBytes
fields (NewMediaBuffer signature unchanged).
- tmpfiles/zz-secubox-media-buffer.conf: pre-creates /data/secubox (0755) and
/data/secubox/media-buffer (0750 secubox:secubox), installed via the
package's existing manual tmpfiles.d mechanism in debian/rules.
- secubox-toolbox-ng-worker@.service: EnvironmentFile=-/etc/secubox/toolbox-ng.env
+ trailing $SBX_MEDIA_BUFFER_ARGS on ExecStart so an operator can opt in
(SBX_MEDIA_BUFFER_ARGS=--media-buffer) without editing the shipped unit;
default stays OFF. Added ReadWritePaths=/data/secubox/media-buffer.
* fix(sbxmitm): media-buffer dir owned by writer (secubox-toolbox) + setgid group secubox; changelog 0.1.27 (ref #812)
Task 6 review Critical: buffer dir was 0750 secubox:secubox but sbxmitm workers run
as secubox-toolbox (not in group secubox) → EACCES on every capture, silently no-op'd
(errors swallowed by design). Fix: 2750 secubox-toolbox:secubox — writer owns, dpi
reader (secubox) is group, setgid propagates group secubox to created objects so the
reader can read them. Important: add debian/changelog 0.1.27 stanza (version bump).
* feat(dpi): Media tab — live capture gallery with replay/download, escaped fields (ref #812)
Adds a Media Buffer card to the DPI dashboard (packages/secubox-dpi/www/dpi/index.html)
that lists sbxmitm's live media-buffer captures via GET /api/v1/dpi/media/buffer.
- Auth: dedicated apiAuthed() reads the canonical `sbx_token` localStorage key
and sends it as a Bearer token (media/buffer requires JWT, unlike the rest
of this page's endpoints which currently ride the SSO-lite session cookie);
401 redirects to /login.html, matching sibling dashboards (e.g. crowdsec).
- Security: host/url/mac_hash are attacker-influenceable (captured live
traffic). Every card is built via createElement + textContent (or the
`.title` IDL property, never HTML-parsed) — no innerHTML string
concatenation of these fields — so a captured value like
`<img onerror=...>` renders as inert text. Replay/download URLs are built
with encodeURIComponent(id) even though the id is hex-validated server-side.
- UI: kind emoji (video/audio/manifest/file), short device id (shortDev,
reused from the existing #785 helpers), ⬆/⬇ direction, human size
(formatBytes) and age (agoStr) per card; expired captures grey out and show
"métatag seul (expiré)" with no action links.
- Refresh: wired into the existing refreshAll() plus its own ~15s interval.
Verified with a Playwright harness stubbing fetch (incl. an XSS payload in
host/url) against a local copy of the page: no console/page errors, XSS did
not fire, no <img>/<script> element materialized from the captured strings,
expired-card gating confirmed.
* fix(sbxmitm): media buffer — upload tee needs a real body, download tee 200-only (whole-branch review); drop dead item.ts (ref #812)
* docs(media-buffer): Phase 2 plan — HLS reassembly via read-time URL join (ref #812)
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* docs(spec): rapport kbin fidèle + media types + WebUI DPI (ref #785)
* docs(plan): rapport kbin media types implementation plan (ref #785)
* feat(core): shared media-catch JSONL aggregator (ref #785)
* fix(core): guard media-catch aggregate against malformed fields + full SPDX header (ref #785)
* feat(toolbox): _media_stats + wire media_exfil into report routes (ref #785)
* feat(toolbox): PDF DPI-exfil/overall donut-grids + media-type block (ref #785)
Replaces the text-bullet DPI/EXFIL section (per-device + overall) with
_pdf_donut_grid 4-donut grids and adds a new "TYPES DE MEDIAS CAPTES"
section (kinds/content-types donut grid + top-hosts emoji table), for
parity with the HTML report. Also guards the PDF footer's italic
set_font against DejaVu-Oblique not being registered (fonts-dejavu-core
ships Regular+Bold only) — a pre-existing latent crash on any full
render_pdf() call, uncovered because no prior test exercised the whole
function end-to-end.
* fix(toolbox): per-grid captions so network/media donuts aren't mislabeled as device stats (ref #785)
* feat(toolbox): media-type cards in report web page (me + overall) (ref #785)
* fix(toolbox): close media card divs so Overall tab + footer aren't nested in DPI pane (ref #785)
* feat(dpi): WebUI card — services by category (bytes) (ref #785)
* feat(dpi): /media_types endpoint (MIME media-catch, fail-empty) (ref #785)
* feat(dpi): WebUI card — MIME media types + refreshAll wiring (ref #785)
* chore: gitignore SDD run scratch + drop stray task report (ref #785)
* perf(core): bounded tail-read for media-catch + import-in-try for uniform fail-empty (ref #785)
* fix(dpi): backport board-live async control endpoints (drift reconciliation, ref #785)
The board ran a manually-deployed main.py with status/restart/start/stop/logs/
interface_list/tc_status/remove_mirred as 'async def' — never committed to git.
Backported so the next .deb install doesn't revert them. Signature-only change,
verified identical to the live-validated deployed file.
* fix(toolbox): PDF render must not wedge the event loop (#785 incident)
Heavier #785 report PDFs (more matplotlib donut-grids, ~9s render) + the WAF
504-page auto-retry storm pegged the single uvicorn worker and 504'd the whole
toolbox vhost. Fixes:
- run render off the event loop (threadpool) so HTML report / landing stay live
- serialize renders through one asyncio lock (pyplot is not thread-safe; also
bounds CPU so retries can't render in parallel)
- short per-device PDF cache with a double-checked lock: a retry storm now
triggers exactly ONE render, all other hits return cached bytes instantly
- persistent MPLCONFIGDIR (systemd drop-in + postinst dir) so matplotlib builds
its font cache once instead of on every process start
Validated live on gk2: cold render caches, 8-request retry storm all served
from cache (~0.1s), /landing 0.012s throughout.
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
#65's symptom (new modules' /api/v1/<m>/ return 404 because their nginx routes
aren't loaded) is already solved in the deployed webui.conf via
'include /etc/nginx/secubox-routes.d/*.conf' — every module package drops a
location-only snippet there at install. The repo template common/nginx/webui.conf
was stale (hardcoded core blocks only, no include), which is misleading. Add the
active include so the template matches production; keep the hardcoded
crowdsec/waf/system blocks (those core packages ship only the legacy secubox.d/
snippet, so no duplicate-location).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## Goal
Make the cabine HONEST : tell the user not just what we inspected, but what
we deliberately BYPASSED and WHY (cert-pinning vendor, banking policy, E2E
opaque by design). Score each destination on passive observable signals.
This is the foundation for #493 (later) full reverse-WAF enforcement —
architecture intentionally aligned so action='block'/'redirect'/'strip' can
be added without refactoring.
## New shared code (common/secubox_core/)
- whitelist.py : YAML loader with hot-reload. fnmatch glob patterns.
Reads /usr/share/secubox/toolbox/whitelist-baseline.yaml +
/etc/secubox/toolbox/whitelist.yaml (operator override). Returns
(match dict | None) for any host.
- classifiers/security_quality.py : grade A+/A/B/C/D/F per destination.
Two modes :
passive : TLS version + JA4 + SNI + ALPN + is_e2e_messaging
active : adds HSTS + CSP + X-Content-Type-Options + cookies attrs
+ mixed-content detection
Returns {grade, score, mode, reasons[]} with each reason's weight.
## Curated whitelist baseline (47 patterns)
- Apple ecosystem (push, iCloud, CDN, mzstatic, cloudkit, smoot)
- Google/Android (googleapis, gstatic, fcm, android, play)
- Messaging E2E (Signal, Threema, SimpleX, Matrix.org, Proton, Tutanota)
- Banking FR (BanquePostale, SocGen, BNP, CA, CM, LCL, Boursorama, Revolut)
- Government FR (service-public, impots.gouv, ameli, france-identite, francetravail, caf)
- Health (Doctolib, Qare, MonEspaceSante)
- Privacy tools (DDG, Qwant, Startpage, Proton, Tutanota, Mullvad, Tailscale)
- Tor Project
- OS updates (Windows Update, Microsoft, Debian, Ubuntu)
## local_store.py wiring
- New _analysis_status(host, sni, decrypted) returns (status, reason).
Order : decrypted > whitelist > E2E regex > pinned-failed fallback.
- DPI events now carry analysis_status + analysis_reason fields.
- New tls_failed_clienthello hook captures the SNI of bypassed/pinned
flows so the report can be honest about them.
## Aggregator (secubox_toolbox/api.py)
- New _build_transparency() builds the dict :
breakdown : raw counts {inspected: N, bypassed-whitelist: N, ...}
breakdown_pct : same as percentages
total_events : sum
per_host[] : worst-first grade table for the report UI
whitelist_stats : how many patterns are loaded
- Exposed under 'transparency' key in /report and /report/me/html.
- Backfill : legacy events (no analysis_status) get post-hoc tagged via
whitelist match check so older sessions also show meaningful breakdowns.
## Reports
- HTML live (report-live.html.j2) : new card 'INSPECTION : CE QUI A ETE
REGARDE' with breakdown bar + 'QUALITE SECURITE PAR DESTINATION'
sortable table. Inserts before the Support card.
- PDF (reports.py) : matching sections rendered ASCII-safe via _ascii_safe.
Top 10 worst-quality hosts.
## Verified on gk2 (2026-06-05)
- whitelist loaded : 47 patterns, baseline OK
- aggregator returns transparency.has_transparency=True
- per_host populated with quality grades
- PDF still generates clean
## Out of scope (deferred to #493 enforcement phase)
- Action types : block/redirect/strip-cookies/downgrade
- nftables sinkhole for known-bad destinations
- dnsmasq hijack for whitelisted FQDN auto-direction
- PhishTank + threat-intel feeds active blocking
- Per-flow TLS version + headers capture (currently we only grade default
'inspected' as C because no TLS metadata in the event yet — Phase 4 will
extend local_store.response() to capture response headers + TLS info).
## Closes#492 (Phase 3 transparency foundation)
## Goal
After Phase 2a+ (#487), mitm addons (dpi/cookies/avatar/soc/ja4) were firing
fire-and-forget POSTs to /run/secubox/<module>.sock/<endpoint> but the
receiving modules had no matching endpoints -> 404 silent drop, no
cross-module data flow.
## Architecture
1. **Shared ingest helper** (common/secubox_core/mitm_ingest.py)
mount_ingest_routes(app, endpoint_path, db_path, kind, enrich_hook=None)
registers POST {endpoint_path} + GET /mitm-events + GET /mitm-events/stats
on the module's FastAPI app. Events persist in a dedicated SQLite db
(mitm-ingest.db) with auto-rotating 48h retention.
The enrich_hook parameter (optional callable) lets each module enrich the
event BEFORE persistence (e.g. secubox-dpi will run nDPI lookup, secubox-
threat-analyst will compute JA4 hash in Phase 2c).
2. **5 modules wired** (mount_ingest_routes call right after FastAPI init)
- secubox-dpi : POST /classify + /var/lib/secubox/dpi/mitm-ingest.db
- secubox-cookies : POST /inject + /var/lib/secubox/cookies/mitm-ingest.db
- secubox-avatar : POST /fingerprint + /var/lib/secubox/avatar/mitm-ingest.db
- secubox-soc : POST /event + /var/lib/secubox/soc/mitm-ingest.db
- secubox-threat-analyst : POST /ja4 + /var/lib/secubox/threat-analyst/mitm-ingest.db
3. **Mitm addons refactored** (5x: dpi, cookies, avatar, soc_relay, ja4)
_common.py gains hash_mac() + mac_hash_of() using shared salt
(/etc/secubox/secrets/toolbox-mac-salt, same algorithm as
secubox_toolbox.mac.hash_mac : HMAC-SHA256 + daily rotation, 16-char hex).
Addons now send client_mac_hash instead of raw client_mac. Raw MAC NEVER
leaves the toolbox process. Privacy + CSPN compliance.
cookies.py extended : extract cookie NAMES only (Set-Cookie + Cookie
headers, max 32 char per name, cap 30 set + 50 sent). Values never
touch the wire/SQLite.
4. **Aggregator** (secubox_toolbox/api.py)
New _pull_mitm_module_events(mac_hash) iterates the 5 sockets,
GETs /mitm-events?mac_hash=X&limit=20 via http.client UDSConnection,
returns {module: {count, sample[]}}. Added to _aggregate_session result
as 'mitm_modules' key -> visible in /report/me JSON.
## E2E test (gk2 2026-06-05)
- 5 socket POSTs OK : dpi/cookies/avatar/soc/threat-analyst all 200
- 5 socket GETs OK : returns ingested events filtered by mac_hash
- _pull_mitm_module_events(test_hash) returns count + sample for all 5
- mitmproxy addon load : no errors after pyc cache clear
## Permissions
NO-OP : all secubox-* services run with UMask=0000 + User=secubox -> sockets
default to 0666 srw-rw-rw-. secubox-toolbox user can write/read all 5
sockets directly.
## Out-of-scope (follow-ups)
- Phase 2c : each receiving module implements its actual enrich_hook
(nDPI lookup for dpi, JA4 hash computation for threat-analyst, etc.)
- Phase 2c : aggregator merges receiving-module enrichment INTO the report
display (top_dpi shows nDPI app names instead of host pattern matching)
## Closes#488 (Phase 2b wire-up)
SecuBox stays source of truth. user_store gains set_password() (the single
password write path = capture point) + list_users(); a new host CLI
secubox-user-sync sets the canonical argon2 hash then pushes the same plaintext
(via SECUBOX_USER_PASSWORD env/stdin, never argv) into each app exposing a
`<app>ctl user-provision` verb. nextcloudctl + photoprismctl gain that verb,
each creating a per-user photo folder (/data/shared/photos/<user>) and a
user-scoped library; admin stays unrestricted (management), others scoped.
gk2 + admin are always provisioned (seed). Tests for set_password included.
Fixes "re-authentication everywhere despite being logged into the webui".
- secubox-core auth.py: require_jwt now accepts the parent-domain session
cookie (secubox_session) as well as the Bearer token (additive — existing
Bearer clients unchanged). Exposes set_session_cookie() publicly.
- secubox-auth main.py: emit the session cookie on every login-success path
(direct, post-MFA, post-TOTP-enrollment) so one SecuBox login (incl. TOTP)
drops a cookie that authenticates all module APIs without re-prompting.
Cookie is HttpOnly + Secure + SameSite=Lax (CSRF-mitigated). Set
api.sso_cookie_domain=".gk2.secubox.in" for cross-subdomain SSO.
Authelia vhost cutover (lyrion/zigbee) is a separate later step.
Foundation to replace Authelia with SecuBox's own users (reuses the argon2
user_store; those hashes can't feed nginx Basic auth, but a verify endpoint
can validate them). Source-only — no live cutover yet.
- auth.py: /login now also sets an HttpOnly, parent-domain-scoped session
cookie (secubox_session = the JWT). New GET /auth/verify validates the
cookie (or Bearer) via the same checks as require_jwt and echoes
Remote-User/Remote-Groups for the proxied app (200 allow / 401 deny).
New POST /auth/logout clears the cookie.
- secubox.conf.example: api.sso_cookie_domain (parent domain, e.g.
".gk2.secubox.in"; empty = host-only).
Remaining for #400 (follow-up, careful + board-specific): confirm which
socket serves /auth/verify given secubox-auth overrides /login; ship the
nginx auth_request snippet with the real SecuBox login-page redirect; live
cutover one vhost first (lyrion/zigbee) then all; stop+disable Authelia.
* feat(mitmproxy): cookie_audit addon — Set-Cookie ledger for RGPD (ref #156)
New mitmproxy addon that hooks the response flow and appends every Set-Cookie
header to /var/log/secubox/cookie-audit/server.jsonl. Cookie values are
sha256-hashed at the addon — the raw value never leaves the process.
Companion to the upcoming browser-side cookie-inventory.js + the
CookieAuditAggregator that will reconcile both streams for RGPD/ePrivacy
audit on operator-owned vhosts.
Also adds the implementation plan under docs/superpowers/plans/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(hub): cookie-inventory.js browser snapshotter (ref #156)
Vanilla JS module that snapshots document.cookie at DOMContentLoaded, +2s,
and on visibilitychange. Names + sha256(value) are POSTed to
/api/v1/cookie-audit/ingest with credentials:'omit'. Hard-capped at 8
snapshots per page to avoid amplification.
Loaded into every HTML response via the WAF banner injection (next commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(waf): inject cookie-inventory.js alongside health banner (ref #156)
Extends the existing CDN banner injection block to also append a script tag
loading shared/cookie-inventory.js. Adds two new CDN-config keys:
- cookie_inventory_url (default: admin.gk2.secubox.in/shared/...)
- cookie_audit_ingest_url (default: admin.gk2.secubox.in/api/v1/...)
Both health banner and cookie inventory now load from the same injected
guard block, so a single </body> replacement covers both.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): CookieAuditAggregator + Classifier (ref #156)
Aggregator class mirroring CertStatusAggregator: tails the mitmproxy JSONL
Set-Cookie ledger and the browser-snapshot ingest dir, then reconciles per
(vhost, cookie-name).
Per-cookie source verdict:
* http — server only
* js — browser only -> RGPD violation unless strictly_necessary
* both — server + browser
Classifier loads regex rules from config, checks categories in order
(strictly_necessary > functional > analytics > marketing), first match wins.
Default cache file: /var/cache/secubox/metrics/cookie-audit.json.
9 unit tests covering parsing, reconciliation, persistence and malformed
input.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): wire /api/v1/cookie-audit/{ingest,report,summary} (ref #156)
Imports + initializes CookieAuditAggregator, schedules it in the FastAPI
lifespan alongside the other aggregators, opens up CORS to POST (was GET
only) for the ingest endpoint, and adds three routes:
POST /api/v1/cookie-audit/ingest — browser snapshot ingest (rate-bounded,
credentials: omit, returns refused if
audit disabled in config)
GET /api/v1/cookie-audit/report?host=… — per-vhost or global report
GET /api/v1/cookie-audit/summary — global rollup
Hard caps: 200 cookies/snapshot, 128B names, 128B hashes, 512B UA.
All 34 existing metrics tests + 9 new cookie-audit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(core): get_cookie_audit_config + RGPD classifier baseline (ref #156)
New config helper get_cookie_audit_config() that returns the merged
[cookie_audit] section with built-in RGPD/CNIL classifier patterns.
Categories (eval order — first match wins):
strictly_necessary PHPSESSID, csrftoken, cart, remember_token, ...
functional lang, locale, theme, cookie_consent, euconsent, ...
analytics _ga*, _gid, _gat*, _pk_*, _hjid, _clck, _matomo*, ...
marketing _fbp, _fbc, __utm*, _gcl_*, _uet*, IDE, MUID, NID
Operator patterns from [cookie_audit.classifier] are MERGED on top of the
baseline by default. Set classifier_override = true to replace it entirely
(opt-out for sites that need a different taxonomy).
secubox.conf.example documents the section with commented-out examples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: cookie-audit API + addon (ref #156)
- secubox-metrics README: new "Cookie Audit" section documenting the
three endpoints, the source verdict semantics (http/js/both), and the
rgpd_violation rule.
- secubox-mitmproxy README: new "Addons" section listing both
secubox_waf and cookie_audit + the dual -s mitmdump invocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(claude): record cookie-audit pipeline in HISTORY+WIP (ref #156)
HISTORY 2026-05-16 entry captures scope (server ledger + browser snapshot
reconciler), the 8 implementation tasks landed in the worktree, and the
two follow-ups left (AppArmor + logrotate).
WIP gets a top entry pointing at the worktree/branch with the test
totals and "awaiting validation, no PR opened" status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): add Cookie Audit live tile + summary URL injection (ref #156)
Adds a "CookieAudit" section to the health-banner live panel alongside
VisitorOrigin / LiveHosts / CertStatus. It polls /api/v1/cookie-audit/summary
every 30s and renders:
- host count
- RGPD verdict row (red ⚠ when violation_count > 0, green ✓ otherwise)
- per-category breakdown: strict / func / analytics / marketing
- unclassified row only shown when > 0
The WAF banner-injection block now also sets
window.SECUBOX_COOKIE_AUDIT_SUMMARY (new cookie_audit_summary_url config
key) so the banner can hit the cross-origin summary endpoint when the
host page is on a non-admin vhost.
Banner bumped to v1.4.0. New section hides itself silently when the
aggregator is disabled or empty — zero visual impact on existing
deployments.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(metrics): Make sibling imports work under uvicorn (api.main:app)
The aggregator imports `from visitor_origin import VisitorOriginAggregator`
worked in pytest (conftest puts api/ on sys.path) but crashed at module load
under `uvicorn api.main:app` with ModuleNotFoundError. The service entered
a restart loop; the new live-panel endpoints were unreachable.
Fix: insert os.path.dirname(__file__) at the top of sys.path before the
sibling imports, mirroring the test conftest.
Verified live on gk2: secubox-metrics now starts cleanly, all three new
endpoints (visitor-origin / live-hosts / cert-status) respond 200 with the
documented disabled payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(nginx): Strip upstream CORS headers in /api/v1/* snippet (ref #92)
Several FastAPI services in this repo (including secubox-metrics, with the
new live-panel endpoints from #98) ship their own CORSMiddleware emitting
Access-Control-Allow-Origin: *. The nginx snippet for /api/v1/* also adds
the same header. Browsers receiving both fold them into
"Access-Control-Allow-Origin: *, *" and reject all cross-origin requests,
which broke health-banner.js v1.3.0 when injected on any vhost other than
admin.<host>.secubox.in.
Fix: declare nginx as the sole CORS authority by stripping the upstream
copies with proxy_hide_header before nginx's own add_header fires. The
project standard already had nginx own CORS at the edge; this just plugs
the leak introduced when individual services started emitting their own.
Verified live on gk2: single Access-Control-Allow-Origin: * header now
returned by /api/v1/metrics/* through HAProxy + mitmproxy + nginx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(spec): Health banner live panel design (ref #92)
Three public banner sections sharing one polling/CORS pipeline:
- VisitorOrigin: nft set seen_src + GeoLite2-ASN.mmdb, threshold-gated
rollup, raw IPs discarded before persistence
- LiveHosts: HAProxy admin socket, 60 x 1-min ring buffer over req_tot
deltas, hostname-heuristic frontend filter
- CertStatus: scan /etc/letsencrypt/live + cryptography parse, classify
valid / expiring_soon / expiring_critical / expired
Each section fails independently; section hidden on enabled=false,
empty entries, or fetch error. All three endpoints are unauthenticated,
CORS-open, Cache-Control max-age=300.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(plan): Health banner live panel implementation plan (ref #92)
14-task TDD plan: tests scaffold -> config helpers -> three aggregators
(visitor-origin / live-hosts / cert-status) -> FastAPI lifespan wiring ->
nftables ruleset -> geoipupdate timer -> debian packaging -> banner v1.3.0
-> README + tracking docs -> full-suite verification + PR.
Also reconciles spec with codebase conventions: service user is 'secubox'
(not 'secubox-metrics'); config lives in /etc/secubox/secubox.conf, not a
separate metrics.toml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(metrics): Scaffold pytest layout for new aggregators (ref #92)
Adds tests/__init__.py and conftest.py that wire packages/secubox-metrics/api
and the repo-wide common/ onto sys.path so individual aggregator modules can
be imported in isolation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(core): Add visitor_origin / live_hosts / cert_status config helpers (ref #92)
Three new section helpers in secubox_core.config that merge defaults with
operator-supplied TOML overrides. Each section defaults to enabled=false so
the live-panel aggregators stay quiet on systems that haven't opted in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(metrics): Use tmp_path fixture + drop unused import (ref #92)
Switches the config-helper tests from a hard-coded /tmp path to pytest's
tmp_path fixture, matching the pattern in packages/secubox-haproxy/tests/.
Removes the now-unused 'from unittest.mock import patch' line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): VisitorOrigin aggregator (ref #92)
Pure-Python aggregator that polls the nft seen_src set, resolves ASNs via
GeoLite2 mmdb, and emits a threshold-gated top-N rollup. Private/loopback IPs
are skipped at lookup time; raw IPs never leave the function scope; the
threshold gate runs before persistence so the cache file never contains
attributable counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(metrics): VisitorOrigin code-review followups (ref #92)
- mmdb auto-reopen on mtime change (Important): close + reopen when stat
reports a different mtime; previously _mmdb_mtime was dead state.
- _read_nft_set defensive parse (Important): guard against {elem: str} shapes
to prevent a latent TypeError.
- current() returns a defensive copy (Minor): no more by-reference leak of
internal state.
- Drop unused ip_address import and unused monkeypatch parameter (Minor).
- Tighten tiebreak test to assert ASN order, not just count order (Minor).
- Add test for mtime-based reopen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(metrics): Pin VisitorOrigin error-path behaviour (ref #92)
Regression tests for refresh_once: disabled config, missing mmdb, and nft
subprocess failure must all yield a non-throwing degraded payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): LiveHosts aggregator with 60x1-min ring buffer (ref #92)
Reads HAProxy admin socket via raw AF_UNIX, parses 'show stat' CSV, filters
internal frontends (leading underscore or no dot), ring-buffers per-frontend
deltas over 60 minutes, and emits a top-N hostname rollup. Counter-reset
detection (cur < prev) yields a fresh-baseline bucket instead of a negative
delta. current() returns a defensive copy mirroring the VisitorOrigin fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(metrics): Pin LiveHosts CSV parser + missing-socket paths (ref #92)
Tests the show-stat CSV parser against the real HAProxy column order and
asserts that an absent admin socket returns a degraded payload rather than
raising.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): CertStatus aggregator (ref #92)
Scans /etc/letsencrypt/live for cert.pem files, classifies each by days
remaining (valid / expiring_soon / expiring_critical / expired) using the
operator's warn_days/critical_days thresholds, and emits a summary + soonest
next-renewal host. A single corrupt PEM never kills the scan. Days remaining
computed with math.ceil so a cert expiring in 2.99d reports 3d, consistent
with certbot/renewal tooling expectations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(metrics): correct expired boundary when using math.ceil (ref #92)
math.ceil maps any cert that expired within the last 24 h to days=0,
which the previous `days < 0` guard treated as expiring_critical instead
of expired. Changing the guard to `days <= 0` closes the gap: with ceil,
days=0 means actual remaining time is in (-86400, 0] — i.e. already
past or exactly at expiry — so classifying it as expired is correct.
All four existing tests continue to pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(metrics): Wire three live-panel aggregators into FastAPI lifespan (ref #92)
Adds the three asyncio background tasks under a single lifespan and exposes
their current() payloads on /api/v1/metrics/{visitor-origin,live-hosts,cert-status}
with a 5-min Cache-Control. Endpoints stay unauthenticated by design — the
aggregators only emit threshold-gated, hostname-only data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(metrics): Await cancelled lifespan tasks + tidy import order (ref #92)
- Important: lifespan finally now awaits gather(*tasks, return_exceptions=True)
after cancel(), so blocking subprocess/socket I/O in aggregator refreshes
doesn't race uvicorn's shutdown timeout.
- Minor: move 'from contextlib import asynccontextmanager' into the stdlib
import group at the top of the file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): Ship nftables ingress tap for visitor-origin (ref #92)
Private inet secubox_metrics table with a timeout'd src-IP set, hooked from
prerouting at priority -300 so additions happen before secubox-firewall's
filter chain decides whether to drop the packet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(metrics): Weekly GeoLite2 ASN refresh timer (ref #92)
Conditional on /etc/secubox/secrets/maxmind.conf existing, so the unit is a
silent no-op on installs that haven't supplied a license key. RandomizedDelay
spreads load when many boxes deploy together.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build(metrics): Package live-panel deps, nft ruleset, and geoipupdate timer (ref #92)
- control: add python3-maxminddb, python3-cryptography, geoipupdate, nftables
- rules: install nftables/ and systemd/ assets
- postinst: secubox -> haproxy group, cache + secrets + GeoIP dirs,
nftables reload, timer enable
- service: ReadWritePaths gains /var/cache/secubox
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(metrics): Run geoipupdate as secubox + restart svc on upgrade (ref #92)
- secubox-geoipupdate.service: User/Group=secubox so .mmdb files inherit the
ownership the metrics service expects when reading them. Previously the
unit ran as root and created root-owned files that secubox-metrics could
not open.
- postinst: switch 'systemctl start' to 'systemctl restart' so the secubox
user's new haproxy-group membership is picked up by an already-running
service after an upgrade.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): v1.3.0 live panel — visitor origin, live hosts, cert status (ref #92)
Three independent fetch loops on a shared 30s cadence, three DOM sections,
per-section hide on enabled=false / empty / fetch error. Uses existing
design tokens (gold/cyan/matrix-green) so no new CSS variables are added.
A failing section never affects the others.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(banner): Correct banner element id lookup in live-panel (ref #92)
The new live-panel sectionContainer() helper looked up
getElementById('sbx-health-banner'), but the actual banner element is
created with id='health-banner'. The mismatch made banner null, so the
three live-panel sections were silently never appended to the DOM.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: Session 160 — Health Banner Live Panel (ref #92)
README documents the three new endpoints + config blocks. HISTORY / WIP /
MIGRATION-MAP entries describe the feature, the spec/plan paths, and the
session's outcome.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(metrics): Address whole-branch review findings (ref #92)
- Wrap _read_nft_set / _read_haproxy_stats in asyncio.to_thread so blocking
I/O (subprocess up to 5s, AF_UNIX recv up to 2s) no longer stalls the
event loop on every refresh tick.
- Replace falsy current() guard with explicit _refreshed flag. Previously,
a successful refresh that produced entries=[] would fall through to the
on-disk cache, serving stale non-empty data during low-traffic periods.
- Move geoipupdate from Depends to Recommends. It lives in bookworm/contrib,
so a hard dependency breaks 'apt install secubox-metrics' on systems
without contrib enabled. The aggregator already degrades gracefully when
the mmdb is absent, making Recommends the correct strength. README
documents the contrib note.
- prerm stops + disables secubox-geoipupdate.timer/service so 'apt remove'
doesn't leave an orphan timer firing weekly with a missing unit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brainstormed design for adding the SPDX-CMSD-1.0 header to every
first-party source file (~2,170 across 6 languages), backed by a
reusable Python tool (scripts/license-headers.py), a CI check, and
a phased per-package rollout. Spec covers scope, header rendering
per language, placement rules, tool architecture, CI integration
with an enrollment allowlist, and verification steps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Login endpoint now captures:
- Client IP (from X-Forwarded-For, X-Real-IP, or connection)
- User-Agent header
Session events include this info for session storage.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Added session callback mechanism in secubox_core.auth
- Login events (success/failure) now emit to registered callback
- secubox-auth module records sessions in JSON file
- Fixed login.html endpoint URLs and JSON parsing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add hybrid-skin.css: Glass Morphism + Matrix Terminal universal theme
- Update sidebar.js v2.23.0:
- Remove light/dark theme switching (single hybrid theme)
- Add global status bar (bottom) with health metrics
- Add global menu bar (top) replacing individual page headers
- Load health data from /api/v1/hub/health endpoint
- Update sidebar.css: Transparent header/footer to fix white corners
- Update design-tokens.css: Add .hybrid-skin to dark theme selector
- Update index.html files: Apply hybrid-skin body class
- Add SOC dashboard (soc/index.html): Reference implementation
- Add Health Doctor Pattern documentation
- Update module APIs with /health endpoints
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- WAF API: Replace ipapi.co HTTP calls with local GeoLite2-Country.mmdb
- CrowdSec frontend: Use backend /api/v1/waf/geoip/ instead of ipapi.co
- nginx: Add routes for /system/ and /api/v1/system/
- Avoids CORS issues and rate limiting from remote GeoIP services
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Eye Remote Interactive UI enhancements:
- TTY mode: Serial terminal display from /dev/ttyGS0
- Flash mode: Progress bar with speed/ETA for USB transfers
- Auth mode: QR code generation for backup authentication
- Mode detection via /etc/secubox/gadget-mode
Hub service VM compatibility fix:
- Changed from Unix socket to TCP port 8001
- Updated nginx configs for TCP proxy
- Fixes 502 errors in VirtualBox VMs
Also includes:
- FAQ/Troubleshooting wiki page with GitHub issue links
- Kiosk launcher --no-sandbox fix for VMs
- Profile Generator GUI mockup
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The require_jwt dependency used Python 3.10+ union syntax
(HTTPAuthorizationCredentials | None) with Annotated, which
causes FastAPI 0.92/Pydantic 1.10 to incorrectly require a
request body on GET endpoints.
Changed to Optional[HTTPAuthorizationCredentials] = Depends(_bearer)
which is compatible with older FastAPI/Pydantic versions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add CORS headers to nginx secubox-proxy.conf for cross-origin API requests
- Fix login.html endpoints: /auth/login -> /login
- Upgrade Python deps in build scripts: pydantic>=2.0, fastapi>=0.100, uvicorn>=0.25
- Add pip upgrade in secubox-core postinst for Debian bookworm compatibility
- Fix display/__init__.py to import existing modules only
Fixes authentication and API issues in VBox and ebin builds.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove prefix="/auth" from secubox_core/auth.py router definition
- Add prefix="/auth" when including auth_router in hub main.py
- Fixes login endpoint from /auth/auth/login to /auth/login
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add HTTP server block in nginx for localhost/127.0.0.1/192.168.255.1
- Change kiosk default URL from https:// to http:// (avoids SSL issues)
- Add fallback index.html creation in build script
- Ensures nginx has content to serve even if packages fail to install
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace non-existent SecuBoxHeader with BoardHeader in SOC screens.
Remove unused get_board_colors import.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
New modules (35 total):
- secubox-repo v1.0.0: APT repository management
- repoctl CLI for package management
- GPG key generation and signing
- Multi-distribution support (bookworm, trixie)
- Web dashboard for repository status
- secubox-hardening v1.0.0: Kernel and system hardening
- hardeningctl CLI for security management
- Sysctl hardening (ASLR, kptr_restrict, SYN cookies)
- Module blacklist (uncommon protocols, filesystems)
- Security benchmark with 100% score on VM
CI/CD workflows:
- build-packages.yml: Dynamic matrix for all packages
- build-image.yml: 5 board images with compression
- publish-packages.yml: APT repo publishing
- release.yml: Unified release orchestration
APT repository scripts:
- export-secrets.sh: Export GPG/SSH keys for GitHub Actions
- local-publish.sh: Local test server
- install.sh: User installation script
Security (Phase 5):
- AppArmor profiles for all services
- Audit rules for SecuBox services
- build-all.sh for local builds
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Includes all package APIs with public dashboard endpoints:
- secubox-system: System Hub with /info, /resources, /security
- secubox-crowdsec: CrowdSec dashboard with /status, /hub, /metrics, actions
- secubox-wireguard: WireGuard VPN with /interfaces, /peers, start/stop
- secubox-netdata: Monitoring with /stats, /processes, /alerts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
