From 6daacb1987e6771cc0176aeafc4ac1b8b3b6ce92 Mon Sep 17 00:00:00 2001 From: CyberMind-FR Date: Thu, 18 Jun 2026 17:07:48 +0200 Subject: [PATCH] feat(#662 Phase 1): MITM-engine migration analysis + phased plan + compiled/tested Go forging-MITM PoC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Analysis: gomitmproxy (unmaintained, dropped) vs martian/goproxy (Go) vs hudsucker (Rust) vs Squid+ICAP, mapped to the 18-addon capability set. Recommendation: Go hot-path core + retained Python analysis sidecars. Phased plan with shadow-run + nft-DNAT-flip rollback (no big-bang cutover). Phase-1 PoC (packages/secubox-toolbox-ng, stdlib-only): forge from ca-wg CA, 204-block, body-inject, SNI-splice, ClientHello/JA4 capture — go vet clean, tests green, arm64 cross-compile OK. NOT wired to live R3. --- .../plans/2026-06-18-mitm-engine-migration.md | 55 ++++++++ ...26-06-18-mitm-engine-migration-analysis.md | 120 ++++++++++++++++++ packages/secubox-toolbox-ng/.gitignore | 3 + packages/secubox-toolbox-ng/README.md | 60 +++++++++ packages/secubox-toolbox-ng/go.mod | 3 + 5 files changed, 241 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-18-mitm-engine-migration.md create mode 100644 docs/superpowers/specs/2026-06-18-mitm-engine-migration-analysis.md create mode 100644 packages/secubox-toolbox-ng/.gitignore create mode 100644 packages/secubox-toolbox-ng/README.md create mode 100644 packages/secubox-toolbox-ng/go.mod diff --git a/docs/superpowers/plans/2026-06-18-mitm-engine-migration.md b/docs/superpowers/plans/2026-06-18-mitm-engine-migration.md new file mode 100644 index 00000000..f967cddb --- /dev/null +++ b/docs/superpowers/plans/2026-06-18-mitm-engine-migration.md @@ -0,0 +1,55 @@ +# Toolbox MITM engine migration — phased plan (#662) + +> Engine: **Go hot-path core + retained Python analysis sidecars** (see analysis doc). +> Discipline: shadow-run before cutover; nft-DNAT flip = instant rollback at every step; NEVER big-bang. This is a multi-PR epic — each phase is its own PR with a gate. + +## Invariants (must hold every phase) +- Reuse the existing CA `/etc/secubox/toolbox/ca-wg/{ca.pem,key.pem}` (what R3 clients already trust) — no new CA, no client re-enroll. +- Live R3 keeps running on the Python mitmproxy workers (8081-8084) until the final cutover. The Go core runs on **separate ports (8090-8093)**, no DNAT, until Phase 6. +- Ad-blocking + anti-track must never regress (the whole point of the appliance). +- arm64; one static Go binary; systemd `secubox-toolbox-ng-worker@N`. + +## Phase 1 — PoC (THIS PR) — GATE: compiles + smoke test passes +**packages/secubox-toolbox-ng/** (Go module). NOT wired to live R3. +- `go.mod`, `cmd/sbxmitm/main.go`: a forging MITM that loads `ca-wg/{ca.pem,key.pem}`, listens on a port, and demonstrates the discriminating capabilities: + - request short-circuit **204** for a sample ad host (proves ad_ghost block), + - response **body inject** of a marker (proves banner/ad CSS), + - **SNI splice** passthrough for a sample host (proves tls_splice), + - **JA4 ClientHello capture** via a `crypto/tls` shim logging cipher suites/exts (proves the Go JA4 gap is closable). +- Smoke test (`make test` / a shell script): build for host, run, `curl -x`/transparent a request through it, assert the 204 + the injected marker + a JA4 line. +- `README.md`: build (`GOOS=linux GOARCH=arm64 go build`), the capability map, and the phase roadmap. +- **No deb packaging, no board deploy, no DNAT.** Pure de-risking spike. + +## Phase 2 — arm64 build + board bench (no traffic) — GATE: forge+throughput ≥ mitmproxy +- CI/build: cross-compile arm64 static binary; debian packaging stub `secubox-toolbox-ng` (binary + systemd unit, unit DISABLED). +- Deploy the binary to gk2, run on :8090 (no DNAT). Bench: cert-forge latency (cold/warm), req/s, multi-core CPU under synthetic load vs a mitmproxy worker. Confirm it reuses ca-wg certs (client trusts forged leaf). + +## Phase 3 — hot-path feature parity — GATE: parity tests green +Port the cheap per-request rewrites into the Go core, reading the SAME data files: +- block 204 from `_AD_HOST`-equivalent + learned-trackers.txt + pure-trackers.txt, with `ad-allowlist.txt` + own-infra guard (#658) honored. +- header/cookie strip (utiq/protective/anonymize), XFF. +- serve `/__toolbox/loader.js` + `/__toolbox/bundle`; banner inject (buffer + streaming). +- SNI splice from the media seed + learned-splice (the safe, no-auto-promote version). +- Parity harness: feed recorded request/response fixtures to both engines, diff the block/inject/strip decisions. + +## Phase 4 — analysis sidecars + anti-track poison — GATE: sidecar contract tests +- Go core fires unix-socket events (fire-and-forget) to the EXISTING Python services for social-graph / dpi / cookies / avatar / soc / ja4-scoring — reuse their socket contracts; they stay Python, off the hot path. +- Port the deterministic anti-track **HMAC jar + Set-Cookie forge** to Go (small, security-critical → exhaustive tests vs the Python `privacy.py` jar output for identical inputs). +- Contextual ad metrics (ad_block_stats / per-visitor) written by a sidecar or the Go core's bg writer. + +## Phase 5 — SHADOW run — GATE: N-day output parity, zero client breakage +- Run the Go core on :8090-8093. Mirror a SMALL fraction of R3 (e.g. one fanout slot, or a passive tee) to it; compare its would-block/would-inject/recorded against the live mitmproxy for the same flows. Do NOT serve clients from it yet. +- Soak; review divergences; fix; repeat until parity. + +## Phase 6 — CUTOVER — GATE: soak, instant rollback ready +- Flip the nft `numgen inc mod 4` fanout from 8081-8084 (mitmproxy) → 8090-8093 (Go core). Keep the mitmproxy workers RUNNING (stopped from receiving DNAT, but up) so rollback = flip the map back (seconds). +- Soak under real load; watch ad-blocking, banner, anti-track, JA4, latency, CPU. + +## Phase 7 — decommission — GATE: stable post-cutover window +- Stop/disable the mitmproxy workers; keep the package installed (rollback) for one release, then remove. + +## Rollback +At every phase the live path is the mitmproxy workers until Phase 6's DNAT flip; Phase 6 rollback is an nft map edit (seconds). No phase removes the fallback until Phase 7. + +## Effort/risk (honest) +Weeks across 7 PRs. Highest-risk areas: JA4-in-Go (de-risked in Phase 1), the anti-track poison port (Phase 4, exhaustively tested), and the cutover (Phase 6, shadow-gated + instant rollback). Recommend pausing after each gate for review. diff --git a/docs/superpowers/specs/2026-06-18-mitm-engine-migration-analysis.md b/docs/superpowers/specs/2026-06-18-mitm-engine-migration-analysis.md new file mode 100644 index 00000000..743e187a --- /dev/null +++ b/docs/superpowers/specs/2026-06-18-mitm-engine-migration-analysis.md @@ -0,0 +1,120 @@ +# Toolbox MITM engine migration — analysis (gomitmproxy / martian·goproxy / hudsucker / Squid+ICAP) + +- **Date:** 2026-06-18 · **Issue:** #662 · **Status:** analysis + recommendation + +## Why +The R3 path runs Python **mitmproxy**: GIL-bound, ~1 core total across 4 workers, +the tunnel's CPU/latency ceiling (#646). Goal: a multi-core engine **without +losing the 18-addon feature set**. TLS termination was never the bottleneck — +the single-thread L7 work is — so a bare TLS proxy is a non-starter (loses every +feature). The only worthwhile target is a faster **L7 engine** that re-implements +the inline logic. + +## The real requirement: our 18 addons' capabilities +| # | Addon | Capability it needs | +|---|-------|---------------------| +| 1 | inject_xff | requestheaders: set XFF from real peer IP | +| 2 | utiq_defense | requestheaders: detect/strip operator (Utiq) headers; short-circuit | +| 3 | protective_mode | requestheaders: strip tracker headers/cookies, spoof | +| 4 | privacy_guard (anti-track v2) | **request 204 / forge Set-Cookie (HMAC jar) / strip headers**; classify; file+key reads | +| 5 | ad_ghost | request **204** + candidate/per-visitor capture; response **CSS body inject**; allowlist; bg SQLite | +| 6 | media_cache | response synthesis from disk cache (range) | +| 7 | local_store | **tls_clienthello** read + async SQLite | +| 8 | social_graph | response cookie-id correlation + **body peek** + SQLite | +| 9 | inject_banner | request short-circuit **serve** /__toolbox/*; **streaming** body inject + buffered inject; CSP detect | +| 10 | dpi | async fire-and-forget POST (unix socket) | +| 11 | cookies | response Set-Cookie read → async POST | +| 12 | avatar | UA → async POST | +| 13 | ja4 | **raw TLS ClientHello** (cipher suites, extensions, ALPN) | +| 14 | soc_relay | events → async POST | +| 15 | cert_pin_detect | **TLS handshake-error** hook → learn ignore_hosts | +| 16 | media_stats | response headers → stats | +| 17 | tls_splice | **tls_clienthello SNI → connection passthrough** (ignore_connection) | +| 18 | (dpi dup/util) | — | + +Capability buckets that discriminate the engines: +- **(C)** request short-circuit (return 204/synth without upstream) — ad_ghost, privacy_guard, inject_banner, media_cache. +- **(E)** **streaming** response body rewrite (inject into first chunk, no buffering) — inject_banner TTFB path. +- **(G)** **raw ClientHello introspection** for JA4 — ja4, local_store. +- **(H)** **TLS-layer SNI passthrough/splice** — tls_splice, cert_pin_detect, bypass list. +- **(I)** TLS handshake-error hook — cert_pin_detect. +- **(J)** async side-effects (socket POST / bg SQLite) — 7 addons. + +## Engine assessment + +### gomitmproxy (Go, AdguardTeam) — DROP +Purpose-built for ad-blocking MITM, but **last release v0.2.1 (2021), effectively +unmaintained**. Reusing an abandoned TLS-handling core for a security appliance +is the wrong bet. Cross off. + +### martian (Google) / goproxy (elazarl) — Go, maintained +- Strong on **B/C/D/F/J** (modifier/handler APIs return custom responses, modify + headers/cookies/body; goroutines for async). Easy **arm64 cross-compile** + (`GOOS=linux GOARCH=arm64`), single static binary — great fit for the appliance. +- **Gaps:** **(G) JA4** — both abstract TLS at the HTTP layer; raw ClientHello + isn't exposed by the modifier API. *Workaround:* wrap the listener with our own + `crypto/tls` `Config.GetConfigForClient`/`GetCertificate` to capture the + ClientHello before handing to the proxy — feasible, extra code. **(E) streaming + inject** is manual (wrap the response body reader). **(H/I)** host-level + splice/cert-error handling is doable at the CONNECT layer. +- Verdict: pragmatic, lowest-friction toolchain, but JA4 + streaming need custom + glue. + +### hudsucker (Rust, omjadas + ideamans fork) — maintained +- **Best technical coverage:** tokio/hyper async (**multi-core**), `HttpHandler` + (C/D/F), **streaming bodies (E)** native, WebSocket. Critically, **rustls + exposes the ClientHello** (Acceptor/`ClientHello` peek pre-handshake) → **JA4 + (G) is clean**, and SNI-based **splice (H)** is natural. +- **Costs:** Rust **arm64 cross-compile friction** (no toolchain here; needs + `cross`/musl setup), and porting 18 addons + the anti-track HMAC-jar/classify + brain to Rust is the **highest re-implementation + re-validation effort**. +- Verdict: technically the strongest (only one covering JA4 + streaming cleanly), + but the heaviest port + ops. + +### Squid + ssl-bump + ICAP — mature C, multi-process +- **Native wins:** ssl-bump forges from one root key (A), **peek-and-splice (H) + is literally tls_splice + the bypass list**, native cert-error handling (I), + multi-process scaling. ICAP REQMOD/RESPMOD covers **C/D/F** (204, body rewrite, + header/cookie mod) — ad_ghost/banner-buffer/poison can live in an ICAP service. +- **Gaps:** **(E) streaming** inject — ICAP buffers, no first-chunk inject. + **(G) JA4** — ICAP is post-decrypt HTTP; ClientHello isn't exposed to ICAP + (Squid logs its own TLS details, not via ICAP). Heavy **ops/config**; each ICAP + call is a round-trip; the anti-track HMAC-jar/poison + social-graph logic in an + ICAP service is awkward (still Python, still off-core for analysis). +- Verdict: least *custom proxy* code + native splice/cert handling, but loses + JA4 + streaming-banner and trades Python addons for Squid-config + an ICAP + service. Good if we drop JA4/streaming; otherwise a poor fit. + +## Recommendation — **Go hot-path core + retained Python analysis sidecars** (hybrid) +Single-engine "rewrite everything in Rust" is the highest risk; Squid loses JA4 + +streaming. The lowest-risk path to multi-core that **preserves the +security-validated Python brain**: + +1. **Go core** (goproxy/martian or a thin `net/http`+`crypto/tls` forging proxy) + owns the **hot path**: TLS forge (reusing `ca-wg`), SNI splice (H), the cheap + per-request rewrites — block 204 (ad_ghost/privacy_guard), header/cookie strip + (utiq/protective/anonymize), banner inject (E via body-reader wrap), serve + /__toolbox/*. Multi-core, one static arm64 binary. +2. **JA4 (G)** in Go via a `crypto/tls` ClientHello-capture shim (no Python). +3. **Heavy/off-path analysis stays Python sidecars** the Go core feeds + fire-and-forget over unix sockets (J): social-graph correlation, classify, + DB/report writers, SOC/DPI relays. These are already async + off the hot path, + so they don't need to be fast — and we DON'T re-validate the anti-track + HMAC-jar/poison + cookie-graph security logic in a new language. +4. The anti-track **poison** (forge Set-Cookie from the HMAC jar) is hot-path + + security-critical → port the *deterministic* jar/forge to Go (small, testable), + keep classify (which list a host is on) as data the Go core reads from the + learned/pure files (already file-based). + +This gets multi-core on the hot path, keeps the risky brain in validated Python, +and only ports the small, mechanical, hot pieces. If JA4-in-Go proves painful, the +fallback is **hudsucker** (Rust) for the core (clean JA4) at higher port cost. + +## Honest effort/risk +- **Weeks, multi-PR.** 18 addons; security-critical; production board. +- Must **shadow-run** the new core alongside mitmproxy (mirror a fraction of R3 + traffic) and compare before any cutover. **Never** big-bang. +- Rollback = the nft fanout still points at the mitmproxy workers until the final + cutover flips the DNAT to the Go core's ports. + +See the phased plan: `docs/superpowers/plans/2026-06-18-mitm-engine-migration.md`. diff --git a/packages/secubox-toolbox-ng/.gitignore b/packages/secubox-toolbox-ng/.gitignore new file mode 100644 index 00000000..6d4cdd51 --- /dev/null +++ b/packages/secubox-toolbox-ng/.gitignore @@ -0,0 +1,3 @@ +sbxmitm +*.test +/cmd/sbxmitm/sbxmitm diff --git a/packages/secubox-toolbox-ng/README.md b/packages/secubox-toolbox-ng/README.md new file mode 100644 index 00000000..60304b11 --- /dev/null +++ b/packages/secubox-toolbox-ng/README.md @@ -0,0 +1,60 @@ +# secubox-toolbox-ng — Go MITM engine (migration spike, #662 Phase 1) + +De-risking PoC for migrating the R3 toolbox MITM engine off Python **mitmproxy** +(GIL-bound, ~1 core) onto a multi-core **Go** core, **without losing the +18-addon feature set**. See: +- Analysis: `docs/superpowers/specs/2026-06-18-mitm-engine-migration-analysis.md` +- Phased plan: `docs/superpowers/plans/2026-06-18-mitm-engine-migration.md` + +> **Status: Phase 1 — PoC only. NOT wired into the live R3 path.** The live +> tunnel still runs on the Python mitmproxy workers (8081-8084). This binary is +> a standalone CONNECT-proxy spike that proves the risky capabilities. + +## What the PoC proves (the discriminating risks from the analysis) +- **CA-compat forging** — loads the *existing* `ca-wg/{ca.pem,key.pem}` and forges + per-host leaf certs the R3 clients already trust (no re-enroll). Cached per host. +- **request 204** — short-circuit block (ad_ghost / privacy_guard). +- **response body inject** — marker before ``/`` (banner / ad-CSS). +- **SNI splice** — raw passthrough, no MITM, by SNI suffix (tls_splice). +- **JA4 material capture** — `crypto/tls` `GetCertificate` receives the + `ClientHelloInfo` (SNI, cipher suites, ALPN, TLS versions) → proves the `ja4` + addon's handshake fingerprint is reachable in Go (full JA4 extension-hash needs + a raw-ClientHello peek — Phase 4). + +All stdlib (no external modules → builds offline). Tests are network-free +(localhost handshake + temp self-signed CA). + +## Build & test +```sh +cd packages/secubox-toolbox-ng +go test ./... # network-free PoC tests +GOOS=linux GOARCH=arm64 go build -o sbxmitm ./cmd/sbxmitm # appliance target +``` + +## Try it (CONNECT proxy, against the board CA) +```sh +./sbxmitm --ca-cert /etc/secubox/toolbox/ca-wg/ca.pem \ + --ca-key /etc/secubox/toolbox/ca-wg/key.pem --listen :8090 +curl -x localhost:8090 --cacert /etc/secubox/toolbox/ca-wg/ca.pem https://doubleclick.net/ # → 204 +curl -x localhost:8090 --cacert /etc/secubox/toolbox/ca-wg/ca.pem https://example.com/ # → body has the sbx-ng marker +# logs print `ja4 t0304_cNN_a... sni=...` per handshake +``` + +## Capability → engine map (recap) +Go covers request-204 / body-rewrite / header-cookie-mod / splice / async-sidecars +cleanly; JA4 needs the ClientHello shim (proven here); streaming inject + the +anti-track HMAC-jar/poison port land in Phase 3/4. Heavy analysis (social-graph, +classify, DB/report writers) stays in the existing Python sidecars, fed +fire-and-forget over unix sockets. + +## Roadmap (do NOT cut over without the gates) +1. ✅ PoC (this) — forge + 204 + inject + splice + ClientHello capture, compiled + tested. +2. arm64 packaging + board bench on :8090 (no DNAT) — forge/throughput vs mitmproxy. +3. Hot-path feature parity (block lists + allowlist + own-infra guard, header/cookie strip, banner, splice) — parity harness vs the Python addons. +4. Analysis sidecars (unix-socket fire-and-forget) + anti-track HMAC-jar/forge port (exhaustively tested vs `privacy.py`). +5. **Shadow run** — mirror a fraction of R3, compare outputs. No client served yet. +6. **Cutover** — flip nft `numgen` fanout 8081-8084 → 8090-8093; mitmproxy stays up for instant rollback. +7. Decommission mitmproxy after a stable soak. + +Rollback is an nft DNAT-map edit at every step; the Python engine is the live +path until Phase 6. diff --git a/packages/secubox-toolbox-ng/go.mod b/packages/secubox-toolbox-ng/go.mod new file mode 100644 index 00000000..e9bdc401 --- /dev/null +++ b/packages/secubox-toolbox-ng/go.mod @@ -0,0 +1,3 @@ +module github.com/CyberMind-FR/secubox-deb/secubox-toolbox-ng + +go 1.22