The Go MITM engine now injects the REAL visible transparency-banner loader
(replacing the invisible `<!-- sbx-ng banner -->` marker regression), mirroring
the authoritative Python inject_banner.py with stream_inject ON.
- banner.go: injectLoader() builds the guarded loader <script src="/__toolbox/
loader.js" data-mh=.. data-wg=.. async> exactly like Python _loader_script;
placement mirrors _LoaderInjector (after <head>'s '>', else before <body>,
else unchanged); bannerGuard idempotency matches _GUARD; data-mh ascii-stripped.
- /__toolbox/loader.js + /__toolbox/bundle short-circuited in BOTH the CONNECT
mitmPipeline and the transparent path, reverse-proxied to the portal
(--portal, default http://127.0.0.1:8088). Startswith match (query-aware),
fail-open to 204 so a banner asset never 502s the navigation.
- mitmPipeline threads `wg bool`: transparent path derives it from the
10.99.1.0/24 peer IP (R3 WG), CONNECT passes false. Injection tightened to
2xx text/html (Python skips non-200). injectMarker/Policy.Inject kept for the
existing PoC tests.
- banner_test.go: guard idempotency, <head>/<body>/neither placement, wg + mh
attributes, non-ascii stripping, path-detection + portal URL construction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transparent mitm/allow path set req.URL.Host = ip:port, so http.Client
TLS-dialed the captured original-dst and verified the cert against the bare IP
→ guaranteed SNI/cert-name mismatch. Add a per-request transparentTransport
whose DialContext pins the TCP dial to the captured ip:port for every
connection while TLSClientConfig.ServerName = the SNI host, so the upstream is
reached at the real IP yet the cert is verified by hostname. req.URL.Host now
carries the SNI host (correct Host header + SNI); verification stays ON (no
InsecureSkipVerify). The CONNECT path is unchanged (dialHost == "" → it still
dials by req.URL.Host exactly as before).
handleTransparent previously forged a cert and terminated TLS BEFORE Decide, so
a splice host was already MITM'd and the splice branch then io.Copy'd decrypted
plaintext into a cleartext dial — a broken relay of a host policy says to pass
through untouched (cert-pinned apps, own media infra).
Now: peek the ClientHello off the raw conn without consuming it (recordingReader
tees the bytes), parse SNI with a new pure stdlib sniFromClientHello (fully
bounds-checked, never panics), and Decide on the peeked SNI with NO decryption.
splice → dial the ORIGINAL dst, replay the buffered ClientHello upstream, pipe
raw TCP both ways, NEVER tls.Server. allow/mitm/block → re-present the buffered
ClientHello to tls.Server via a prefixConn (Read drains the prefix then
delegates) and run the shared pipeline as before.
Adds table tests for sniFromClientHello (hand-assembled ClientHello with/without
SNI, non-handshake, truncated, not-ClientHello → ("",false)), a no-panic
truncation sweep, and prefixConn replay tests.
main.go (untagged) referenced px.handleTransparent + the transparent accept
loop unconditionally, so the linux-only transparent.go made `GOOS=darwin
go build ./...` fail. Move the accept loop into a linux-tagged runTransparent
helper and add a non-linux transparent_stub.go that log.Fatals; main.go now
calls runTransparent only when --transparent. Verified GOOS=linux/arm64,
linux/amd64 and darwin all build.
Add cmd/sbxmitm/transparent.go (//go:build linux): parseOrigDst decodes a raw
sockaddr_in/sockaddr_in6 blob (endianness-robust family, big-endian port) into
host:port — PURE, fully unit-tested. origDst recovers the pre-DNAT destination
via getsockopt(SO_ORIGINAL_DST=80) using syscall.Syscall6 on the raw fd
(stdlib-only). handleTransparent recovers origDst, terminates TLS by SNI,
splices raw TCP to the REAL captured dst or runs mitmPipeline dialling it.
transparent_test.go table-tests parseOrigDst (v4/v6, both family endiannesses,
BE port, short-blob errors). End-to-end getsockopt capture needs nft DNAT and
is validated at Phase 5 shadow on the board, not in unit tests (documented).
Factor handleConnect's post-handshake logic (read request, apply verdict,
anonymize, proxy upstream, poison, inject, write) into mitmPipeline so the
CONNECT and transparent accept paths can't drift. dialHost param lets the
transparent path dial the captured original-dst instead of the SNI. Add a
--transparent bool flag: when set, a raw net.Listen accept loop dispatches each
conn to handleTransparent; default keeps the CONNECT http.Server EXACTLY.
CONNECT path + its tests unchanged.
clientHashFromConn now resolves the peer IP via macHashOf (WG persona hash,
byte-identical to Python for 10.99.1.0/24), falling back to the raw peer IP for
non-WG/test conns so poison stays deterministic. Updated the TODO block: WG
mac_hash wiring DONE; remaining gap is only the transparent original-dst
plumbing (Deliverable 2) and the intentionally-out-of-scope R0-R2 ARP path.
test_machash_parity.py drives _common.mac_hash_of on the SAME fixtures; both
engines agree. Anti-rig verified on the Python side too.
Port _common._wg_hash_of / mac_hash_of to cmd/sbxmitm/machash.go: WG peers on
10.99.1.0/24 resolve to sha256(peer_pubkey)[:16], mtime-cached behind a mutex
(Go is concurrent; Python relied on the GIL). Off-subnet / R0-R2 ARP path is
out of scope for the R3 transparent engine; any error fails open to "".
Parity fixtures (testdata/wg-peers-fixture.json + machash-fixtures.json) carry
Python-authored expected values; machash_test.go asserts macHashOf matches.
Anti-rig verified: [:16]->[:15] fails the test.
New packages/secubox-toolbox-ng/debian/ producing the secubox-toolbox-ng
binary package (Architecture: arm64):
- control: Maintainer Gerald KERMA; B-D golang-go; Depends
only (static CGO_ENABLED=0 binary → no shlib deps). Compat 13 via
debhelper-compat build-dep (debhelper rejects compat both ways).
- changelog: 0.1.0-1~bookworm1.
- rules: dh; GOOS=linux GOARCH=arm64 CGO_ENABLED=0 GOPROXY=off go build
(pure stdlib, offline). dh_installsystemd --no-enable --no-start so the
unit is shipped but NEVER enabled/started.
- secubox-toolbox-ng-worker@.service: systemd template mirroring the Python
mitm-wg worker@ but running sbxmitm on 127.0.0.1:809%i (distinct from the
Python 808%i so both fleets coexist during cutover). Reads the ca-wg CA.
DISABLED BY DESIGN — header documents Phase-6-cutover-only enablement.
- postinst: daemon-reload only; explicitly NO enable/start; NO nft.
Built locally for arm64: dpkg-deb verified — ships /usr/sbin/sbxmitm (arm64
static ELF) + the disabled template; postinst contains ZERO deb-systemd-helper
enable lines. .gitignore extended for in-tree build artifacts. DARK: install
changes no runtime behaviour (no service start, no DNAT, no live-R3 wiring).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the hardcoded action() short-circuit in handleConnect with the
ported Decide(host,sni) + always-on anonymize + Set-Cookie poison:
- allow/own-infra → clean MITM (anonymize only, NO block/poison)
- splice → raw passthrough (unchanged)
- block → 204 (unchanged)
- mitm + tracker → poison tracking-id Set-Cookies via the HMAC jar
New pure, unit-testable helpers (privacy.go):
- anonymizeRequest(http.Header): drop operator/carrier + re-id headers
(mirrors privacy_guard._STRIP), pin DNT:1 + Sec-GPC:1.
- isTrackingCookieName / poisonSetCookies: replace tracking-id cookie
values with fakeID(clientHash,host,name,jarKey); attrs preserved,
benign cookies untouched, fail-closed-to-clean when no key/clientHash.
- Policy.isTracker / Policy.shouldPoison: poison ONLY on MITM'd tracker
flows, never on allow/own-infra (same dark safety as the block path).
- clientHashFromConn: PoC peer-IP stub, TODO(#662 P6) mac_hash via
SO_ORIGINAL_DST + WG-peer map.
writeResponse (util.go) preserves multi-valued Set-Cookie headers.
Poison gated behind --poison (default on) AND a loaded --jar-key.
DARK: nothing wired to live R3. +8 tests (14→22), all green; vet clean;
arm64 cross-build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add sidecar.go (package main, stdlib only): emit(socketPath, route, payload)
relays a signal to a SecuBox module's unix socket in a detached goroutine —
never blocks the proxy flow, never raises into the caller (mirrors
_common.fire_forget_post + queue_async). emitSync is the same-package,
test-observable synchronous form under a 2s timeout (mirrors httpx timeout=2).
Documents the addon→socket mapping the live engine will use
(cookies/dpi/avatar/ja4/soc_relay → /run/secubox/*.sock; social_graph is
in-process). NOT wired into the live path — transport only (Phase 5+ wiring).
sidecar_test.go: delivery over a throwaway unix socket, dead-socket
no-panic/no-block, empty-route defaulting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port privacy.py's _jar_key / _shape / fake_id into jar.go (package main,
stdlib only): loadJarKey (read+TrimSpace, empty->nil), shape (GA1/fb uint64
big-endian modulo math, uuid via rune-length >=32 branch, hex[:32] default),
fakeID (HMAC-SHA256 over client|registrable(tracker)|cookie, reuses the
Phase-3 registrable()). Returns ("",false) where Python returns None.
Cross-engine parity proven: testdata/jar-fixtures.json (expect values
GENERATED by privacy.fake_id with a FIXED test key, not the real /etc key)
covers _ga, _ga_<prop> GA4, _fbp, uuid, _pk_id, name>=32, generic hex, and a
doubleclick.net subdomain-folding case. jar_test.go (Go) and
tests/test_jar_parity.py (Python) load the SAME fixtures+key and both pass ->
byte-exact. No int-math or rune-length divergence found.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tests/test_engine_parity.py loads the SAME parity-fixtures.json + testdata
config from ../secubox-toolbox-ng/testdata, monkeypatches ad_ghost paths
(_ALLOW_PATH/_LEARNED_PATH/_SELF_REGS + cache resets) at the snapshot, and
drives the production decision logic — ad_ghost._allowed + _AD_HOST + the
learned-trackers check composed with splice.should_splice — under the SAME
precedence as Go's Decide. Asserts action == expect for every fixture.
Parity proven: this run caught a real Go↔Python divergence — Go had
comment-stripped learned-trackers, but ad_ghost._learned_set does not; Go was
fixed (loadLinesRaw) to match Python. Python is the source of truth.
test_fixtures_present guards that all four action classes are exercised.
testdata/parity-fixtures.json + testdata/config/ snapshot: a FIXED config both
engines load identically. Fixtures cover every action class — static ad host,
learned-tracker, pure-tracker that is also a splice candidate (never wins →
block), own-infra secubox.in subdomain (allow), allowlisted host (allow),
splice-seed + splice-learned hosts (splice), fortknox site in never-set (mitm),
no-false-suffix negative (notdoubleclick.net → mitm), plain site (mitm).
policy_test.go: TestParityDecide loads the fixtures+config and asserts
Decide == expect; TestPolicyActionVerbs checks the legacy action() surface;
TestRegistrable exercises the _registrable port incl. 2-level TLDs.
Add cmd/sbxmitm/policy.go: a LoadPolicy() layer that reads the SAME on-disk
config the Python addons use (ad-allowlist.txt, learned-trackers.txt,
tls-splice-seed.conf, splice-learned.txt, pure-trackers.txt) with the same
env overrides, plus a unified Decide(host, sni) -> {allow,block,splice,mitm}.
Ports, byte-for-byte against the Python source of truth:
- _AD_HOST regex (RE2-safe → Go (?i) inline flag, no fallback needed)
- _registrable incl. the _2L two-level-TLD list
- splice.host_matches / should_splice (never wins; then seed∪learned)
- ad_ghost._allowed (own-infra + allowlist ALWAYS win first)
Loader nuance preserved: ad_ghost._learned_set does NOT comment-strip
(machine-generated file), unlike the splice/allowlist loaders — mirrored via
loadLinesRaw vs loadLines so a '#' in learned-trackers is kept verbatim.
Decide precedence: allow > splice (never-set excludes trackers) > block > mitm.
Wire the loaded policy into the PoC CONNECT proxy (replacing the hardcoded
AdHosts/SpliceHosts); action() keeps the legacy 3-verb surface (allow→mitm).
Old TestActionDecision removed (drove the removed hardcoded fields); coverage
moves to the parity harness.
Analysis: gomitmproxy (unmaintained, dropped) vs martian/goproxy (Go) vs hudsucker
(Rust) vs Squid+ICAP, mapped to the 18-addon capability set. Recommendation: Go
hot-path core + retained Python analysis sidecars. Phased plan with shadow-run +
nft-DNAT-flip rollback (no big-bang cutover). Phase-1 PoC (packages/secubox-toolbox-ng,
stdlib-only): forge from ca-wg CA, 204-block, body-inject, SNI-splice, ClientHello/JA4
capture — go vet clean, tests green, arm64 cross-compile OK. NOT wired to live R3.
renders d.top_visitors into #ads-visitors; clickable mac_hash calls
loadAdsClient → /admin/ad-stats/client/{mac_hash} into #ads-client-detail.
all values escaped.
