* docs(sentinel): design — inline IOC gate + async Go analyzer + spyware packs (ref #823)
* docs(sentinel): implementation plan — 12 tasks (ref #823)
* feat(sentinel): IOC model + IOCSet matchers (ref #823)
* feat(sentinel): pack loader + live-overlay merge + hot-reload (ref #823)
Add internal/sentinel/pack.go: Pack (JSON base/overlay content bundle),
LoadBasePack, MergePacks (base + N overlays, override by Type+Value via
IOCSet's per-type maps), and Loader (NewLoader/Set/MaybeReload/YaraRules).
Loader mirrors cmd/sbxmitm/policy.go's LoadPolicy/maybeReload mtime-based
hot-reload pattern, adapted for a directory of a DYNAMIC *.json file set
(glob-based) rather than policy.go's fixed Target list. Fail-safe: a
corrupt overlay pack is skipped+logged, never wipes the base or panics;
MaybeReload is a cheap stat-only no-op when nothing on disk changed;
Set()/YaraRules() read via atomic.Pointer so a reload swap never torn-reads
a concurrent hot-path lookup.
Adds JSON tags to Task 1's IOC struct (type/value/class/severity/source/
action) so packs round-trip through encoding/json.
* fix(sentinel): pack loader — fail-closed on base-wipe + throttle MaybeReload + url_regex override + yara dedup (ref #823)
Review found 1 Critical + 2 Important + 1 Minor in Task 2 (pack loader):
- Critical: a vanished/degraded base pack directory (deleted dir, mount
blip, mid-dpkg-upgrade window) could silently empty the live detection
set on the next MaybeReload. loaderState now tracks baseCount; a reload
that would drop base IOCs from >0 to 0 is refused (prior state kept,
warning logged) instead of swapped in.
- Important: MaybeReload() is called per-flow by Task 3's inline gate: add
a 5s default throttle (mirrors cmd/sbxmitm/policy.go's reloadThrottle)
so a hot-path call is a pure no-op until the window elapses.
- Important: url_regex IOCs previously only appended, so a live overlay
could never override a base pack's URL rule (base always matched
first). Add urlIndex (pattern -> slice index) so re-adding the same
pattern replaces the entry in place, keeping deterministic match order.
- Minor: YaraRules() now dedups identical rule bodies across base+overlay
packs instead of emitting duplicates.
go test ./internal/sentinel/: 21/21 PASS. go vet/gofmt/go build clean.
* feat(sentinel): inline gate matcher (fail-open, highest-severity) (ref #823)
* feat(sentinel): bbolt verdict store (pure-Go, WAL-free, TTL prune) (ref #823)
* feat(sentinel): action scorer — heuristic/low-confidence forced report-only (ref #823)
* feat(sentinel): bounded fire-and-forget mirror channel (drop-with-count) (ref #823)
* fix(sentinel): mirror Emit always copies body — no aliasing of caller's hot-path buffer (ref #823)
* feat(sbxmitm): inline Sentinel gate — neutralize high-confidence, mirror rest, fail-open (ref #823)
Wire internal/sentinel's inline IOC gate into the shared mitmPipeline response
path (covers both the transparent R3 and CONNECT accept paths without drift):
build a FlowMeta from Host/URL/ClientIP/MacHash after up.Do, call the gate, and
on a HIGH-CONFIDENCE known-infra hit neutralize inline — block/sinkhole serve a
Sentinel block page instead of the upstream response, strip serves the response
with an emptied body. Everything else (miss, heuristic, low-confidence, report)
is mirrored bounded fire-and-forget to the async analyzer and proceeds unchanged.
Fail-open everywhere: SENTINEL_ENABLED unset/false, a pack-load failure, a nil
hook, or a gate panic all degrade to a transparent passthrough — a Sentinel
setup problem can never break browsing. With the engine off, behavior is
byte-identical to today. Env: SENTINEL_ENABLED / SENTINEL_PACK_DIR /
SENTINEL_OVERLAY_DIR / SENTINEL_MIRROR_SOCK.
Hot path (BenchmarkSentinelInspectMiss, -benchmem): 101.0 ns/op, 0 B/op,
0 allocs/op — one throttled gate Match + a non-blocking mirror Emit; no YARA or
heavy analysis inline.
go test ./cmd/sbxmitm/... -race → ok (all pre-existing + 7 new green).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(sentinel): sbx-sentinel async daemon scaffold + store wiring (ref #823)
* feat(sentinel): YARA engine (cgo, build-tagged) + no-cgo stub (ref #823)
Add internal/sentinel/yara.go (cgo && yara) wrapping
github.com/hillu/go-yara/v4 for compiled-rule body scanning, and
yara_stub.go (!yara) as the no-op default: NewYaraEngine always
succeeds, Scan/Analyze always return nil. Both expose identical
signatures (YaraEngine, NewYaraEngine, Scan, Analyze, Close) so
sbx-sentinel/sbxmitm can wire a YaraEngine unconditionally.
yara_test.go (untagged) proves the default build works with YARA
disabled. yara_real_test.go (yara tag) compiles a trivial EICAR rule
and exercises the real libyara-backed engine — split into its own
file since Go build constraints are file-scoped and can't mix a
tagged and untagged test in one file.
go-yara is now an indirect go.mod requirement and vendored, but is
only reachable via the yara build tag: default `go build ./...` and
`CGO_ENABLED=0 go build ./...` both stay clean, no libyara/cgo
required. `-tags yara` needs libyara-dev + a C toolchain (documented
in yara.go's header).
* fix(sentinel): YARA verdict confidence 90 (>= threshold) so a confirmed malware match actually strips, not report-only (ref #823)
* feat(sentinel): behavioral engine — beaconing + zero-click heuristics (report-only) (ref #823)
* fix(sentinel): tighten one-time-link heuristic — exclude file-extension + base64 shapes (cut zero-click FPs) (ref #823)
* feat(sentinel): spyware indicator engine + commercial-spyware base pack (MVT/CitizenLab) (ref #823)
* fix(sentinel): base-pack declared-block IOCs severity>=85 so they actually block + general guard test (ref #823)
Three shipped base-pack IOCs declared action:"block" but severity:80,
below scorer.go's HighConfidenceThreshold=85 — FinalizeAction silently
downgrades any Confidence below that threshold to report-only, so these
entries never actually blocked despite being declared as known-bad:
- packs/base/malware.json: ip 198.51.100.23 (class malware)
- packs/base/malware.json: cert_sha1 ...01234567 (class trojan)
- packs/base/botnet.json: ja3 deadbeef... (class botnet_c2)
Bumped all three to severity 90 so they survive FinalizeAction and
actually block, matching every other declared-block entry in the base
packs.
Added TestBasePackDeclaredBlockIsHighSeverity, a general guard over
every base pack file/entry: any IOC with an auto-neutralize action
(block/strip/sinkhole) must carry Severity >= HighConfidenceThreshold.
This generalizes the narrower TestBasePackKnownInfraIsHighSeverity
(spyware classes only) so a future pack entry in any class can't ship
a "block" that silently never blocks. Confirmed the new test fails red
against the pre-fix JSON (all 3 offenders reported) and passes green
after the severity bump.
* feat(sentinel): proposal/solution reporter — RenderReport(Verdict) (ref #823)
* feat(sentinel): wire real analyzers into sbx-sentinel + read-only status HTTP surface (ref #823)
* feat(sentinel): live threat-feed → overlay pack fetcher (fail-safe, atomic) (ref #823)
* feat(sentinel): live feed overlay + reporter + daemon analyzer wiring + packaging (ref #823)
* fix(sentinel): feeds fetcher — add curl --fail so HTTP 4xx/5xx is treated as a fetch failure (ref #823)
* fix(sbxmitm): drop mis-wired ClientIP from inline Sentinel FlowMeta (source IP must not match destination-C2 IP IOCs) (ref #823)
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs(media-buffer): design spec — dashcam capture + replay + metatag lifecycle (ref #812)
* docs(media-buffer): phase-1 implementation plan (capture+janitor+API+DPI tab) (ref #812)
* feat(sbxmitm): media buffer store + metatag writer (ref #812)
* fix(sbxmitm): media-buffer review — 4096 atomic-append cap + *string buffer_ref (ref #812)
Task 1 review (Important): mediaBufferLineMax 8192→4096 to match mediacatch.go's
PIPE_BUF atomic-append guarantee across 4 workers; BufferRef string→*string so the
janitor (Task 3) can flip it to JSON null on eviction.
* feat(sbxmitm): tee media up/download bodies into the buffer (ref #812)
Wire the R4 media buffer tees into mitmPipeline: the download hook wraps
resp.Body and the upload path wraps req.Body via teeReadCloser, capturing
whole-file media into the rolling /data buffer. Add the mbuf field + flags
(--media-buffer default OFF, --media-buffer-root, --media-buffer-per-object).
Design upgrade over the plan sketch: ObjectWriter.Write is now NON-BLOCKING.
Instead of io.TeeReader driving a synchronous os.File.Write on the client read
path (which would add latency / could stall large media on ARM eMMC), Write
copies each chunk into a bounded channel and returns len(p),nil immediately; a
background goroutine started at Capture drains the channel to disk. A full
channel drops the chunk and flags truncated — never blocks, never errors the
proxied flow (design 4.2/7). Close signals the drainer, waits for it to flush
queued chunks + close the file, then appends the metatag; it is idempotent,
deadlock-free (sole closer of ch) and safe if Write is never called or races
Close.
* fix(sbxmitm): media download tee — defer closure so w.Close runs (ref #812)
defer resp.Body.Close() bound the ORIGINAL upstream body at the defer
statement, before the media-buffer tee reassigned resp.Body — so the
tee's Close() (and w.Close, which flushes the sink/metatag/drain
goroutine) never ran on any download. Wrap the close in a closure so
it reads resp.Body at return time instead.
Adds TestTeeDeferOrderingMatchesMainGo, which replicates main.go's
exact defer-then-reassign ordering and fails against the old plain
defer form (verified manually, then restored the fix).
Also: drop the dead write-only ObjectWriter.dropped field (truncated
already covers it), document Capture's reserved contentLen param, and
note that ObjectWriter.Close is allowed to block (post-stream
finalisation), unlike Write.
* feat(sbxmitm): media buffer janitor — time + LRU eviction, metatag survives (ref #812)
SweepOnce evicts a session's bytes (RemoveAll, idempotent) once
first_ts+retentionSecs elapses, then falls back to oldest-first LRU
eviction if the buffer root (media bytes only, log excluded) still
exceeds sizeCeilBytes. Eviction never rewrites the append-only
metatag log in place — it appends a corrected line with the same id
(expired:true, buffer_ref:null); readers dedup by id, last line wins,
so the flip is durable and safe across the 4 concurrent sbxmitm
worker processes each running their own RunJanitor (idempotent
RemoveAll + harmless duplicate flip lines).
RunJanitor ticks SweepOnce every 30s (test-overridable period) until
its context is cancelled; wired at sbxmitm startup as a
process-lifetime goroutine when --media-buffer is enabled.
NewMediaBuffer's signature is unchanged; retentionSecs (1200),
sizeCeilBytes (24 GiB) and nowFn now default inside the constructor.
* feat(core): media-buffer metatag reader (dedup by id, fail-empty) (ref #812)
* test(core): media-buffer reader — tail-read truncation + malformed-field regressions (ref #812)
* chore(#812): keep SDD scratch reports out of the feature diff (restore to master)
* feat(dpi): media-buffer list/replay/thumb API (admin/owner, audited, 410-on-evict, traversal-safe) (ref #812)
* harden(dpi): media API — \Z-anchored id regex + role short-circuit security note (ref #812)
* feat(sbxmitm): media-buffer retention/size-ceil flags + /data tmpfiles + opt-in worker env toggle (ref #812)
- main.go: --media-buffer-retention / --media-buffer-size-ceil flags, applied
onto the already-constructed mbuf's unexported retentionSecs/sizeCeilBytes
fields (NewMediaBuffer signature unchanged).
- tmpfiles/zz-secubox-media-buffer.conf: pre-creates /data/secubox (0755) and
/data/secubox/media-buffer (0750 secubox:secubox), installed via the
package's existing manual tmpfiles.d mechanism in debian/rules.
- secubox-toolbox-ng-worker@.service: EnvironmentFile=-/etc/secubox/toolbox-ng.env
+ trailing $SBX_MEDIA_BUFFER_ARGS on ExecStart so an operator can opt in
(SBX_MEDIA_BUFFER_ARGS=--media-buffer) without editing the shipped unit;
default stays OFF. Added ReadWritePaths=/data/secubox/media-buffer.
* fix(sbxmitm): media-buffer dir owned by writer (secubox-toolbox) + setgid group secubox; changelog 0.1.27 (ref #812)
Task 6 review Critical: buffer dir was 0750 secubox:secubox but sbxmitm workers run
as secubox-toolbox (not in group secubox) → EACCES on every capture, silently no-op'd
(errors swallowed by design). Fix: 2750 secubox-toolbox:secubox — writer owns, dpi
reader (secubox) is group, setgid propagates group secubox to created objects so the
reader can read them. Important: add debian/changelog 0.1.27 stanza (version bump).
* feat(dpi): Media tab — live capture gallery with replay/download, escaped fields (ref #812)
Adds a Media Buffer card to the DPI dashboard (packages/secubox-dpi/www/dpi/index.html)
that lists sbxmitm's live media-buffer captures via GET /api/v1/dpi/media/buffer.
- Auth: dedicated apiAuthed() reads the canonical `sbx_token` localStorage key
and sends it as a Bearer token (media/buffer requires JWT, unlike the rest
of this page's endpoints which currently ride the SSO-lite session cookie);
401 redirects to /login.html, matching sibling dashboards (e.g. crowdsec).
- Security: host/url/mac_hash are attacker-influenceable (captured live
traffic). Every card is built via createElement + textContent (or the
`.title` IDL property, never HTML-parsed) — no innerHTML string
concatenation of these fields — so a captured value like
`<img onerror=...>` renders as inert text. Replay/download URLs are built
with encodeURIComponent(id) even though the id is hex-validated server-side.
- UI: kind emoji (video/audio/manifest/file), short device id (shortDev,
reused from the existing #785 helpers), ⬆/⬇ direction, human size
(formatBytes) and age (agoStr) per card; expired captures grey out and show
"métatag seul (expiré)" with no action links.
- Refresh: wired into the existing refreshAll() plus its own ~15s interval.
Verified with a Playwright harness stubbing fetch (incl. an XSS payload in
host/url) against a local copy of the page: no console/page errors, XSS did
not fire, no <img>/<script> element materialized from the captured strings,
expired-card gating confirmed.
* fix(sbxmitm): media buffer — upload tee needs a real body, download tee 200-only (whole-branch review); drop dead item.ts (ref #812)
* docs(media-buffer): Phase 2 plan — HLS reassembly via read-time URL join (ref #812)
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* docs(spec): rich PDF character sheet + persona/dpi/media route parity (ref #790)
* docs(plan): rich PDF character sheet + route parity implementation plan (ref #790)
* feat(toolbox): rich PDF character sheet — pips, on/off inventory, quests (ref #790)
* feat(toolbox): factor _enrich_report_data + apply to token/admin PDF routes (ref #790)
* feat(toolbox): populate Quêtes/menaces from raw DPI alerts (dest/detail) — HTML + PDF (closes#792, ref #790)
_dpi_stats now exposes alerts_raw (kind/service/dst/detail) alongside the donut
'alerts' ({label,count}). The persona Quêtes section (PDF _persona_block + HTML
.nr card) reads alerts_raw so each threat shows its destination + detail instead
of a dangling em-dash; donut 'alerts' kept for the DPI-Exfil tab; graceful
fallback to the donut label (no dangling dash) when alerts_raw is absent.
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* docs(spec): rapport kbin fidèle + media types + WebUI DPI (ref #785)
* docs(plan): rapport kbin media types implementation plan (ref #785)
* feat(core): shared media-catch JSONL aggregator (ref #785)
* fix(core): guard media-catch aggregate against malformed fields + full SPDX header (ref #785)
* feat(toolbox): _media_stats + wire media_exfil into report routes (ref #785)
* feat(toolbox): PDF DPI-exfil/overall donut-grids + media-type block (ref #785)
Replaces the text-bullet DPI/EXFIL section (per-device + overall) with
_pdf_donut_grid 4-donut grids and adds a new "TYPES DE MEDIAS CAPTES"
section (kinds/content-types donut grid + top-hosts emoji table), for
parity with the HTML report. Also guards the PDF footer's italic
set_font against DejaVu-Oblique not being registered (fonts-dejavu-core
ships Regular+Bold only) — a pre-existing latent crash on any full
render_pdf() call, uncovered because no prior test exercised the whole
function end-to-end.
* fix(toolbox): per-grid captions so network/media donuts aren't mislabeled as device stats (ref #785)
* feat(toolbox): media-type cards in report web page (me + overall) (ref #785)
* fix(toolbox): close media card divs so Overall tab + footer aren't nested in DPI pane (ref #785)
* feat(dpi): WebUI card — services by category (bytes) (ref #785)
* feat(dpi): /media_types endpoint (MIME media-catch, fail-empty) (ref #785)
* feat(dpi): WebUI card — MIME media types + refreshAll wiring (ref #785)
* chore: gitignore SDD run scratch + drop stray task report (ref #785)
* perf(core): bounded tail-read for media-catch + import-in-try for uniform fail-empty (ref #785)
* fix(dpi): backport board-live async control endpoints (drift reconciliation, ref #785)
The board ran a manually-deployed main.py with status/restart/start/stop/logs/
interface_list/tc_status/remove_mirred as 'async def' — never committed to git.
Backported so the next .deb install doesn't revert them. Signature-only change,
verified identical to the live-validated deployed file.
* fix(toolbox): PDF render must not wedge the event loop (#785 incident)
Heavier #785 report PDFs (more matplotlib donut-grids, ~9s render) + the WAF
504-page auto-retry storm pegged the single uvicorn worker and 504'd the whole
toolbox vhost. Fixes:
- run render off the event loop (threadpool) so HTML report / landing stay live
- serialize renders through one asyncio lock (pyplot is not thread-safe; also
bounds CPU so retries can't render in parallel)
- short per-device PDF cache with a double-checked lock: a retry storm now
triggers exactly ONE render, all other hits return cached bytes instantly
- persistent MPLCONFIGDIR (systemd drop-in + postinst dir) so matplotlib builds
its font cache once instead of on every process start
Validated live on gk2: cold render caches, 8-request retry storm all served
from cache (~0.1s), /landing 0.012s throughout.
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* feat(p2p): DHT, Federation, MasterLink evolutions for P2P-EVO-2026-07-001
Implements three major features for secubox-p2p:
1. DHT (Distributed Hash Table) based on Kademlia algorithm
- Peer discovery across different subnets
- Resilience to temporary node failures
- Scalability for large number of nodes
- REST API endpoints for DHT management
2. Service Federation
- Service registration and discovery
- Multi-version service management
- Automatic health checks
- DHT integration for service propagation
3. Master-Link Hierarchical Topology
- Master/satellite/leaf hierarchical structure
- Automatic master election and failover
- Heartbeat monitoring
- Routing policy management
- OPAD security integration
New files:
- api/dht.py: Kademlia DHT implementation
- api/federation.py: Service federation system
- api/masterlink.py: Hierarchical topology manager
- api/main_evolutions.py: Integrated FastAPI server
- tests/test_dht.py: Unit tests for DHT
Issue: P2P-EVO-2026-07-001
Worktree: secubox-p2p-evolutions
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
* revert(p2p): remove non-integrating DHT/federation/masterlink code, keep requirements
The DHT/federation/masterlink modules + main_evolutions.py + test_dht.py did not
integrate with the module: tests imported 'secubox.p2p.api.*' (the package uses
'from api import ...' via conftest sys.path) so nothing even collected, and the
modules reinvented standalone aiohttp servers instead of building on the existing
WireGuard mesh (mesh.py) + annuaire-backed registry (registry.py/annuaire_client.py).
Removing them to restart on the real base. The requirements doc
(.github/ISSUES/2026-07-P2P-EVOLUTIONS.md) is kept as the spec source.
* docs(p2p): point the local evolutions note to real issue #774
The '.github/ISSUES/2026-07-P2P-EVOLUTIONS.md' was a local file with a fabricated
id, not a tracked issue. Replace its contents with a pointer to the real GitHub
issue #774 (Kademlia DHT + hierarchical master-link + federation health-checks),
framed against the already-done federation (#766) and directory (#768) work and
the master-link/auto-enrollment issue (#762).
* docs(spec): p2p Kademlia DHT + master-link + federation health-checks (#774)
Clean redesign on the real base (mesh.py + registry/annuaire_client) after
reverting the non-integrating attempt. Custom minimal Kademlia (pure asyncio
UDP, JSON wire, signed reachability records), federation health-checks layered
over the annuaire federation (#766) publishing status via the DHT, and a
deterministic-election master-link (term-based failover, aligns #762). All
feature-flagged off by default (OPAD opt-in), from api import convention, TDD
per module. Refs #774.
* docs(plan): p2p DHT/federation/master-link TDD implementation plan (#774)
17 bite-sized TDD tasks across 3 phases (DHT 1-9, federation health-checks
10-13, master-link 14-16, finalization 17). Each task = failing test -> minimal
impl -> pass -> commit, on the real base (from api import ...). Crypto (Ed25519
sign/verify) and UDP transport are module-level seams injected in unit tests so
the Kademlia primitives, health debounce, and election/term logic are tested
without real sockets. Refs #774.
* feat(p2p): DHT scaffold — node id + xor distance (#774)
* feat(p2p): DHT k-bucket with LRU (#774)
Add DHTNode contact + DHTBucket (k-bucket with LRU ordering).
- DHTNode: dataclass with node_id, did, endpoint, last_seen
- DHTBucket: Kademlia k-bucket with OrderedDict LRU management
- add(node) → bool (True if stored/refreshed, False if full)
- refresh moves node to tail (most-recent)
- oldest() returns head node
- remove(node_id) and nodes property for list access
All Task 1 tests + new Task 2 tests passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(p2p): DHT routing table + closest-N (#774)
* feat(p2p): DHT signed reachability records + verify (#774)
Implement canonical_record, sign_record, verify_record functions with
crypto SEAMS (_did_from_pubkey, _verify_sig, _sign_sig) for testing.
Tests use monkeypatching; real crypto wired in Task 8.
Canonical records are deterministic sorted JSON. verify_record checks
for sig field, DID validity via wg_pubkey, and cryptographic signature.
* feat(p2p): DHT JSON UDP codec + hardening (#774)
* feat(p2p): DHTNetwork store+dispatch (transport-injected) (#774)
* feat(p2p): DHT iterative find_node/find_value + announce (#774)
Add asyncio-based iterative Kademlia lookup on top of the injected
DHTNetwork transport from Task 6:
- pending: dict[rpc_id, Future] on DHTNetwork, resolved by handle_message
when a reply (pong/nodes/value/ok) arrives with a matching rpc_id.
- _rpc(node, msg): send + await the matching future with RPC_TIMEOUT,
returns None on timeout.
- iterative_find(target_id, mode): alpha-parallel iterative lookup,
dedups queried nodes, merges discovered contacts into the shortlist
(dedup by node_id, skip self), terminates on no-improvement-in-a-round
or exhausted candidates; mode="value" short-circuits on first verified
record.
- find_peer(did) / announce(): DID resolution and self-record
publication (sign, cache locally, push to the k closest nodes found).
Tests (tests/test_dht.py, +2 async, pytest-asyncio strict mode):
- in-process UDP router delivering via call_soon (never inline) so
awaited RPC futures resolve on a later loop turn;
- find_peer resolves a record announced by a node reachable only
through a bootstrap intermediary;
- dedup: a node introduced twice into the shortlist (via two different
repliers) is queried exactly once.
All 16 tests green (14 prior + 2 new).
* fix(p2p): harden DHT iterative lookup — skip malformed contacts, cap shortlist, tolerate rpc exceptions (#774)
* feat(p2p): DHT record schema (id+wg keys) + real Ed25519 sign/verify (#774)
* feat(p2p): DHT UDP transport + routing persistence + bootstrap (#774)
* feat(p2p): mount DHT — endpoints + startup guard + [dht] config (#774)
Task 9: wire the finished Kademlia DHT (api/dht.py) into the FastAPI
app as a feature-flagged background service, fully backward compatible.
- api/mesh.py: load_p2p_config() now also returns a `dht` sub-dict
(enabled/port/bootstrap/announce/announce_interval/rps) built from an
optional [dht] toml section, with defaults when absent.
- api/main.py: @app.on_event("startup") _dht_startup creates+binds a
DHTNetwork only when [dht].enabled is true; any failure (missing
identity, bind error, ...) is caught and leaves app.state.dht = None
so the app always starts. _dht_shutdown persists the routing table
and closes the UDP transport.
- New endpoints (bare paths, nginx adds /api/v1/p2p prefix): GET
/dht/peers (routing snapshot), POST /dht/announce (JWT-protected,
audit-logged to /var/log/secubox/p2p-audit.log), GET /dht/find/{did}.
Tests: tests/test_dht_wiring.py (config defaults/overrides + disabled
TestClient smoke). All 22 existing DHT tests + 3 new tests green (74
total in the package).
* feat(p2p): federation HealthStore + debounce (#774)
* feat(p2p): federation HealthChecker sweep (#774)
* feat(p2p): federation real probe + registry services (#774)
* fix(p2p): bound TCP-probe close, declare aiohttp dep, guard non-dict service (#774)
* feat(p2p): federation health publish-via-DHT + endpoints + [federation] config (#774)
* fix(p2p): wire federation probe_timeout + harden sweep/health-TTL tests (#774)
* feat(p2p): master-link elect() + term store (#774)
* feat(p2p): master-link state machine + failover (#774)
* fix(p2p): master-link equal-term tie-break (no zero-master window) + defensive peers (#774)
* feat(p2p): master-link signed heartbeats + UDP transport + tick loop (#774)
* feat(p2p): mount master-link — endpoints + startup guard + [masterlink] config (#774)
- masterlink.MasterLink.request_promotion(): on-demand promotion attempt
(bump term, run election, become MASTER + heartbeat on win, else CANDIDATE)
- mesh.load_p2p_config(): add [masterlink] section defaults/overrides
(enabled, role_preference, priority, heartbeat_interval, election_timeout,
port, peer_addrs) alongside existing [dht]/[federation]
- main.py: _masterlink_startup/_masterlink_shutdown mirror the DHT/federation
defensive startup pattern (feature-flagged, never breaks app startup);
GET /masterlink/topology (public) + POST /masterlink/promote (JWT, audited
to p2p-audit.log)
* fix(p2p): master-link request_promotion records winner on loss + strengthen test (#774)
* docs(p2p): document DHT/federation/master-link evolutions (#774)
Add an "Evolutions (#774)" section to packages/secubox-p2p/README.md
covering the DHT (api/dht.py), federation health-checks (api/federation.py)
and hierarchical master-link (api/masterlink.py) subsystems: what each does,
their API endpoints, the real p2p.toml config defaults from api/mesh.py,
and the audit log. All three remain OFF by default / OPAD opt-in. Docs only,
no behavior change. Append a matching dated entry to .claude/HISTORY.md.
* fix(p2p): final-review fixes — audit-log path, master-link peers_fn, wg_pubkey, handle_message hardening (#774)
* fix(p2p): implement mesh visualization tab — load on activation + master-link roles (#774)
The Mesh tab canvas was never rendered (the tab-switch handler didn't call
initMesh, and drawing into a hidden tab gave clientWidth 0 -> blank canvas).
Now: initMesh fires when the Mesh tab is activated, retries if the canvas has
no layout yet, and enriches the star graph with the live master-link state
(role/term/master + DHT peer count from /masterlink/topology + /dht/peers) —
center node coloured gold=MASTER / cyan=SATELLITE, peers by online status.
* docs(p2p): poster GPT + roadmap des évolutions DHT/federation/master-link; maj WIP/HISTORY/TODO (ref #774)
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Mistral Vibe <vibe@mistral.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
First slice of gondwana P1: make the annuaire the distributed directory.
Adds Op.NODE_PUBLISH/CONFIG_PUBLISH/CONFIG_REVOKE and the NodeRecord (signed
mesh peer registry, public wg key only) + ConfigBlob (signed, versioned,
single-writer config distribution) models — modelled on ServiceOffer.
Carries the consolidated clone/distribute/emancipate roadmap doc.
TDD: tests/test_directory.py (9 tests); full annuaire suite 136 green.
Analysis: gomitmproxy (unmaintained, dropped) vs martian/goproxy (Go) vs hudsucker
(Rust) vs Squid+ICAP, mapped to the 18-addon capability set. Recommendation: Go
hot-path core + retained Python analysis sidecars. Phased plan with shadow-run +
nft-DNAT-flip rollback (no big-bang cutover). Phase-1 PoC (packages/secubox-toolbox-ng,
stdlib-only): forge from ca-wg CA, 204-block, body-inject, SNI-splice, ClientHello/JA4
capture — go vet clean, tests green, arm64 cross-compile OK. NOT wired to live R3.
Per-package inspection of the mesh cluster found:
- secubox-mesh (Yggdrasil daemon control), secubox-meshname
(Meshname DNS resolver), secubox-daemon (Go mesh binary) — all
distinct, keep.
- secubox-mirror — miscategorized: it's an APT/CDN cache, not
mesh-related at all. Audit grouped it by prefix matching.
- secubox-master-link — effectively dead on a running system:
ships an 851-LOC API and frontend but no nginx config; its 21
API endpoints (/run/secubox/master-link.sock) are unreachable
from the web. The "master-link" UI operators actually see is
served by secubox-p2p (which has its own /master-link/status +
/master-link/token endpoints + ships the frontend to
/var/www/secubox/master-link/).
- secubox-p2p is the canonical master-link owner.
Real consolidation candidate: master-link → p2p fold. Bigger than
the mmpm/magicmirror fold (issue #381) because of asymmetric API
surface — master-link's 21 endpoints vs p2p's 2 existing
/master-link/* endpoints means the merge isn't mechanical. Defer
to a separate per-cluster issue with its own scope decisions.
Mesh cluster net reduction potential: -1 (after master-link → p2p
fold), not the -4 the original audit projected. Total realistic
floor unchanged from previous revision (~130-136 packages).
Per-package code inspection of the streamlit (already revised), dpi
(revised in 382), dns, and threats clusters all returned the same
finding: the audit's initial cluster groupings were based on naming
similarity, not architectural redundancy.
DNS cluster (5 → 5, was 5 → 1):
Five distinct DNS-layer subsystems, no shared config files. The
audit's "all overlap on /etc/resolv.conf and/or unbound" claim was
wrong — grep for /etc/resolv.conf in any of them returns zero hits.
Each touches its own config + data directory.
Threats cluster (7 → 7, was 7 → 1):
Seven distinct security capabilities with distinct backends
(crowdsec scenarios, nftables, dnsmasq logs, remote feeds, own
data dirs). Only the umbrella secubox-threats actually subscribes
to the CrowdSec/Suricata event bus the audit claimed all seven
did.
Tier 4 table revised to "candidates for inspection" — each cluster
needs the same per-package code check before any merge work.
Realistic-reduction floor updated from ~100 packages to ~130-136
(a 4-8% reduction from 141 rather than the originally projected 28%).
Where consolidation actually paid off: Tier 0 packaging bug (#378),
Tier 1 mail dead-code (#380), Tier 2 mmpm-magicmirror scaffolding
duplicate (#381), Tier 2 dpi description clarity (#382).
The pattern observation section explicitly flags this for future
work: most "fuzzy clusters" in out/clusters-fuzzy.txt are naming
overlaps, not merge targets. Treat the audit's cluster lists as
candidates for Description-clarity passes rather than packaging
merges.
After investigation in #382, the dpi cluster's four packages have
distinct operator roles, not duplicated ones:
- secubox-netifyd: daemon lifecycle dashboard
- secubox-dpi: analytics layer on top of netifyd
- secubox-ndpid: dedicated dashboard for the different nDPId engine
- secubox-mediaflow: streaming/VoIP consumer of DPI
The audit's original "dual-stream netifyd/nDPId" framing was
aspirational; the code is netifyd-only and nDPId analysis already
belongs to secubox-ndpid.
Real bug was a misleading Description on secubox-dpi (fixed in #382),
not a redundant-package situation. Keep all four packages.
Updates the realistic-reduction table: Tier 2 net is -1
(magicmirror only) rather than -3.
On closer inspection, secubox-streamlit and secubox-streamforge serve
distinct operator workflows that map to two different LuCI apps in the
OpenWrt source. Streamlit is the LXC-backed runtime that runs apps
(pulls in lxc + debootstrap + sudoers); streamforge is the
template/authoring tool (core deps only, 660-line FastAPI).
Merging would force every streamforge operator to install lxc and
debootstrap or require a complicated Recommends-vs-Depends split for
marginal benefit (net -1 package).
Also revises the realistic-reduction table: Tier 2 net is -3, not -4.
Adds scripts/audit-packages.py to scan packages/secubox-*/ and emit
structured artifacts under out/ (gitignored): packages.json,
packages.dot dependency graph, empty-packages.txt, prefix-based
clusters, and a hand-curated fuzzy-cluster report with per-member
evidence (deps, routes, services).
Findings doc 2026-05-27-secubox-consolidation-audit.md synthesises the
data into Tier 0-4 merge proposals grounded in real package contents,
not the plan stub's illustrative guesses. Key corrections vs the stub:
- 141 real packages (not "100+"); secubox-core is the substrate for 131
- streamlit-idle is already an in-package service variant, not a
separate package — that consolidation is half-done
- meta{blogizer,bolizer,catalog,oblizer} are real distinct services,
not typo proliferation (recommend rename for clarity, not merge)
- Two genuine packaging bugs surfaced: secubox-c3box binary name
collision between Python dashboard and Go daemon (issue #378),
secubox-smart-strip has no debian/ directory at all
Also imports two pre-existing 2026-05-26 plan stubs (consolidation,
build-scripts-refactor) that had been sitting untracked.
Re-run after any packaging change: python3 scripts/audit-packages.py
* docs(plan): #349 sentinelle-gsm v0.3.1 L3 + scoring + baseline (ref #349)
* feat(sentinelle-gsm): GsmtapListener exposes raw_l3 payload bytes (ref #349)
v0.3.0's Observation carried only header metadata (arfcn, frame_nr,
channel, sub_type), which was enough to prove the GSMTAP pipe but not
to drive L3-aware scoring. v0.3.1 needs the post-header payload to
decode BCCH System Information (LAC/CI/MCC/MNC) and CCCH paging
requests (IMSI/TMSI for the privacy-hashed subscriber digest).
Slice the L3 payload from the datagram via data[hdr["hdr_len"]:] in
_on_datagram() and surface it as a new Observation field
`raw_l3: bytes = b""`. Default empty bytes keeps every existing
caller backward compatible — only downstream decoders that opt in
will consume it (wired in Task 5).
Test: existing test_listener_receives_a_datagram now appends a known
18-byte payload and asserts obs.raw_l3 matches verbatim.
* feat(sentinelle-gsm): l3_decode — BCCH SI + CCCH paging request parsing (ref #349)
Pure-Python TLV decoder for the minimal L3 subset needed by the
IMSI-catcher scoring engine: BCCH System Information Types 3/4/6 and
CCCH Paging Request Types 1/2/3. No scapy dependency — the parser
operates directly on the post-GSMTAP-header bytes exposed by
Observation.raw_l3 (Task 1).
Privacy invariant : the public dataclasses (CellInfo, PagedIdentity,
ParsedPagingRequest, ParsedFrame) NEVER carry plaintext IMSI/TMSI/IMEI.
The internal _try_extract_mobile_id() returns plaintext bytes;
_parse_paging() immediately hands them to Anonymizer.anonymize() and
only the HMAC-truncated subscriber_hash escapes into PagedIdentity.
The plaintext goes out of scope at function return.
Permissive parsing : every helper returns an empty CellInfo() / None on
unexpected layouts rather than raising — real GSMTAP frames have
surprising structural variants and exceptions in the consume loop would
crash livemon ingestion.
Tests : 9 cases (6 from the plan + 3 added for coverage). BCD
encoder/decoder round-trip helpers build synthetic frames
programmatically rather than hand-crafting bytes, which also validates
the TS 24.008 §10.5.1.3/4 nibble ordering.
* feat(sentinelle-gsm): baseline.py — operator-baseline learning + cell_baseline table (ref #349)
Adds CellBaseline wrapper + cell_baseline table colocated in observations.db.
Cells graduate to baseline once observed >= LEARN_THRESHOLD (default 3) times.
Key design choices:
* LEARN_THRESHOLD = 3 by default — three sightings before a cell is
considered part of the operator's legitimate baseline.
* set_learn_mode(seconds) skips the count entirely — every cell observed
inside that explicit-learn window is INSERTed with learn_count = 3 and
graduates on first sighting (used by the operator's calibration sweep).
* CellBaseline accepts a raw sqlite3.Connection (NOT an ObservationsDB
instance) so it stays decoupled from observations.py's public surface
while still sharing the same .db file.
* cell_baseline table is created idempotently (IF NOT EXISTS) in
ObservationsDB.__init__, so pre-existing observations.db files keep
working after upgrade.
Privacy invariant: BaselineCell + cell_baseline have NO subscriber-id
columns — paged identities never enter the baseline path.
Tests (5/5 passing):
- test_first_consider_starts_at_1_not_baseline_yet
- test_three_considers_graduate_to_baseline
- test_learn_mode_graduates_first_consider
- test_consider_updates_metadata_via_coalesce
- test_list_orders_by_last_learned_desc
Full pkg sweep: 62 passing (57 prior + 5 new).
* feat(sentinelle-gsm): scoring_engine — 8 heuristics + thresholds (ref #349)
Replaces the v0.1 shape-only scoring.py stub with the full v0.3.1 detector
brain. ScoringEngine wires against CellBaseline + L3Decode (Tasks 2-3)
and runs every parsed frame through 8 heuristics, summing the triggered
contributions into a CellScore clamped to [0, 100].
Heuristics + default scores (sum-clamped to 100):
1. cipher_downgrade 40 A5/X observed < baseline.cipher_a5
2. ghost_bts 35 cell unknown to baseline + paging seen
3. identity_mismatch 30 observed (mcc, mnc) != baseline pair
4. relocalization_storm 25 > 3 distinct LACs in 60s on cell
5. identity_request_abuse 35 > 2 paging reqs in 300s per hash
6. anomalous_neighbours 15 neighbour list went non-empty -> empty
7. t3212_out_of_band 15 T3212 outside [1, 60] min
8. orphan_arfcn 20 FR carrier announces ARFCN outside plan
Rolling-window state (in-memory only, no DB persistence in v0.3.1):
_lac_window cell_id -> deque[(ts, lac)]
_idreq_window subscriber_hash -> deque[ts]
_neighbour_baseline cell_id -> set[int] (cumulative ARFCNs)
Thresholds:
* DEFAULT_THRESHOLDS class constant carries per-heuristic config dicts.
* update_thresholds() does a per-heuristic deep-merge so callers can
pass partial updates like {"cipher_downgrade": {"score": 55}} without
clobbering "enabled" or any other unspecified field.
* thresholds() returns a deep-copy so callers can't mutate engine state.
Privacy invariant kept: identity_request_abuse keys windows on
subscriber_hash strings (HMAC-trunc, supplied by l3_decode.Anonymizer),
never plaintext IMSI/TMSI. Reasons truncate the hash to 8 chars defensively.
France GSM-900 ARFCN plan is approximate and permissive: unknown
(mcc, mnc) pairs return triggered=False to avoid false-positives.
Dropped lib/sentinelle_gsm/scoring.py — the v0.1.0 shape-only stub
(empty Baseline + score()=0) is replaced. No other module imported it.
Tests:
* api/tests/test_scoring_engine.py — 11 tests, one per heuristic
(plus permissive-unknown-carrier guard), one for evaluate()
aggregation/clamping, one for update_thresholds() deep-merge.
* Full sweep: 73 passed (was 62; +11 new).
* feat(sentinelle-gsm): consume loop wires L3 + scoring + trusted match → Alert (ref #349)
- Extracted `_process_observation(obs)` helper for testability — the consume
coroutine now delegates per-frame work so tests can drive synthesized
Observations through the full pipeline without binding UDP.
- New endpoints: GET /baseline, POST /baseline/learn (sweep window),
GET /scoring/thresholds, POST /scoring/thresholds (deep-merge).
- Audit log entry on POST /scoring/thresholds via the existing
`secubox.sentinelle-gsm.api` logger — surfaces in /journal/stream.
- Alert emission decision tree:
score < threshold → drop (no write)
score >= threshold + match → one labeled alert per matched trusted phone
score >= threshold + no match → single anomaly-only alert
- ALERT_THRESHOLD configurable via SENTINELLE_ALERT_THRESHOLD env (default 60).
- TrustedRegistry now hashes the IMSI as `imsi.encode("ascii").hex()` so
the persisted token matches the canonical form produced by the L3
decoder's paging-request path. Existing trusted tests don't pin the
hash value, so behaviour is unchanged for read-back / lookup-by-id.
- 5 new integration tests at api/tests/test_alert_emission.py cover the
below-threshold drop, anomaly-only emission, trusted-match labeling,
paging-event hash persistence, and audit-log emission on threshold
update. Full sweep 78/78 passing.
* feat(sentinelle-gsm): webui baseline + scoring panels + trusted_label chip (ref #349)
Adds two new webui panels and a trusted-label chip on the alerts row:
- #baseline panel: Cell ID / MCC / MNC / LAC / ARFCN / cipher / learn count
/ last learned table backed by GET /api/v1/sensor/gsm/baseline. "Start
learn (5 min)" button arms POST /baseline/learn?sweep_seconds=300 and
re-polls the table every 15s for the 5-minute window so cells appear as
they graduate.
- #scoring panel: per-heuristic enable + score input (0..100, clamped
client-side) backed by GET/POST /scoring/thresholds. Form values are
cached in _thresholdState so any backend-only fields (heuristic-specific
knobs) survive the POST round trip.
- Alerts table: trusted_label is now rendered as a violet pill via the
new .trusted-chip class (replaces the older .target-pill markup for
the same data path).
Local navigation gets two new anchors (Baseline, Scoring) between
Observations and Actions. The 10s scan poll now also refreshes baseline
while a scan is running.
Wires 4 new event listeners (refresh-baseline, baseline-learn,
refresh-scoring, save-scoring). No backend changes; all endpoints already
exist (Tasks 1..5).
Tests: 78/78 passing.
* chore(sentinelle-gsm): bump 0.3.1 + extend privacy invariant tests (closes#349)
Append 4 new structural privacy tests under tests/test_privacy_invariant.py
to lock the v0.3.1 invariants in place:
* test_l3_decode_returns_no_plaintext_fields — CellInfo, PagedIdentity,
and ParsedPagingRequest dataclasses must not carry any plaintext
subscriber-id field (imsi/tmsi/imei/msisdn/iccid/subscriber_id).
* test_paging_request_hashes_paged_identities — feeds a synthetic
Paging Request Type 1 with a known TMSI through L3Decode and asserts
the plaintext TMSI bytes (as hex) NEVER appear in the resulting
PagedIdentity.subscriber_hash.
* test_cell_baseline_has_no_subscriber_fields — the BaselineCell
dataclass must remain operator-side metadata only: no subscriber_hash,
no plaintext identifier.
* test_scoring_engine_reasons_contain_no_plaintext_imsi — the free-text
reason strings produced by ScoringEngine.evaluate() must never echo
a 15-digit token (plaintext-IMSI shape) for typical observations.
These four tests pass against the v0.3.1 modules (Tasks 1-6) with no
code change — the modules were designed for these invariants from the
start, this commit just locks them in regression-test form.
Full sweep: 78 -> 82 passing.
Changelog: 0.3.0 -> 0.3.1 with the v0.3.1 entry summarising the new
l3_decode, baseline, scoring_engine modules, the consume-loop wiring
that emits qualified Alerts with trusted_label, the webui panels, and
the updated test breakdown (l3_decode x9, baseline x5, scoring_engine
x11, alert_emission x5, privacy x4 = 34 new, full sweep 82/82).
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
* docs(plan): #347 sentinelle-gsm v0.3.0 observation pipeline (ref #344#347)
* feat(sentinelle-gsm): livemon_runner — managed grgsm subprocess (closes#344 ref #347)
Introduces LivemonRunner, the asyncio subprocess manager that owns the
single grgsm_livemon_headless child for v0.3.0.
Device-claim invariant (the load-bearing point that closes#344): the
parent service NEVER opens the RTL-SDR itself. The RTL-SDR is claimed
ONLY by the spawned grgsm_livemon_headless child, and ONLY for the
lifetime of an active scan. start() spawns with `--args=rtl=0 -f <freq>`
so gr-osmosdr binds RTL-SDR instead of its UHD default; stop() sends
SIGTERM and falls back to SIGKILL after a 5 s timeout, releasing the
device cleanly.
Concretely this means ad-hoc tooling (rtl_test, rtl_adsb, kalibrate)
keeps working alongside an idle secubox-sentinelle-gsm service — the
race the v0.1 livemon.py USB-detect stub created is gone, because the
parent has no reason to touch the dongle outside an explicit scan.
Stderr from the child is drained into a 16 KiB-capped ring buffer so
status() can surface the last 2 KiB without unbounded memory growth,
and double-start() raises RuntimeError instead of silently leaking
processes.
Tests cover: initial-state, RTL args propagation to create_subprocess_exec,
double-start refusal, SIGTERM-on-stop + state clear, and stop-as-noop
when nothing is running. asyncio.create_subprocess_exec is fully mocked
— no real grgsm process is spawned by the test suite.
Test sweep: 5 new + 29 existing = 34 passing.
* feat(sentinelle-gsm): gsmtap_listener — async UDP 4729 + GSMTAP v2 parse (ref #347)
Task 2 of v0.3.0 observation pipeline.
- `lib/sentinelle_gsm/gsmtap_listener.py`: asyncio DatagramProtocol bound
to 127.0.0.1:4729 (configurable). Tiny manual 16-byte GSMTAP v2 header
parser — no scapy runtime dep on the hot path. scapy stays reserved
for the deeper L3 decode (BCCH/CCCH paging) landing in v0.3.1.
- `Observation` dataclass exposes ts/arfcn/frame_nr/channel/sub_type plus
optional lac/ci/mcc/mnc/cell_id/subscriber_hash. No plaintext-id field;
`subscriber_hash` is HMAC-only (populated when L3 decode lands).
- Bounded queue (maxsize=2048); on QueueFull we drop silently —
observations are stochastic, backpressure to the radio would be worse.
- 4 tests: minimum-valid parse, too-short reject, wrong-version reject,
end-to-end UDP roundtrip on port 47291 (avoids colliding with a
running grgsm on 4729).
All tests pass: 4/4 new, 38/38 sweep.
* feat(sentinelle-gsm): observations.py — SQLite sightings + paging_events (ref #347)
Adds ObservationsDB persistence layer for the v0.3 observation pipeline.
* Sighting (cell_id PK, mcc/mnc/lac/ci/arfcn, first_seen/last_seen,
sighting_count) — UPSERT semantics: INSERT on new cell_id, else bump
sighting_count + refresh last_seen with COALESCE on optional fields so
partial updates don't clobber previously-observed values.
* PagingEvent (ts, cell_id, subscriber_hash, request_type) — seeds the
paging history table that v0.3.1 will hash-match against trusted_phones.
Schema deliberately omits plaintext IMSI/TMSI/IMEI fields.
* _guard_plaintext() mirrors the regex guard in alert_sink.AlertSink.write
(same \b\d{15}\b pattern, same "plaintext-IMSI shape detected — refusing
write" error contract). Applied on every write path: upsert_sighting
guards cell_id; record_paging guards both cell_id and subscriber_hash.
* Two indexes on paging_events (ts, cell_id) for the v0.3.1 lookup path.
* 4 tests cover the contract: new-insert, bump-count, hash accept, and
the plaintext-IMSI refusal. Full sweep: 42 passing (38 prior + 4 new).
* feat(sentinelle-gsm): API — /scan/start /scan/stop /scan/status /observations (ref #347)
Task 4 of the v0.3.0 observation-pipeline plan. Wires the LivemonRunner +
GsmtapListener + ObservationsDB introduced in Tasks 1–3 to four new HTTP
endpoints behind the existing nginx + Authelia JWT gate.
- POST /scan/start {freq} → bind GSMTAP UDP listener, spawn
grgsm_livemon_headless on freq, start a
background consume task that drains
Observations into the SQLite store
- POST /scan/stop → cancel the consume task FIRST so no
orphan subprocess is left behind, then
SIGTERM the runner, then close the
listener socket (the CHILD owns the
RTL-SDR, never the parent)
- GET /scan/status → current runner state (running / pid /
freq / started_at / stderr_tail)
- GET /observations?limit → recent cell sightings, newest first;
limit clamped to [1, 1000], default 200
Synthetic cell_id formation until v0.3.1's L3 BCCH decode lands:
`f"arfcn-{obs.arfcn}-ch-{obs.channel}"`. The ObservationsDB schema
already accepts NULL for the operator quadruple (mcc/mnc/lac/ci) so
nothing downstream has to change when the real cell_id arrives.
Three new tests in api/tests/test_scan_api.py exercise the
happy path with mocked LivemonRunner + mocked GsmtapListener (so no
gr-gsm binary is spawned and no UDP socket is bound) plus a real
ObservationsDB under tmp_path so GET /observations exercises the
real SQLite path. Fixture clears `_consume_task` between tests to
keep them order-independent.
Test sweep: 42 (existing) + 3 (this) = 45 passing.
* feat(sentinelle-gsm): webui scan control + observations panels (ref #347)
Adds two new panels to the standalone MIND webui:
* Scan control — Start/Stop buttons, freq input (default 925.4M = E-GSM 900
ARFCN 975), 4-cell status row (state dot, freq, pid, started_at), and
a collapsible <details> showing the last 2 KB of stderr from
grgsm_livemon_headless. Start sends {freq: <value>} to /scan/start.
* Observations — live table of cell sightings with columns Cell ID,
ARFCN, MCC, MNC, LAC, CI, First seen, Last seen, Sightings. Badge in
the panel header tracks the current count.
JS additions (sentinelle.js):
* loadScanStatus / startScan / stopScan / loadObservations helpers
* 3 new event listeners (btnScanStart, btnScanStop, btnRefreshObs)
* Background interval (10s) that polls /scan/status always and
/observations only while a scan is running — keeps the post-scan
terminal state on screen without hammering sqlite.
CSS additions (sentinelle.css):
* .scan-status-row (4-col grid inside a panel, matches .status-strip)
* .scan-stderr / .scan-stderr-details (monospace tail block, MIND palette)
* Responsive collapse to 2 cols then 1 col matching the existing breakpoints.
API surface untouched — Tasks 1-4 already shipped /scan/start, /scan/stop,
/scan/status and /observations under /api/v1/sensor/gsm/. Test sweep still
45 passing.
* build(sentinelle-gsm): merge detect_rtlsdr_usb into livemon_runner; drop v0.1 stub; add scapy + gr-gsm deps (ref #344#347)
Co-locate RTL-SDR USB detection with the grgsm_livemon orchestrator.
- Move RTLSDR_USB_IDS + detect_rtlsdr_usb() from livemon.py into
livemon_runner.py. The function is pure sysfs (reads
/sys/bus/usb/devices/*/idVendor + idProduct) — it does NOT claim
the USB device, so it was never the source of #344. The #344 bug
was fixed by Task 1's LivemonRunner model (child-only device claim
via subprocess.create_subprocess_exec).
- Delete lib/sentinelle_gsm/livemon.py — the v0.1 stub also carried a
useless NotImplementedError docstring for listen_gsmtap_udp, fully
superseded by gsmtap_listener.py (Task 1).
- api/main.py: collapse the two-line "from sentinelle_gsm.livemon ...
/ from sentinelle_gsm.livemon_runner ..." block into a single import
pulling LivemonRunner + detect_rtlsdr_usb from livemon_runner.
- debian/control: promote python3-scapy from Recommends to Depends
(runtime requirement once the v0.3.1 L3 decode path ships); keep
gr-gsm in Recommends (subprocess that runs scans).
dpkg-buildpackage: ok (secubox-sentinelle-gsm_0.2.3-1~bookworm1_all.deb).
dpkg-deb -c | grep livemon: only livemon_runner.py + test_livemon_runner.py,
no v0.1 stub. pytest api/tests/ tests/: 45 passed.
* chore(sentinelle-gsm): bump 0.3.0 + extend privacy invariant tests (closes#344#347)
Privacy invariant coverage extended for the v0.3.0 observation pipeline:
* test_observation_has_no_plaintext_fields — Observation dataclass
(gsmtap_listener) exposes ONLY subscriber_hash, never plaintext
IMSI/TMSI/IMEI/MSISDN/ICCID/subscriber_id.
* test_paging_event_only_stores_hash — PagingEvent dataclass
(observations) has no plaintext-identifier fields and exposes
subscriber_hash.
* test_observations_db_refuses_plaintext_imsi — ObservationsDB.record_paging
raises ValueError("plaintext-IMSI ...") when handed a 15-digit
subscriber_hash (write-time shape guard).
Version bump 0.2.3 → 0.3.0 with the full v0.3.0 changelog:
livemon_runner.py (RTL-SDR subprocess wrapper, closes#344) +
gsmtap_listener.py (asyncio UDP 4729) + observations.py (SQLite
sightings/paging_events) + /scan API + webui panels + scapy promoted
to Depends + dropped v0.1 livemon.py stub. 48/48 tests passing.
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
