mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-28 21:17:36 +00:00
* docs: spec — Sentinel C2/botnet auto-learning (high-precision, report-only) (ref #823) * docs: implementation plan — Sentinel C2/botnet auto-learning (ref #823) * feat(sentinel): C2 autolearn false-positive allowlist gate (ref #826) * fix(sentinel): serialize c2allow Add + robust atomicWriteFile (relative-path + temp cleanup) (ref #826) * feat(sentinel): C2 autolearn corroborating signals (rare/non-browser-JA/DGA) (ref #826) * feat(sentinel): C2 autolearn candidate lifecycle + persistence (ref #826) * fix(sentinel): c2cand deep-copy snapshots (persist/snapshot/record race) + evict order + device cap (ref #826) * feat(sentinel): C2 autolearn orchestrator — learned set + report-only verdicts (ref #826) * feat(sentinel): wire C2 learner into daemon + /c2 endpoints + seeded allowlist (ref #826) * fix(sentinel): make /c2/allow writable in packaged deploy (RW path + ownership) + sanitize Add (ref #826) POST /c2/allow could never write in production: sbx-sentinel.service runs User=secubox-toolbox under ProtectSystem=strict + ReadOnlyPaths=/etc/secubox, and the seeded c2-allow.txt ships root:root. Fix both halves: nest ReadWritePaths=/etc/secubox/sentinel under the existing ReadOnlyPaths=/etc/secubox in the unit, and chown/chmod 0750 the sentinel/ subdir + file (never the shared /etc/secubox parent) in postinst's configure branch, fail-safe. Also: fix the /c2/allow doc comment (form/query only, not JSON — r.FormValue never parses a body), and sanitize C2Allow.Add against newline injection now that host is network-reachable. * feat(toolbox): WebUI C2 appris sub-view + Ignorer allowlist proxy (ref #826) * fix(toolbox): C2 view XSS (data-host not onclick) + normalize candidate signal maps (ref #826) * fix(sentinel): bound c2 beaconing/learned maps + drop promoted from candidates + reject bare-TLD allow (ref #826) --------- Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
This commit is contained in:
CyberMind
GitHub
parent
916cb91b72
commit
62e8631f2b
|
|
@ -1,140 +1,101 @@
|
|||
# Task 5 Report: Packaging mesh-exclusion timers (#806)
|
||||
# Task 5 Report: C2 learner daemon wiring + `/c2` endpoints + seeded config (#826)
|
||||
|
||||
**Date**: 2026-07-04
|
||||
**Status**: ✅ **COMPLETE**
|
||||
**Status**: COMPLETE
|
||||
**Commit**: `db22668f` — `feat(sentinel): wire C2 learner into daemon + /c2 endpoints + seeded allowlist (ref #826)`
|
||||
|
||||
## Test summary
|
||||
|
||||
```
|
||||
cd packages/secubox-toolbox-ng && go test ./cmd/sbx-sentinel/... ./internal/sentinel/...
|
||||
ok .../secubox-toolbox-ng/cmd/sbx-sentinel 0.453s
|
||||
ok .../secubox-toolbox-ng/internal/sentinel 0.886s
|
||||
```
|
||||
`go build ./...` clean. `gofmt -l` clean.
|
||||
|
||||
## What changed
|
||||
|
||||
- `cmd/sbx-sentinel/main.go`: `buildAnalyzers` now wraps `sentinel.NewBehavioral()` in `sentinel.NewC2Learner(..., sentinel.C2Config{...5 SENTINEL_C2_* env vars via getenvDefault...})` and appends the `C2Learner` (not the raw `Behavioral`) to the pipeline — analyzer count is still 3 (spyware, c2-learner, yara). Added package-level `var c2Learner *sentinel.C2Learner` (set inside `buildAnalyzers`) so `run()`'s optional status-HTTP goroutine can pass it to `serveStatus`. Added fail-safe `readLinesFile(path) []string` (missing/unreadable → nil; skips blank/`#`-comment lines) used for `SENTINEL_C2_BROWSER_JA4`.
|
||||
- `cmd/sbx-sentinel/http.go`: `newStatusMux(store, c2)` and `serveStatus(ctx, addr, store, c2)` signatures extended with `*sentinel.C2Learner`. When `c2 != nil`, registers `GET /c2/learned`, `GET /c2/candidates`, `POST /c2/allow` (form/JSON `host`, 400 on empty, 500 on `c2.Allow` error, else `{"ok":true}`). Nil `c2` → routes simply not registered (no panic, no behavior change for existing `/stats`/`/verdicts`).
|
||||
- `cmd/sbx-sentinel/http_test.go`: updated the 4 pre-existing `newStatusMux(store)` call sites to `newStatusMux(store, nil)`; added `TestC2Endpoints` (GET learned/candidates → 200, POST allow with `host=fp.example` → 200).
|
||||
- `cmd/sbx-sentinel/wiring_test.go`: **incidental fix required to keep tests green** — `TestBuildAnalyzersReturnsThree` asserted a `*sentinel.Behavioral` type was present in the returned slice; since `buildAnalyzers` now appends the wrapping `*sentinel.C2Learner` instead, updated the type-switch case to check for `*sentinel.C2Learner`. Not in the original brief's file list but necessary for the existing test suite to still compile/pass — `TestBuildAnalyzersLoadsBasePack` and `TestDefaultConfigWiresPipeline` needed no changes (analyzer count stays 3; the spyware verdict path they exercise is unaffected).
|
||||
- `debian/c2-allow.txt` (new) + `debian/browser-ja4.txt` (new): seed files, verbatim per brief.
|
||||
- `debian/rules`: added `install -d .../etc/secubox/sentinel` + two `install -m 0644` lines for the two seeds (allow file under `/etc/secubox/sentinel/`, browser-JA4 under the already-created `/usr/share/secubox/sentinel/`).
|
||||
- `debian/sentinel.env`: appended the 5 `SENTINEL_C2_*` vars with defaults matching the code, inserted before the "Live feed source URLs" section.
|
||||
- No tmpfiles change needed: `/var/lib/secubox/sentinel` (candidates/learned JSON) is already created 0750 secubox-toolbox by the existing `tmpfiles/zz-secubox-sentinel.conf`.
|
||||
|
||||
## Blocking concerns
|
||||
|
||||
None. `git add` was scoped to only `cmd/sbx-sentinel/` and the `debian/` files touched — two unrelated pre-existing modified files in this worktree (`.superpowers/sdd/task-2-report.md`, `task-4-report.md`, apparently from a different/earlier session reusing this worktree) were left untouched and unstaged.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
## Review-fix pass — `/c2/allow` writability in packaged deploy (2026-07-07)
|
||||
|
||||
Successfully packaged the mesh-exclusion publish + sync CLI scripts as systemd timers with installation and enablement on postinst.
|
||||
**Status**: COMPLETE
|
||||
**Commit**: `<see final report below>` — `fix(sentinel): make /c2/allow writable in packaged deploy (RW path + ownership) + sanitize Add (ref #826)`
|
||||
|
||||
## Commit
|
||||
Three review findings on the Task 5 work fixed:
|
||||
|
||||
**Hash:** `d6076912`
|
||||
**Message:** `feat(toolbox): package mesh-exclusion publish+sync timers (ref #806)`
|
||||
### Finding 1 (CRITICAL) — `/c2/allow` could never write in production
|
||||
|
||||
---
|
||||
`sbx-sentinel.service` runs `User=secubox-toolbox` under `ProtectSystem=strict` with
|
||||
`ReadOnlyPaths=/etc/secubox`, and the seeded `c2-allow.txt` ships root:root — so
|
||||
`C2Allow.Add`'s write to `/etc/secubox/sentinel/c2-allow.txt` would fail (EROFS from
|
||||
the mount namespace and/or EACCES from ownership). Fixed both halves:
|
||||
|
||||
## Changes Made
|
||||
- `debian/sbx-sentinel.service`: added `ReadWritePaths=/etc/secubox/sentinel` (nested
|
||||
under the existing `ReadOnlyPaths=/etc/secubox`, which systemd allows — re-grants
|
||||
write to just that subtree, rest of `/etc/secubox` stays read-only).
|
||||
- `debian/postinst` (`configure` branch, right after the existing daemon-reload /
|
||||
no-enable comment block): added a fail-safe block that `chown`s
|
||||
`/etc/secubox/sentinel` + `c2-allow.txt` to `secubox-toolbox:secubox-toolbox` and
|
||||
`chmod 0750` the dir. **`/etc/secubox` itself is never touched** — only the
|
||||
`sentinel/` subdir and its file, consistent with the project's shared-parent
|
||||
traversal constraint (parent must stay 0755).
|
||||
|
||||
### 1. Created 4 systemd unit files
|
||||
### Finding 2 (Important) — doc comment claimed JSON support that doesn't exist
|
||||
|
||||
Under `packages/secubox-toolbox/systemd/`:
|
||||
`cmd/sbx-sentinel/http.go`'s package doc said `POST /c2/allow` accepts "form/JSON
|
||||
`host`", but the handler only calls `r.FormValue("host")` (form-encoded/query only,
|
||||
no JSON body parsing — and none was added, since the portal already posts
|
||||
form-encoded). Corrected the comment to say x-www-form-urlencoded/query only.
|
||||
|
||||
- `secubox-toolbox-mesh-exclusion-publish.service`
|
||||
- `secubox-toolbox-mesh-exclusion-publish.timer`
|
||||
- `secubox-toolbox-mesh-exclusion-sync.service`
|
||||
- `secubox-toolbox-mesh-exclusion-sync.timer`
|
||||
### Finding 3 (Minor) — newline injection in `C2Allow.Add`
|
||||
|
||||
All unit files created with exact specifications from brief:
|
||||
`internal/sentinel/c2allow.go`'s `Add` wrote `host` as a raw line with no
|
||||
sanitization, so a network caller posting `host=good.com\nevil.com` could inject a
|
||||
second allowlist entry. Since this is now reachable over the network via
|
||||
`POST /c2/allow`, added a guard: `Add` rejects (returns nil, no write — fail-safe,
|
||||
not an error) any host containing `\n`, `\r`, or a space. Added regression test
|
||||
`TestC2AllowAddRejectsInjection` to `c2allow_test.go`.
|
||||
|
||||
- Service units with `Type=oneshot`, `ExecStart` pointing to `/usr/sbin/` scripts
|
||||
- Timer units with `OnBootSec`, `OnUnitActiveSec=30min`, `Persistent=true`, `RandomizedDelaySec=3min`
|
||||
- Proper `After=` dependencies on `secubox-toolbox.service` and `secubox-annuaire.service`
|
||||
|
||||
### 2. Modified `packages/secubox-toolbox/debian/rules`
|
||||
|
||||
Added 6 install lines in `override_dh_installsystemd` after autolearn timer lines:
|
||||
|
||||
- 2 lines to install the sbin scripts (publish, sync)
|
||||
- 4 lines to install the systemd unit files (.service and .timer files)
|
||||
|
||||
Placement matches existing patterns for similar helpers (autolearn, tor, blacklist).
|
||||
|
||||
### 3. Modified `packages/secubox-toolbox/debian/postinst`
|
||||
|
||||
Added 4 enable+start lines inside the systemd conditional block:
|
||||
|
||||
```sh
|
||||
systemctl enable secubox-toolbox-mesh-exclusion-publish.timer 2>/dev/null || true
|
||||
systemctl start secubox-toolbox-mesh-exclusion-publish.timer 2>/dev/null || true
|
||||
systemctl enable secubox-toolbox-mesh-exclusion-sync.timer 2>/dev/null || true
|
||||
systemctl start secubox-toolbox-mesh-exclusion-sync.timer 2>/dev/null || true
|
||||
```
|
||||
|
||||
Placement after autolearn timer enables, following existing convention with `2>/dev/null || true` guards.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**Command executed:**
|
||||
|
||||
```bash
|
||||
ls packages/secubox-toolbox/systemd/secubox-toolbox-mesh-exclusion-*.{service,timer}
|
||||
grep -c mesh-exclusion packages/secubox-toolbox/debian/rules packages/secubox-toolbox/debian/postinst
|
||||
```
|
||||
|
||||
**Output:**
|
||||
### Verification
|
||||
|
||||
```
|
||||
packages/secubox-toolbox/systemd/secubox-toolbox-mesh-exclusion-publish.service
|
||||
packages/secubox-toolbox/systemd/secubox-toolbox-mesh-exclusion-publish.timer
|
||||
packages/secubox-toolbox/systemd/secubox-toolbox-mesh-exclusion-sync.service
|
||||
packages/secubox-toolbox/systemd/secubox-toolbox-mesh-exclusion-sync.timer
|
||||
---
|
||||
packages/secubox-toolbox/debian/rules:6
|
||||
packages/secubox-toolbox/debian/postinst:4
|
||||
cd packages/secubox-toolbox-ng && go test ./internal/sentinel/ -run TestC2Allow -race -v
|
||||
=== RUN TestC2AllowSuffixAndLan
|
||||
--- PASS: TestC2AllowSuffixAndLan (0.00s)
|
||||
=== RUN TestC2AllowFailSafeMissingFiles
|
||||
--- PASS: TestC2AllowFailSafeMissingFiles (0.00s)
|
||||
=== RUN TestC2AllowAddAppends
|
||||
--- PASS: TestC2AllowAddAppends (0.00s)
|
||||
=== RUN TestC2AllowAddRejectsInjection
|
||||
--- PASS: TestC2AllowAddRejectsInjection (0.00s)
|
||||
=== RUN TestC2AllowAddConcurrent
|
||||
--- PASS: TestC2AllowAddConcurrent (0.00s)
|
||||
PASS
|
||||
ok github.com/CyberMind-FR/secubox-deb/secubox-toolbox-ng/internal/sentinel 1.017s
|
||||
```
|
||||
|
||||
**Result**: ✅ **PASSED**
|
||||
`go build ./...` clean (exit 0). Confirmed via grep:
|
||||
`ReadWritePaths=/etc/secubox/sentinel` present in `debian/sbx-sentinel.service`;
|
||||
`etc/secubox/sentinel` chown/chmod block present in `debian/postinst`'s `configure`
|
||||
branch, guarded fail-safe (`|| true` throughout, dir-existence check first).
|
||||
|
||||
- 4 unit files present
|
||||
- rules: 6 matches (exactly required)
|
||||
- postinst: 4 matches (exactly required)
|
||||
### Files changed
|
||||
|
||||
---
|
||||
|
||||
## Integration Notes
|
||||
|
||||
- The 2 CLI scripts (`secubox-toolbox-mesh-exclusion-publish` and `secubox-toolbox-mesh-exclusion-sync`) from Tasks 2–3 exist and are correctly referenced by the service units
|
||||
- Install paths follow exact pattern of nearby helpers (e.g., `debian/secubox-toolbox/usr/sbin/`, `debian/secubox-toolbox/lib/systemd/system/`)
|
||||
- Postinst enable/start lines placed in same conditional block as existing timer enables
|
||||
- No modifications outside `packages/secubox-toolbox/` per requirements
|
||||
|
||||
---
|
||||
|
||||
## Ready for Deploy
|
||||
|
||||
Task 5 is complete and committed. The systemd timer units are now:
|
||||
|
||||
1. Packaged into `secubox-toolbox.deb`
|
||||
2. Installed at `dpkg install` time
|
||||
3. Enabled and started in postinst
|
||||
|
||||
Next steps (manual, per brief Deploy section):
|
||||
|
||||
1. Cross-compile sbxmitm arm64 binary
|
||||
2. Deploy sbxmitm binary to gk2/c3box/amd64
|
||||
3. Rsync the 2 CLI scripts to all nodes
|
||||
4. `systemctl daemon-reload` on all nodes
|
||||
5. Verify timers fire: `systemctl status secubox-toolbox-mesh-exclusion-*.timer`
|
||||
|
||||
---
|
||||
|
||||
## #806 Final Whole-Branch Review — Fix Wave
|
||||
|
||||
**Date**: 2026-07-04
|
||||
**Status**: ✅ **COMPLETE**
|
||||
**Commit**: `c6257154` — `fix(toolbox): mesh-exclusion churn guard + never-raises + env overrides + fed-disabled enabled flag (ref #806)`
|
||||
|
||||
### Test command + result
|
||||
|
||||
```
|
||||
cd packages/secubox-toolbox && PYTHONPATH=. python -m pytest tests/test_mesh_exclusion_publish.py tests/test_mesh_exclusion_sync.py tests/test_filter_list_mesh_tag.py -q
|
||||
```
|
||||
```
|
||||
.......... [100%]
|
||||
10 passed in 0.36s
|
||||
```
|
||||
(full `tests/` run: 209 passed, 3 pre-existing/unrelated failures confirmed present before this change via `git stash` — `test_bypass_sources.py::test_load_bypass_tagged_missing_source_skipped` (stale pre-#809 assertion shape) and 2x `test_media_stats.py` (`ModuleNotFoundError: secubox_core` in this local venv) — not touched, out of scope.)
|
||||
|
||||
### Fixes applied
|
||||
|
||||
1. **Publish churn guard** — `mesh_exclusion.py`: added `LAST_PUBLISHED` path + `_read_last_published()`/`_write_last_published()`; `publish()` now computes the content hash first and returns `True` without POSTing when it matches the last successfully-published hash, only persisting the new fingerprint after a successful POST. TDD: added `test_publish_skips_when_payload_unchanged` (red → green).
|
||||
2. **`_atomic_write` + decode safety** — wrapped `_atomic_write`'s write/replace body in try/except (returns `False` on any error instead of raising); broadened `except OSError` → `except Exception` in `_read_list` and `node_id` so a non-UTF-8/decode error can't escape the best-effort boundary.
|
||||
3. **Env overrides in `mesh_exclusion.py`** — `LOCAL_SPLICE`/`LOCAL_BYPASS`/`LOCAL_DISABLED`/`FED_SPLICE`/`FED_BYPASS`/`FED_DISABLED` now read `os.environ.get(...)` with the same var names as `policy.go`/`api.py` (`SECUBOX_SPLICE_LEARNED`, `SECUBOX_BYPASS_DYNAMIC`, `SECUBOX_FILTER_DISABLED`, `SECUBOX_FED_SPLICE`, `SECUBOX_FED_BYPASS`, `SECUBOX_FED_DISABLED`), same default paths, no divergence possible.
|
||||
4. **Fed-disabled → `enabled` flag** — `api.py`: factored `_read_disabled_file(path)` and made `_load_disabled()` return the union of the local `MITM_FILTER_DISABLED_FILE` and `FED_DISABLED_FILE`, so a fleet-wide-disabled pattern (mesh-disabled row) now renders `enabled=False` in Filtres MITM, matching the R3 engine's `disabledLocal ∪ disabledFed`. TDD: added `test_fed_disabled_pattern_shows_enabled_false`.
|
||||
|
||||
No public names/signatures changed. Scope limited to `packages/secubox-toolbox/`.
|
||||
- `packages/secubox-toolbox-ng/debian/sbx-sentinel.service`
|
||||
- `packages/secubox-toolbox-ng/debian/postinst`
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2allow.go`
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2allow_test.go`
|
||||
- `packages/secubox-toolbox-ng/cmd/sbx-sentinel/http.go` (doc comment only)
|
||||
|
|
|
|||
1505
docs/superpowers/plans/2026-07-07-sentinel-c2-autolearn.md
Normal file
1505
docs/superpowers/plans/2026-07-07-sentinel-c2-autolearn.md
Normal file
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,229 @@
|
|||
# Sentinel C2/Botnet Auto-Learning — Design
|
||||
|
||||
**Date:** 2026-07-07
|
||||
**Related:** #823 (Sentinel engine), builds on the live activation (#825)
|
||||
**Status:** Design — pending user review
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Automatically learn C2/botnet indicators from observed traffic and grow a
|
||||
learned indicator overlay, with **high-precision false-positive suppression** so
|
||||
legitimate periodic traffic (mail/webmail polling, admin WebUI dashboards,
|
||||
monitoring/health checks, NTP, update-checkers) is never learned as C2.
|
||||
|
||||
**Report-only.** Learned indicators carry `action=report`; `FinalizeAction`
|
||||
keeps behavioral/learned classes report-only forever — never auto-block.
|
||||
|
||||
Non-goal: new *detection* of beaconing (the `Behavioral` engine already fires
|
||||
`ClassBotnetC2` beacon verdicts). This adds a **promotion layer + FP gate** that
|
||||
turns sustained, corroborated beaconing into persistent learned indicators.
|
||||
|
||||
---
|
||||
|
||||
## Background — what already exists
|
||||
|
||||
- `internal/sentinel/behavioral.go`: the beaconing detector. Per
|
||||
`(MacHash, Host)` it tracks inter-arrival times; once `beaconMinHits=6` hits
|
||||
with coefficient-of-variation `≤ beaconJitterCV=0.20`, it fires one
|
||||
`ClassBotnetC2` verdict (`sev/conf 70`, `ActionReport`, evidence
|
||||
`{pattern:beaconing, host, interval_s}`), latched once per key.
|
||||
- `internal/sentinel/pack.go`: `Loader` globs an overlay directory of `*.json`
|
||||
packs, `MergePacks` merges base + overlays (overlay entry overrides base on
|
||||
same `(Type,Value)`), hot-reload with a throttle, fail-closed on base-dir
|
||||
disappearance. Overlay dir on gk2 = `/var/lib/secubox/sentinel/overlay`.
|
||||
- IOC pack entry shape (`botnet.json`): `{type, value, class, severity, source,
|
||||
action}`. `type ∈ {domain, ip, ja3, ...}`.
|
||||
- `FlowMeta` carries: `Host, URL, ClientIP, JA3, JA4, CertSHA1, FileSHA256,
|
||||
MacHash`. (No Referer — signals are computed from these fields only.)
|
||||
- `cmd/sbxmitm/adcand*.go`: the existing autolearn candidate→confirm pattern
|
||||
(records a candidate host with a hits count, an allowlist, and first-party
|
||||
suppression) — the model this mirrors.
|
||||
- The live-feed fetcher `secubox-sentinel-feeds` writes overlay packs atomically
|
||||
(write-temp + validate non-empty JSON + rename) — the atomic-write model this
|
||||
reuses.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
A new package unit `internal/sentinel/c2learn.go` (`C2Learner`) sits downstream
|
||||
of `Behavioral`. It is an `Analyzer` wrapper / post-step in the daemon: it never
|
||||
blocks the analyze path, is fully fail-safe, and bounded in memory.
|
||||
|
||||
```
|
||||
mirror → Behavioral(beacon verdict) ─┐
|
||||
├─▶ C2Learner.Observe(flow, beaconVerdict)
|
||||
FlowMeta ────────────────────────────┘ │
|
||||
FP-gate (c2allow) ──drop──▶ (suppressed) │
|
||||
signals (rare / non-browser JA4 / DGA) ─────┤
|
||||
candidate store (c2cand) ───────────────────┤
|
||||
sustained + corroborated? ──▶ promote ───▶ learned-c2.json (overlay)
|
||||
│
|
||||
Loader hot-reload → Gate match (report)
|
||||
│
|
||||
/stats · /verdicts · 3 surfaces
|
||||
```
|
||||
|
||||
### Component 1 — FP gate (`c2allow.go`)
|
||||
|
||||
`Allowed(host) bool` — the host is **allowlisted (never learned)** if it matches
|
||||
ANY of:
|
||||
- **box's own vhosts / admin domains** — read from a box-domains source at
|
||||
startup + on reload. Source: the HAProxy routes file
|
||||
(`/etc/secubox/waf/haproxy-routes.json`) keys and/or a seeded
|
||||
`box-domains.txt`; both optional/fail-safe. This is the primary "admin WebUI"
|
||||
suppressor.
|
||||
- **first-party / LAN** — `Host` whose registrable domain equals the box's own
|
||||
registrable domain, or a `Host` that is an RFC1918 / loopback / link-local IP
|
||||
literal.
|
||||
- **seeded operator allowlist** `c2-allow.txt` — one host/suffix per line,
|
||||
pre-seeded with: mail/webmail providers, monitoring/health endpoints, NTP,
|
||||
OS/browser update + telemetry, and major CDNs. Operator-editable; hot-reloaded.
|
||||
- (optional, bundled) a high-reputation top-domains list — suffix match.
|
||||
|
||||
Matching is suffix-aware (`sub.example.com` matches an `example.com` entry).
|
||||
All sources are fail-safe: a missing/corrupt file contributes no entries (logs
|
||||
once), never errors the learner.
|
||||
|
||||
### Component 2 — corroborating signals (`c2signal.go`)
|
||||
|
||||
Computed from `FlowMeta` for the beaconing host. Promotion requires **≥1**:
|
||||
- **rare/unknown destination** — `Host` global first-seen recently / very low
|
||||
cumulative hit-frequency across the daemon's observation window (a bounded
|
||||
frequency map). A long-established destination is not "rare".
|
||||
- **non-browser JA3/JA4** — `JA4` (fallback `JA3`) not in a bundled
|
||||
known-browser fingerprint set. An admin dashboard polled by a real browser has
|
||||
a browser JA4 → this signal does NOT fire → suppressed. Empty JA4 (non-TLS or
|
||||
unknown) is treated as *unknown*, not automatically "non-browser" (avoid FP on
|
||||
missing data) — configurable.
|
||||
- **DGA-ish / high-entropy domain** — Shannon entropy of the registrable label
|
||||
above a threshold, or long random-looking label. Reuses the entropy helper
|
||||
style from `behavioral.go`'s one-time-link check.
|
||||
|
||||
### Component 3 — candidate store + confirm (`c2cand.go`)
|
||||
|
||||
Bounded (LRU-capped) map keyed by `Host`:
|
||||
`{host, firstSeen, lastSeen, windows, distinctDevices set, signals set,
|
||||
meanInterval}`. A beacon that passes the gate AND has ≥1 signal
|
||||
records/updates a candidate. **Promotion criteria (all required):**
|
||||
- **sustained**: observed across `≥ c2MinWindows` (e.g. 3) separate beacon
|
||||
windows spanning `≥ c2MinSpan` (e.g. 30 min) — not one burst;
|
||||
- **corroborated**: `≥1` corroborating signal present (see Component 2);
|
||||
- distinct-device count raises the recorded confidence (more devices → higher),
|
||||
but a single device still promotes (targeted implant).
|
||||
|
||||
Persisted to `/var/lib/secubox/sentinel/c2-candidates.json` (atomic write) so
|
||||
candidates survive daemon restarts. LRU + TTL bounded.
|
||||
|
||||
### Component 4 — promotion to learned overlay
|
||||
|
||||
On confirm, upsert the host into
|
||||
`/var/lib/secubox/sentinel/overlay/learned-c2.json`:
|
||||
`{type:"domain", value:<host>, class:"botnet_c2", severity:75, action:"report",
|
||||
source:"autolearn", evidence:{signals, interval_s, devices}}`.
|
||||
- **Atomic** write-temp-rename; validated non-empty JSON before rename (reuse
|
||||
the feeds fetcher's fail-safe model).
|
||||
- **Deduped** by host; **size-capped** (e.g. ≤ 2000 entries, evict oldest).
|
||||
- **TTL decay**: an entry not re-confirmed within `c2LearnedTTL` (e.g. 30 days)
|
||||
is aged out on the next write — the list self-cleans when a host goes quiet.
|
||||
- The existing `Loader` hot-reloads the overlay → the `Gate` matches the host
|
||||
(report-only) → verdicts flow to `/stats`, `/verdicts`, and all three surfaces
|
||||
with `class=botnet_c2, source=autolearn`.
|
||||
|
||||
### Component 5 — surfaces + safety valve
|
||||
|
||||
- The daemon status HTTP gains read endpoints:
|
||||
`GET /c2/learned` and `GET /c2/candidates` (bounded, read-only, localhost).
|
||||
- The portal proxies them (`/admin/sentinel/c2`), and the WebUI Sentinelle tab
|
||||
gains a **"C2 appris"** sub-view listing learned + candidate hosts with their
|
||||
signals/interval/devices.
|
||||
- Each entry has **"Ignorer (allowlist)"** → `POST /admin/sentinel/c2/allow`
|
||||
(host) → the daemon appends the host to `c2-allow.txt` and removes it from the
|
||||
learned overlay + candidate store. One click corrects a rare FP; the allowlist
|
||||
addition prevents re-promotion.
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
- Fully fail-safe: the learner never blocks or panics on the analyze path;
|
||||
malformed `FlowMeta` fields make a signal a no-op.
|
||||
- All file reads (allowlist, box-domains, candidates, overlay) fail to
|
||||
empty/no-op on missing/corrupt input, logged once — never fatal.
|
||||
- All file writes atomic (temp + rename); a write failure is logged and dropped,
|
||||
never corrupts the live overlay.
|
||||
- Bounded memory: LRU caps on the candidate map and the frequency map.
|
||||
- Report-only invariant enforced at the source (`action:report` on every learned
|
||||
entry) AND by `FinalizeAction` (`ClassBotnetC2` is heuristic → report).
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
- **FP gate**: box-vhost host suppressed; first-party (box registrable) host
|
||||
suppressed; RFC1918 IP host suppressed; a `c2-allow.txt` entry (and its
|
||||
subdomains) suppressed; a browser-JA4 beacon to a known domain not learned.
|
||||
- **Multi-signal**: a sustained beacon to an *unknown-but-browser-JA4* host with
|
||||
no other signal → NOT promoted (periodicity alone insufficient); the same host
|
||||
with a non-browser JA4 → candidate.
|
||||
- **Lifecycle**: one burst (single window) → candidate, not promoted; sustained
|
||||
across `≥ c2MinWindows` over `≥ c2MinSpan` + ≥1 signal → promoted; distinct
|
||||
devices raise recorded confidence.
|
||||
- **Promotion**: learned-c2.json written atomically, deduped, capped; a
|
||||
never-re-confirmed entry past TTL is evicted on next write; overlay hot-reload
|
||||
makes the Gate match the host (report-only).
|
||||
- **Safety valve**: allow-add removes the learned entry + candidate and prevents
|
||||
re-promotion; `c2-allow.txt` gets the host.
|
||||
- **Report-only**: no learned/behavioral path ever yields `action=block`.
|
||||
- **Fail-safe**: missing/corrupt allowlist, box-domains, candidates, overlay →
|
||||
learner still runs, no panic, no false promotion from garbage.
|
||||
- **Surfaces**: `/c2/learned` + `/c2/candidates` shapes; portal proxy fail-safe
|
||||
(daemon dark → empty, HTTP 200); WebUI sub-view renders learned/candidate/empty
|
||||
states; escaping on all daemon-supplied host strings.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2learn.go` — **create**
|
||||
(orchestrator: Observe + promote).
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2allow.go` — **create** (FP gate).
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2signal.go` — **create**
|
||||
(rare / non-browser-JA4 / DGA signals + bounded frequency map).
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/c2cand.go` — **create**
|
||||
(candidate store + confirm + persistence).
|
||||
- `packages/secubox-toolbox-ng/internal/sentinel/*_test.go` — **create**.
|
||||
- `packages/secubox-toolbox-ng/cmd/sbx-sentinel/main.go` — **modify** (wire the
|
||||
learner after Behavioral; pass overlay/allowlist/candidate paths from env).
|
||||
- `packages/secubox-toolbox-ng/cmd/sbx-sentinel/http.go` — **modify**
|
||||
(`/c2/learned`, `/c2/candidates`, `POST /c2/allow`).
|
||||
- `packages/secubox-toolbox-ng/debian/` — **modify** (ship a seeded
|
||||
`c2-allow.txt` + known-browser-JA4 set; env vars in `sentinel.env`).
|
||||
- `packages/secubox-toolbox/secubox_toolbox/sentinel_link.py` +
|
||||
`api.py` — **modify** (proxy `/admin/sentinel/c2*` + allow POST, fail-safe).
|
||||
- `packages/secubox-toolbox/www/toolbox/index.html` — **modify** (C2 appris
|
||||
sub-view + Ignorer button).
|
||||
|
||||
---
|
||||
|
||||
## Config (env, defaults tuned for precision)
|
||||
|
||||
- `SENTINEL_C2_ALLOW_FILE=/etc/secubox/sentinel/c2-allow.txt`
|
||||
- `SENTINEL_C2_BOX_DOMAINS=/etc/secubox/waf/haproxy-routes.json` (+ optional
|
||||
`box-domains.txt`)
|
||||
- `SENTINEL_C2_CANDIDATES=/var/lib/secubox/sentinel/c2-candidates.json`
|
||||
- `SENTINEL_C2_LEARNED=/var/lib/secubox/sentinel/overlay/learned-c2.json`
|
||||
- thresholds: `c2MinWindows=3`, `c2MinSpan=30m`, `c2LearnedTTL=720h`, learned
|
||||
`severity=75`, candidate/learned caps `2000`. Constants, tunable in one place.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
- Auto-*blocking* of learned C2 (stays report-only; a future operator opt-in).
|
||||
- IP-literal C2 beyond the RFC1918 suppression (mirror is mostly HTTP host-based).
|
||||
- Cross-device fleet correlation via the mesh (single-node learner for now).
|
||||
- Payload-size / cert-chain signals (not reliably available from the async
|
||||
mirror today).
|
||||
|
|
@ -5,16 +5,28 @@
|
|||
|
||||
// Optional read-only status HTTP surface for sbx-sentinel. It is OFF by
|
||||
// default (only stood up when Config.HTTPAddr / SENTINEL_HTTP_ADDR is set) and
|
||||
// serves two GET-only endpoints backed by the verdict Store:
|
||||
// serves GET-only endpoints backed by the verdict Store:
|
||||
//
|
||||
// - GET /stats → {"detections":N,"blocked":N,"spyware":N} for a sidebar.
|
||||
// - GET /verdicts → the recent verdicts, each with its RenderReport text.
|
||||
//
|
||||
// This is a minimal local read for an operator/portal; it accepts no writes,
|
||||
// carries no PII beyond mac_hash, and does NOT route through the WAF-bypass
|
||||
// path. The richer operator UI (verdicts panel, per-report view) belongs in
|
||||
// the separate Python secubox-toolbox portal — see debian/README.sentinel.md
|
||||
// for the intended /api/v1/toolbox/sentinel/* routes it should expose.
|
||||
// When the daemon was built with the #826 C2 auto-learn analyzer wired
|
||||
// (production default — see buildAnalyzers in main.go), three more routes
|
||||
// are registered:
|
||||
//
|
||||
// - GET /c2/learned → the confirmed learned-C2 set ([]sentinel.LearnedC2).
|
||||
// - GET /c2/candidates → the in-progress candidate set ([]sentinel.C2Candidate).
|
||||
// - POST /c2/allow → operator "Ignorer": an x-www-form-urlencoded (or
|
||||
// query-string) `host` param moves a learned/candidate host onto the
|
||||
// allowlist. NOT a JSON body — r.FormValue only parses form/query values.
|
||||
// The only write this surface accepts — it edits the local allow-list
|
||||
// file only, never the network.
|
||||
//
|
||||
// This is a minimal local read for an operator/portal; carries no PII beyond
|
||||
// mac_hash and does NOT route through the WAF-bypass path. The richer
|
||||
// operator UI (verdicts panel, per-report view) belongs in the separate
|
||||
// Python secubox-toolbox portal — see debian/README.sentinel.md for the
|
||||
// intended /api/v1/toolbox/sentinel/* routes it should expose.
|
||||
package main
|
||||
|
||||
import (
|
||||
|
|
@ -83,7 +95,11 @@ func computeStats(vs []sentinel.Verdict) sentinelStats {
|
|||
// newStatusMux builds the read-only status router over store. Exposed
|
||||
// (unexported to the package) so http_test.go can exercise the handlers
|
||||
// directly with httptest, independent of run()'s listener lifecycle.
|
||||
func newStatusMux(store *sentinel.Store) *http.ServeMux {
|
||||
//
|
||||
// c2 is the C2 auto-learn analyzer (#826); when nil (the daemon was built
|
||||
// without one, e.g. a test-injected pipeline) the /c2/* routes are simply
|
||||
// not registered — a request to them 404s rather than panicking.
|
||||
func newStatusMux(store *sentinel.Store, c2 *sentinel.C2Learner) *http.ServeMux {
|
||||
mux := http.NewServeMux()
|
||||
|
||||
mux.HandleFunc("/stats", func(w http.ResponseWriter, r *http.Request) {
|
||||
|
|
@ -139,6 +155,42 @@ func newStatusMux(store *sentinel.Store) *http.ServeMux {
|
|||
writeJSON(w, out)
|
||||
})
|
||||
|
||||
// Read-only C2 auto-learn views + the operator "Ignorer" allow-list
|
||||
// write (#826). Registered only when a C2Learner is actually wired —
|
||||
// the daemon still runs fine without one (fail-safe).
|
||||
if c2 != nil {
|
||||
mux.HandleFunc("/c2/learned", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
writeJSON(w, c2.Learned())
|
||||
})
|
||||
mux.HandleFunc("/c2/candidates", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
writeJSON(w, c2.Candidates())
|
||||
})
|
||||
mux.HandleFunc("/c2/allow", func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
host := r.FormValue("host")
|
||||
if host == "" {
|
||||
http.Error(w, "host required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := c2.Allow(host); err != nil {
|
||||
http.Error(w, "allow failed", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
writeJSON(w, map[string]bool{"ok": true})
|
||||
})
|
||||
}
|
||||
|
||||
return mux
|
||||
}
|
||||
|
||||
|
|
@ -154,10 +206,10 @@ func writeJSON(w http.ResponseWriter, v any) {
|
|||
// cancelled, then shuts it down gracefully. A ListenAndServe error other than
|
||||
// the expected post-Shutdown ErrServerClosed is logged (the daemon's core
|
||||
// socket/store keeps running regardless — the status surface is non-critical).
|
||||
func serveStatus(ctx context.Context, addr string, store *sentinel.Store) {
|
||||
func serveStatus(ctx context.Context, addr string, store *sentinel.Store, c2 *sentinel.C2Learner) {
|
||||
srv := &http.Server{
|
||||
Addr: addr,
|
||||
Handler: newStatusMux(store),
|
||||
Handler: newStatusMux(store, c2),
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ func TestStatusStats(t *testing.T) {
|
|||
sentinel.Verdict{Class: sentinel.ClassBotnetC2, Action: sentinel.ActionSinkhole, MacHash: "b", TS: now},
|
||||
sentinel.Verdict{Class: sentinel.ClassZeroClick, Action: sentinel.ActionReport, MacHash: "c", TS: now},
|
||||
)
|
||||
mux := newStatusMux(store)
|
||||
mux := newStatusMux(store, nil)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/stats", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
|
@ -74,7 +74,7 @@ func TestStatusVerdicts(t *testing.T) {
|
|||
TS: time.Now().Unix(),
|
||||
},
|
||||
)
|
||||
mux := newStatusMux(store)
|
||||
mux := newStatusMux(store, nil)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/verdicts", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
|
@ -113,7 +113,7 @@ func TestVerdictsFilterByMac(t *testing.T) {
|
|||
Evidence: map[string]string{"ioc_value": "b.example"},
|
||||
MacHash: "bbbb", TS: time.Now().Unix(),
|
||||
})
|
||||
mux := newStatusMux(store)
|
||||
mux := newStatusMux(store, nil)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/verdicts?mac=aaaa", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
|
@ -146,7 +146,7 @@ func TestVerdictsFilterByMac(t *testing.T) {
|
|||
|
||||
func TestStatusRejectsNonGet(t *testing.T) {
|
||||
store := openTestStore(t)
|
||||
mux := newStatusMux(store)
|
||||
mux := newStatusMux(store, nil)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/stats", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
|
|
@ -155,3 +155,34 @@ func TestStatusRejectsNonGet(t *testing.T) {
|
|||
t.Fatalf("expected 405 for POST, got %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestC2Endpoints asserts the #826 /c2/* routes are wired when a C2Learner is
|
||||
// passed to newStatusMux: the two read views return 200, and POST /c2/allow
|
||||
// accepts a form-encoded host.
|
||||
func TestC2Endpoints(t *testing.T) {
|
||||
store := openTestStore(t)
|
||||
dir := t.TempDir()
|
||||
c2 := sentinel.NewC2Learner(sentinel.NewBehavioral(), sentinel.C2Config{
|
||||
AllowFile: filepath.Join(dir, "allow.txt"),
|
||||
CandFile: filepath.Join(dir, "cand.json"),
|
||||
LearnedFile: filepath.Join(dir, "learned.json"),
|
||||
})
|
||||
mux := newStatusMux(store, c2)
|
||||
|
||||
for _, path := range []string{"/c2/learned", "/c2/candidates"} {
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("%s status %d", path, rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/c2/allow", strings.NewReader("host=fp.example"))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("allow status %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,6 +59,16 @@ type Analyzer interface {
|
|||
Analyze(sentinel.MirrorMsg) []*sentinel.Verdict
|
||||
}
|
||||
|
||||
// c2Learner is the package-level handle to the production C2 auto-learn
|
||||
// analyzer (#826), set by buildAnalyzers when it wires the real pipeline
|
||||
// (defaultConfig()'s path). It exists purely so run()'s optional status HTTP
|
||||
// surface can wire the read-only /c2/* endpoints (see http.go) without
|
||||
// threading a new field through Config for a single production caller.
|
||||
// Tests that build their own Config/analyzers directly (bypassing
|
||||
// buildAnalyzers) leave it nil, and newStatusMux/serveStatus treat a nil
|
||||
// *sentinel.C2Learner as "endpoints disabled" — fail-safe.
|
||||
var c2Learner *sentinel.C2Learner
|
||||
|
||||
// Default tuning, overridable via Config for production and tests alike.
|
||||
const (
|
||||
defaultTTL = 72 * time.Hour
|
||||
|
|
@ -158,7 +168,7 @@ func run(ctx context.Context, cfg Config) error {
|
|||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
serveStatus(ctx, cfg.HTTPAddr, store)
|
||||
serveStatus(ctx, cfg.HTTPAddr, store, c2Learner)
|
||||
}()
|
||||
}
|
||||
|
||||
|
|
@ -306,14 +316,36 @@ func splitPaths(v string) []string {
|
|||
return out
|
||||
}
|
||||
|
||||
// readLinesFile reads path as a list of newline-separated entries, trimming
|
||||
// whitespace and skipping blank lines and "#"-prefixed comments. Fail-safe by
|
||||
// construction: a missing or unreadable file (the common case — the seed
|
||||
// ships empty/absent until an operator/feed populates it) returns nil rather
|
||||
// than an error, so callers never need special-case "file not found".
|
||||
func readLinesFile(path string) []string {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// buildAnalyzers constructs the production detection pipeline: the
|
||||
// commercial-spyware IOC correlator (backed by a pack Loader over
|
||||
// packDir/overlayDir), the behavioral heuristic engine, and the YARA engine
|
||||
// (the no-cgo stub in the default build; the libyara-backed engine under
|
||||
// `-tags yara`). It is resilient: a failure to build the pack loader or the
|
||||
// YARA engine is returned (joined) but does NOT drop the analyzers that could
|
||||
// be built — a bad base pack must not silently disable the behavioral engine,
|
||||
// which needs no pack. The happy path (readable/empty dirs) returns all three.
|
||||
// packDir/overlayDir), the behavioral heuristic engine wrapped in the C2
|
||||
// auto-learn orchestrator (#826), and the YARA engine (the no-cgo stub in the
|
||||
// default build; the libyara-backed engine under `-tags yara`). It is
|
||||
// resilient: a failure to build the pack loader or the YARA engine is
|
||||
// returned (joined) but does NOT drop the analyzers that could be built — a
|
||||
// bad base pack must not silently disable the behavioral/C2 engine, which
|
||||
// needs no pack. The happy path (readable/empty dirs) returns all three.
|
||||
func buildAnalyzers(packDir, overlayDir string, yaraRules []string) ([]Analyzer, error) {
|
||||
var analyzers []Analyzer
|
||||
var errs []error
|
||||
|
|
@ -325,7 +357,15 @@ func buildAnalyzers(packDir, overlayDir string, yaraRules []string) ([]Analyzer,
|
|||
analyzers = append(analyzers, sentinel.NewSpyware(loader))
|
||||
}
|
||||
|
||||
analyzers = append(analyzers, sentinel.NewBehavioral())
|
||||
c2 := sentinel.NewC2Learner(sentinel.NewBehavioral(), sentinel.C2Config{
|
||||
AllowFile: getenvDefault("SENTINEL_C2_ALLOW_FILE", "/etc/secubox/sentinel/c2-allow.txt"),
|
||||
BoxFile: getenvDefault("SENTINEL_C2_BOX_DOMAINS", "/etc/secubox/waf/haproxy-routes.json"),
|
||||
CandFile: getenvDefault("SENTINEL_C2_CANDIDATES", "/var/lib/secubox/sentinel/c2-candidates.json"),
|
||||
LearnedFile: getenvDefault("SENTINEL_C2_LEARNED", "/var/lib/secubox/sentinel/c2-learned.json"),
|
||||
BrowserJA4: readLinesFile(getenvDefault("SENTINEL_C2_BROWSER_JA4", "/usr/share/secubox/sentinel/browser-ja4.txt")),
|
||||
})
|
||||
analyzers = append(analyzers, c2)
|
||||
c2Learner = c2 // package-level handle for the status mux (see http.go wiring)
|
||||
|
||||
yara, err := sentinel.NewYaraEngine(yaraRules)
|
||||
if err != nil {
|
||||
|
|
|
|||
|
|
@ -14,9 +14,10 @@ import (
|
|||
)
|
||||
|
||||
// TestBuildAnalyzersReturnsThree asserts the production analyzer construction
|
||||
// wires exactly the three real engines (spyware, behavioral, YARA) — this is
|
||||
// the path defaultConfig()/main() use, distinct from the tests' injected
|
||||
// stubAnalyzer path.
|
||||
// wires exactly the three real engines (spyware, the behavioral engine
|
||||
// wrapped in the #826 C2 auto-learn orchestrator, YARA) — this is the path
|
||||
// defaultConfig()/main() use, distinct from the tests' injected stubAnalyzer
|
||||
// path.
|
||||
func TestBuildAnalyzersReturnsThree(t *testing.T) {
|
||||
// Empty/missing dirs are best-effort (no error) — NewLoader tolerates
|
||||
// them — so the happy path returns all three analyzers with no error.
|
||||
|
|
@ -28,19 +29,19 @@ func TestBuildAnalyzersReturnsThree(t *testing.T) {
|
|||
t.Fatalf("expected 3 analyzers, got %d", len(analyzers))
|
||||
}
|
||||
|
||||
var haveSpyware, haveBehavioral, haveYara bool
|
||||
var haveSpyware, haveC2Learner, haveYara bool
|
||||
for _, a := range analyzers {
|
||||
switch a.(type) {
|
||||
case *sentinel.Spyware:
|
||||
haveSpyware = true
|
||||
case *sentinel.Behavioral:
|
||||
haveBehavioral = true
|
||||
case *sentinel.C2Learner:
|
||||
haveC2Learner = true
|
||||
case *sentinel.YaraEngine:
|
||||
haveYara = true
|
||||
}
|
||||
}
|
||||
if !haveSpyware || !haveBehavioral || !haveYara {
|
||||
t.Fatalf("missing an analyzer type: spyware=%v behavioral=%v yara=%v", haveSpyware, haveBehavioral, haveYara)
|
||||
if !haveSpyware || !haveC2Learner || !haveYara {
|
||||
t.Fatalf("missing an analyzer type: spyware=%v c2learner=%v yara=%v", haveSpyware, haveC2Learner, haveYara)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
7
packages/secubox-toolbox-ng/debian/browser-ja4.txt
Normal file
7
packages/secubox-toolbox-ng/debian/browser-ja4.txt
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Known browser JA4/JA3 fingerprints (one per line). A destination polled with
|
||||
# one of these is treated as browser-driven (an admin dashboard), so the
|
||||
# non_browser_ja corroborating signal does NOT fire for it.
|
||||
#
|
||||
# Empty by default: operators/feeds extend this list; an empty file simply
|
||||
# disables the non_browser_ja signal shortcut (the other C2 corroborating
|
||||
# signals — rarity, DGA-looking labels, etc. — still apply).
|
||||
15
packages/secubox-toolbox-ng/debian/c2-allow.txt
Normal file
15
packages/secubox-toolbox-ng/debian/c2-allow.txt
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Seeded C2-autolearn allowlist — legitimate periodic traffic that must never
|
||||
# be learned as C2. Operators may append hosts (also via the WebUI "Ignorer").
|
||||
# Mail / webmail
|
||||
mail.google.com
|
||||
imap.gmail.com
|
||||
outlook.office365.com
|
||||
# OS / browser update + telemetry
|
||||
clients2.google.com
|
||||
update.googleapis.com
|
||||
push.services.mozilla.com
|
||||
# Time / connectivity checks
|
||||
connectivitycheck.gstatic.com
|
||||
time.cloudflare.com
|
||||
# Monitoring / CDN keep-alives
|
||||
ping.chartbeat.net
|
||||
|
|
@ -17,6 +17,15 @@ case "$1" in
|
|||
fi
|
||||
# Intentionally NO `systemctl enable --now`. See the unit header and
|
||||
# debian/changelog: enabled only at the Phase 6 cutover.
|
||||
|
||||
# #826 C2 autolearn: the daemon (secubox-toolbox) appends operator-allowlisted
|
||||
# hosts to c2-allow.txt via POST /c2/allow. Make the dir + file writable by it
|
||||
# WITHOUT touching the shared /etc/secubox parent (stays 0755).
|
||||
if [ -d /etc/secubox/sentinel ]; then
|
||||
chown secubox-toolbox:secubox-toolbox /etc/secubox/sentinel 2>/dev/null || true
|
||||
chmod 0750 /etc/secubox/sentinel 2>/dev/null || true
|
||||
[ -f /etc/secubox/sentinel/c2-allow.txt ] && chown secubox-toolbox:secubox-toolbox /etc/secubox/sentinel/c2-allow.txt 2>/dev/null || true
|
||||
fi
|
||||
;;
|
||||
abort-upgrade|abort-remove|abort-deconfigure)
|
||||
;;
|
||||
|
|
|
|||
|
|
@ -49,6 +49,11 @@ override_dh_auto_install:
|
|||
# #823 Sentinel: shared SENTINEL_* config (dpkg conffile under /etc).
|
||||
install -d debian/secubox-toolbox-ng/etc/secubox
|
||||
install -m 0644 debian/sentinel.env debian/secubox-toolbox-ng/etc/secubox/sentinel.env
|
||||
# #826 C2 auto-learn: seeded operator allowlist (dpkg conffile) + the
|
||||
# shipped browser-JA4 corroborating-signal seed (read-only content).
|
||||
install -d debian/secubox-toolbox-ng/etc/secubox/sentinel
|
||||
install -m 0644 debian/c2-allow.txt debian/secubox-toolbox-ng/etc/secubox/sentinel/c2-allow.txt
|
||||
install -m 0644 debian/browser-ja4.txt debian/secubox-toolbox-ng/usr/share/secubox/sentinel/browser-ja4.txt
|
||||
# #736 media catcher: own its /run log so the append never fails (see conf)
|
||||
install -d debian/secubox-toolbox-ng/usr/lib/tmpfiles.d
|
||||
install -m 0644 tmpfiles/zz-secubox-toolbox-ng.conf debian/secubox-toolbox-ng/usr/lib/tmpfiles.d/
|
||||
|
|
|
|||
|
|
@ -53,6 +53,12 @@ ReadOnlyPaths=/etc/secubox
|
|||
# tmpfiles). ProtectSystem=strict otherwise mounts both read-only.
|
||||
ReadWritePaths=/run/secubox
|
||||
ReadWritePaths=/var/lib/secubox/sentinel
|
||||
# #826 C2 auto-learn: POST /c2/allow appends to c2-allow.txt under
|
||||
# /etc/secubox/sentinel. ReadOnlyPaths=/etc/secubox above would otherwise make
|
||||
# this EROFS; a ReadWritePaths nested under a ReadOnlyPaths ancestor re-grants
|
||||
# write access to just this subtree (systemd path-mangling rule), so the rest
|
||||
# of /etc/secubox stays read-only to this daemon.
|
||||
ReadWritePaths=/etc/secubox/sentinel
|
||||
MemoryHigh=96M
|
||||
MemoryMax=128M
|
||||
TasksMax=64
|
||||
|
|
|
|||
|
|
@ -44,6 +44,27 @@ SENTINEL_HTTP_ADDR=
|
|||
# (-tags yara + libyara-dev) reads ":"-separated rule file paths from here.
|
||||
SENTINEL_YARA_RULES=
|
||||
|
||||
# ── C2 auto-learn (#826) ──────────────────────────────────────────────────────
|
||||
# Operator allowlist — hosts here are never learned as C2 (the WebUI
|
||||
# "Ignorer" action appends here too). This is a dpkg conffile: local edits
|
||||
# survive package upgrades.
|
||||
SENTINEL_C2_ALLOW_FILE=/etc/secubox/sentinel/c2-allow.txt
|
||||
|
||||
# Box/vhost domains (HAProxy routes) — cross-referenced so the board's own
|
||||
# exposed services are never mistaken for C2 destinations.
|
||||
SENTINEL_C2_BOX_DOMAINS=/etc/secubox/waf/haproxy-routes.json
|
||||
|
||||
# In-progress candidate set (beaconing hosts not yet promoted to learned).
|
||||
SENTINEL_C2_CANDIDATES=/var/lib/secubox/sentinel/c2-candidates.json
|
||||
|
||||
# Confirmed learned-C2 set (persisted, report-only re-contact detection).
|
||||
SENTINEL_C2_LEARNED=/var/lib/secubox/sentinel/c2-learned.json
|
||||
|
||||
# Known browser JA4/JA3 fingerprints — a destination polled with one of these
|
||||
# is treated as browser-driven, so the non_browser_ja corroborating signal
|
||||
# does not fire for it. Empty file = signal simply never fires.
|
||||
SENTINEL_C2_BROWSER_JA4=/usr/share/secubox/sentinel/browser-ja4.txt
|
||||
|
||||
# ── Live feed source URLs (secubox-sentinel-feeds) ───────────────────────────
|
||||
# Override to pin/mirror a feed; set empty to DISABLE a feed (its last-good
|
||||
# overlay is kept, never wiped).
|
||||
|
|
|
|||
185
packages/secubox-toolbox-ng/internal/sentinel/c2allow.go
Normal file
185
packages/secubox-toolbox-ng/internal/sentinel/c2allow.go
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
// C2 auto-learn false-positive gate (#823 C2 autolearn): a host is "allowed"
|
||||
// (never learned as C2) if it is a box-owned vhost, a private/loopback IP
|
||||
// literal, or listed (suffix-matched) in the operator allowlist. Every source
|
||||
// is fail-safe: a missing/corrupt file contributes no entries, never an error.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
type C2Allow struct {
|
||||
allowFile string
|
||||
boxFile string
|
||||
|
||||
mu sync.RWMutex
|
||||
suffix map[string]bool // host suffixes (allow entries + box vhosts), lowercased
|
||||
|
||||
addMu sync.Mutex // serializes Add's read-modify-write of allowFile
|
||||
}
|
||||
|
||||
// NewC2Allow builds the gate from an operator allowlist file (one host/suffix
|
||||
// per line, # comments allowed) and a box-domains source (a HAProxy
|
||||
// routes JSON whose KEYS are the box's own vhost domains). Either path may be
|
||||
// "" or missing. Loads immediately (fail-safe).
|
||||
func NewC2Allow(allowFile, boxFile string) *C2Allow {
|
||||
a := &C2Allow{allowFile: allowFile, boxFile: boxFile}
|
||||
a.Reload()
|
||||
return a
|
||||
}
|
||||
|
||||
// Reload re-reads both sources into a fresh suffix set. Fail-safe.
|
||||
func (a *C2Allow) Reload() {
|
||||
set := make(map[string]bool)
|
||||
for _, l := range readLinesSafe(a.allowFile) {
|
||||
l = strings.ToLower(strings.TrimSpace(l))
|
||||
if l == "" || strings.HasPrefix(l, "#") {
|
||||
continue
|
||||
}
|
||||
set[l] = true
|
||||
}
|
||||
for _, d := range readBoxDomainsSafe(a.boxFile) {
|
||||
d = strings.ToLower(strings.TrimSpace(d))
|
||||
if d != "" {
|
||||
set[d] = true
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.suffix = set
|
||||
a.mu.Unlock()
|
||||
}
|
||||
|
||||
// Allowed reports whether host must NOT be learned as C2. Empty host → true
|
||||
// (never learn a blank). Private/loopback/link-local IP literals → true.
|
||||
// Otherwise suffix-matched against the allow set.
|
||||
func (a *C2Allow) Allowed(host string) bool {
|
||||
if host == "" {
|
||||
return true
|
||||
}
|
||||
h := strings.ToLower(strings.TrimSpace(host))
|
||||
if ip := net.ParseIP(h); ip != nil {
|
||||
return ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast()
|
||||
}
|
||||
a.mu.RLock()
|
||||
defer a.mu.RUnlock()
|
||||
// exact + progressive parent-suffix match: a.b.c matches entries a.b.c, b.c, c
|
||||
labels := strings.Split(h, ".")
|
||||
for i := 0; i < len(labels); i++ {
|
||||
if a.suffix[strings.Join(labels[i:], ".")] {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Add appends host to the operator allowlist file (atomic rewrite). The
|
||||
// in-memory set is refreshed by a subsequent Reload (caller's responsibility,
|
||||
// so a batch of Adds costs one reload).
|
||||
//
|
||||
// host is reachable over the network (POST /c2/allow), so it is sanitized:
|
||||
// anything containing a newline, carriage return, or space is rejected
|
||||
// (fail-safe — the invalid host is simply ignored, no error, no write) to
|
||||
// prevent a single call from injecting extra allowlist lines. A bare
|
||||
// single-label host (no ".") is also rejected: Allowed's progressive-suffix
|
||||
// match would let it blind-match an entire TLD.
|
||||
func (a *C2Allow) Add(host string) error {
|
||||
a.addMu.Lock()
|
||||
defer a.addMu.Unlock()
|
||||
host = strings.ToLower(strings.TrimSpace(host))
|
||||
if host == "" || a.allowFile == "" {
|
||||
return nil
|
||||
}
|
||||
if strings.ContainsAny(host, "\n\r ") {
|
||||
return nil
|
||||
}
|
||||
if !strings.Contains(host, ".") {
|
||||
// A bare label (e.g. "com", "localhost") would suffix-match an
|
||||
// entire TLD (or worse) via the progressive-suffix check in
|
||||
// Allowed, blinding detection far beyond the intended host.
|
||||
// Fail-safe ignore — no write.
|
||||
return nil
|
||||
}
|
||||
existing := readLinesSafe(a.allowFile)
|
||||
for _, l := range existing {
|
||||
if strings.ToLower(strings.TrimSpace(l)) == host {
|
||||
return nil // already present
|
||||
}
|
||||
}
|
||||
existing = append(existing, host)
|
||||
return atomicWriteFile(a.allowFile, []byte(strings.Join(existing, "\n")+"\n"), 0o644)
|
||||
}
|
||||
|
||||
// readLinesSafe returns the file's lines, or nil on any error.
|
||||
func readLinesSafe(path string) []string {
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return strings.Split(string(b), "\n")
|
||||
}
|
||||
|
||||
// readBoxDomainsSafe returns the KEYS of a HAProxy-routes-style JSON object
|
||||
// (the box's own vhost domains), or nil on any error.
|
||||
func readBoxDomainsSafe(path string) []string {
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
log.Printf("sentinel c2allow: box-domains %s unparseable (ignored): %v", path, err)
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// atomicWriteFile writes data to path via a temp file + rename in the same dir.
|
||||
func atomicWriteFile(path string, data []byte, perm os.FileMode) error {
|
||||
dir := filepath.Dir(path)
|
||||
f, err := os.CreateTemp(dir, ".c2tmp-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := f.Name()
|
||||
if _, err := f.Write(data); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Chmod(perm); err != nil {
|
||||
f.Close()
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
157
packages/secubox-toolbox-ng/internal/sentinel/c2allow_test.go
Normal file
157
packages/secubox-toolbox-ng/internal/sentinel/c2allow_test.go
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
package sentinel
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeLines(t *testing.T, path string, lines ...string) {
|
||||
t.Helper()
|
||||
body := ""
|
||||
for _, l := range lines {
|
||||
body += l + "\n"
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowSuffixAndLan(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
allow := filepath.Join(dir, "c2-allow.txt")
|
||||
box := filepath.Join(dir, "haproxy-routes.json")
|
||||
writeLines(t, allow, "mail.example.com", "# comment", "", "monitoring.example.org")
|
||||
os.WriteFile(box, []byte(`{"admin.gk2.secubox.in":["127.0.0.1",9080],"dash.gk2.secubox.in":["127.0.0.1",9081]}`), 0o644)
|
||||
|
||||
a := NewC2Allow(allow, box)
|
||||
cases := []struct {
|
||||
host string
|
||||
want bool
|
||||
}{
|
||||
{"mail.example.com", true}, // exact allow
|
||||
{"imap.mail.example.com", true}, // subdomain of allow entry
|
||||
{"monitoring.example.org", true}, // second allow entry
|
||||
{"admin.gk2.secubox.in", true}, // box vhost (haproxy key)
|
||||
{"api.dash.gk2.secubox.in", true}, // subdomain of box vhost
|
||||
{"192.168.1.50", true}, // RFC1918 literal
|
||||
{"127.0.0.1", true}, // loopback literal
|
||||
{"10.10.0.2", true}, // RFC1918 literal
|
||||
{"evil-c2-xyz.example", false}, // unknown → not allowed
|
||||
{"", true}, // empty host → treat as allowed (never learn a blank)
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := a.Allowed(c.host); got != c.want {
|
||||
t.Errorf("Allowed(%q)=%v want %v", c.host, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowFailSafeMissingFiles(t *testing.T) {
|
||||
a := NewC2Allow("/nonexistent/allow.txt", "/nonexistent/box.json")
|
||||
if !a.Allowed("192.168.0.1") {
|
||||
t.Error("RFC1918 must be allowed even with no files")
|
||||
}
|
||||
if a.Allowed("evil.example") {
|
||||
t.Error("unknown host must not be allowed when files are missing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowAddAppends(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
allow := filepath.Join(dir, "c2-allow.txt")
|
||||
writeLines(t, allow, "seed.example")
|
||||
a := NewC2Allow(allow, "")
|
||||
if err := a.Add("newfp.example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a.Reload()
|
||||
if !a.Allowed("newfp.example") {
|
||||
t.Error("added host must be allowed after Add+Reload")
|
||||
}
|
||||
if !a.Allowed("seed.example") {
|
||||
t.Error("seed host must remain allowed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowAddRejectsInjection(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
allow := filepath.Join(dir, "c2-allow.txt")
|
||||
writeLines(t, allow, "seed.example")
|
||||
a := NewC2Allow(allow, "")
|
||||
a.Add("good.com\nevil.com")
|
||||
a.Reload()
|
||||
if a.Allowed("evil.com") {
|
||||
t.Error("newline-injected second host must not be added")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowAddRejectsBareTLD(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
allow := filepath.Join(dir, "c2-allow.txt")
|
||||
writeLines(t, allow, "seed.example")
|
||||
a := NewC2Allow(allow, "")
|
||||
|
||||
if err := a.Add("com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := a.Add("localhost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a.Reload()
|
||||
|
||||
if a.Allowed("anything.com") {
|
||||
t.Error("bare TLD 'com' must not have been added — it would blind-match the entire TLD")
|
||||
}
|
||||
if a.Allowed("evil.localhost") {
|
||||
t.Error("bare single-label 'localhost' must not have been added")
|
||||
}
|
||||
// existing dotted entry still works
|
||||
if !a.Allowed("seed.example") {
|
||||
t.Error("seed host must remain allowed after rejected bare-TLD Adds")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2AllowAddConcurrent(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
allow := filepath.Join(dir, "c2-allow.txt")
|
||||
writeLines(t, allow, "seed.example")
|
||||
a := NewC2Allow(allow, "")
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 20; i++ {
|
||||
wg.Add(1)
|
||||
go func(n int) { defer wg.Done(); a.Add(fmt.Sprintf("h%d.example", n)) }(i)
|
||||
}
|
||||
wg.Wait()
|
||||
a.Reload()
|
||||
for i := 0; i < 20; i++ {
|
||||
if !a.Allowed(fmt.Sprintf("h%d.example", i)) {
|
||||
t.Errorf("concurrent Add lost h%d.example", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAtomicWriteFileRelativePathAndNoLeak(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// relative path (no slash) must still write correctly next to CWD-safe temp
|
||||
p := filepath.Join(dir, "sub.json")
|
||||
if err := atomicWriteFile(p, []byte("x"), 0o640); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b, _ := os.ReadFile(p); string(b) != "x" {
|
||||
t.Error("atomicWriteFile did not write content")
|
||||
}
|
||||
// rename onto an existing directory fails → temp must be cleaned up
|
||||
d := filepath.Join(dir, "adir")
|
||||
os.Mkdir(d, 0o755)
|
||||
_ = atomicWriteFile(d, []byte("y"), 0o640) // rename file→dir fails
|
||||
entries, _ := os.ReadDir(dir)
|
||||
for _, e := range entries {
|
||||
if strings.HasPrefix(e.Name(), ".c2tmp-") {
|
||||
t.Errorf("leaked temp file %s after failed rename", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
178
packages/secubox-toolbox-ng/internal/sentinel/c2cand.go
Normal file
178
packages/secubox-toolbox-ng/internal/sentinel/c2cand.go
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
// C2 auto-learn candidate lifecycle (#823): a beaconing host that passed the FP
|
||||
// gate and carries >=1 corroborating signal becomes a candidate. It is promoted
|
||||
// only when sustained across c2MinWindows separate beacon reports spanning at
|
||||
// least c2MinSpanSec — never on a single burst.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
c2MaxEntries = 2000
|
||||
c2MinWindows = 3
|
||||
c2MaxDevicesPerHost = 256
|
||||
)
|
||||
|
||||
var c2MinSpanSec int64 = 1800 // 30 min; var so tests/config can adjust
|
||||
|
||||
type C2Candidate struct {
|
||||
Host string `json:"host"`
|
||||
FirstSeen int64 `json:"first_seen"`
|
||||
LastSeen int64 `json:"last_seen"`
|
||||
Windows int `json:"windows"`
|
||||
Devices map[string]bool `json:"devices"`
|
||||
Signals map[string]bool `json:"signals"`
|
||||
IntervalS float64 `json:"interval_s"`
|
||||
Promoted bool `json:"promoted"`
|
||||
}
|
||||
|
||||
type C2Cand struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
m map[string]*C2Candidate
|
||||
}
|
||||
|
||||
// cloneCandidate returns a deep value copy of cd, including independent
|
||||
// Devices/Signals map allocations, so callers holding the returned value can
|
||||
// never race a concurrent Record() mutating the live candidate's maps.
|
||||
func cloneCandidate(cd *C2Candidate) C2Candidate {
|
||||
out := *cd
|
||||
out.Devices = make(map[string]bool, len(cd.Devices))
|
||||
for k, v := range cd.Devices {
|
||||
out.Devices[k] = v
|
||||
}
|
||||
out.Signals = make(map[string]bool, len(cd.Signals))
|
||||
for k, v := range cd.Signals {
|
||||
out.Signals[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// NewC2Cand loads persisted candidates from path (fail-safe: missing/corrupt →
|
||||
// empty).
|
||||
func NewC2Cand(path string) *C2Cand {
|
||||
c := &C2Cand{path: path, m: make(map[string]*C2Candidate)}
|
||||
if b, err := os.ReadFile(path); err == nil {
|
||||
var list []*C2Candidate
|
||||
if json.Unmarshal(b, &list) == nil {
|
||||
for _, cd := range list {
|
||||
if cd != nil && cd.Host != "" {
|
||||
if cd.Devices == nil {
|
||||
cd.Devices = map[string]bool{}
|
||||
}
|
||||
if cd.Signals == nil {
|
||||
cd.Signals = map[string]bool{}
|
||||
}
|
||||
c.m[cd.Host] = cd
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Record folds one beacon observation into host's candidate and reports whether
|
||||
// this call is the FIRST to satisfy the promotion criteria (sustained across
|
||||
// >=c2MinWindows spanning >=c2MinSpanSec). Latched: returns true at most once.
|
||||
func (c *C2Cand) Record(host, mac string, ts int64, intervalS float64, signals []string) (bool, C2Candidate) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
cd := c.m[host]
|
||||
isNew := cd == nil
|
||||
if isNew {
|
||||
cd = &C2Candidate{Host: host, FirstSeen: ts, Devices: map[string]bool{}, Signals: map[string]bool{}}
|
||||
c.m[host] = cd
|
||||
}
|
||||
cd.LastSeen = ts
|
||||
cd.Windows++
|
||||
cd.IntervalS = intervalS
|
||||
if mac != "" {
|
||||
if _, ok := cd.Devices[mac]; ok || len(cd.Devices) < c2MaxDevicesPerHost {
|
||||
cd.Devices[mac] = true
|
||||
}
|
||||
}
|
||||
for _, s := range signals {
|
||||
cd.Signals[s] = true
|
||||
}
|
||||
|
||||
promote := false
|
||||
if !cd.Promoted &&
|
||||
cd.Windows >= c2MinWindows &&
|
||||
(cd.LastSeen-cd.FirstSeen) >= c2MinSpanSec &&
|
||||
len(cd.Signals) >= 1 {
|
||||
cd.Promoted = true
|
||||
promote = true
|
||||
}
|
||||
if isNew {
|
||||
// Evict only after cd.LastSeen is set to a real timestamp, so the
|
||||
// just-inserted entry can never be mistaken for the oldest (LastSeen
|
||||
// == 0) candidate and evict itself.
|
||||
c.evictLocked()
|
||||
}
|
||||
return promote, cloneCandidate(cd)
|
||||
}
|
||||
|
||||
// evictLocked keeps the map under c2MaxEntries by dropping the oldest-seen
|
||||
// non-promoted candidate. Caller holds the lock.
|
||||
func (c *C2Cand) evictLocked() {
|
||||
if len(c.m) <= c2MaxEntries {
|
||||
return
|
||||
}
|
||||
var victim string
|
||||
var oldest int64
|
||||
for h, cd := range c.m {
|
||||
if cd.Promoted {
|
||||
continue
|
||||
}
|
||||
if victim == "" || cd.LastSeen < oldest {
|
||||
victim, oldest = h, cd.LastSeen
|
||||
}
|
||||
}
|
||||
if victim != "" {
|
||||
delete(c.m, victim)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *C2Cand) Snapshot() []C2Candidate {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
out := make([]C2Candidate, 0, len(c.m))
|
||||
for _, cd := range c.m {
|
||||
out = append(out, cloneCandidate(cd))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (c *C2Cand) Remove(host string) {
|
||||
c.mu.Lock()
|
||||
delete(c.m, host)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// Persist atomically writes the candidate set to path.
|
||||
func (c *C2Cand) Persist() error {
|
||||
c.mu.Lock()
|
||||
list := make([]C2Candidate, 0, len(c.m))
|
||||
for _, cd := range c.m {
|
||||
list = append(list, cloneCandidate(cd))
|
||||
}
|
||||
c.mu.Unlock()
|
||||
b, err := json.Marshal(list)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if c.path == "" {
|
||||
return nil
|
||||
}
|
||||
return atomicWriteFile(c.path, b, 0o640)
|
||||
}
|
||||
88
packages/secubox-toolbox-ng/internal/sentinel/c2cand_test.go
Normal file
88
packages/secubox-toolbox-ng/internal/sentinel/c2cand_test.go
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestC2CandPromotionRequiresSustained(t *testing.T) {
|
||||
c := NewC2Cand(filepath.Join(t.TempDir(), "cand.json"))
|
||||
base := int64(1_000_000)
|
||||
// window 1
|
||||
if p, _ := c.Record("c2.example", "devA", base, 60, []string{"rare", "dga"}); p {
|
||||
t.Fatal("must not promote on window 1")
|
||||
}
|
||||
// window 2 (still < c2MinWindows)
|
||||
if p, _ := c.Record("c2.example", "devA", base+900, 60, []string{"rare"}); p {
|
||||
t.Fatal("must not promote on window 2")
|
||||
}
|
||||
// window 3, span now 1800s (>= c2MinSpanSec) → promote once
|
||||
p, cand := c.Record("c2.example", "devA", base+1800, 60, []string{"rare"})
|
||||
if !p {
|
||||
t.Fatal("must promote on window 3 with span met")
|
||||
}
|
||||
if cand.Windows < c2MinWindows || cand.Host != "c2.example" {
|
||||
t.Errorf("bad candidate on promote: %+v", cand)
|
||||
}
|
||||
// subsequent records must NOT re-promote (latched)
|
||||
if p, _ := c.Record("c2.example", "devA", base+2700, 60, []string{"rare"}); p {
|
||||
t.Error("must not re-promote after first promotion")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2CandSpanGuard(t *testing.T) {
|
||||
c := NewC2Cand(filepath.Join(t.TempDir(), "cand.json"))
|
||||
base := int64(2_000_000)
|
||||
// 3 windows but all within 10s → span not met → no promote
|
||||
c.Record("burst.example", "devA", base, 5, []string{"rare"})
|
||||
c.Record("burst.example", "devA", base+3, 5, []string{"rare"})
|
||||
p, _ := c.Record("burst.example", "devA", base+9, 5, []string{"rare"})
|
||||
if p {
|
||||
t.Error("must not promote a tight burst (span < c2MinSpanSec)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2CandPersistRoundTrip(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "cand.json")
|
||||
c := NewC2Cand(path)
|
||||
c.Record("x.example", "devA", 100, 30, []string{"rare"})
|
||||
if err := c.Persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c2 := NewC2Cand(path)
|
||||
if len(c2.Snapshot()) != 1 {
|
||||
t.Errorf("expected 1 candidate after reload, got %d", len(c2.Snapshot()))
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2CandSnapshotIsolated(t *testing.T) {
|
||||
c := NewC2Cand(filepath.Join(t.TempDir(), "cand.json"))
|
||||
c.Record("h.example", "devA", 100, 30, []string{"rare"})
|
||||
snap := c.Snapshot()
|
||||
// mutating the snapshot's maps must not affect the live candidate
|
||||
snap[0].Devices["injected"] = true
|
||||
c.Record("h.example", "devB", 200, 30, []string{"dga"})
|
||||
for _, cd := range c.Snapshot() {
|
||||
if cd.Devices["injected"] {
|
||||
t.Error("snapshot map aliases live candidate map (race risk)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2CandDevicesCapped(t *testing.T) {
|
||||
c := NewC2Cand(filepath.Join(t.TempDir(), "cand.json"))
|
||||
for i := 0; i < c2MaxDevicesPerHost+50; i++ {
|
||||
c.Record("h.example", fmt.Sprintf("dev%d", i), int64(100+i), 30, []string{"rare"})
|
||||
}
|
||||
for _, cd := range c.Snapshot() {
|
||||
if len(cd.Devices) > c2MaxDevicesPerHost {
|
||||
t.Errorf("devices not capped: %d", len(cd.Devices))
|
||||
}
|
||||
}
|
||||
}
|
||||
420
packages/secubox-toolbox-ng/internal/sentinel/c2learn.go
Normal file
420
packages/secubox-toolbox-ng/internal/sentinel/c2learn.go
Normal file
|
|
@ -0,0 +1,420 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
// C2 auto-learn orchestrator (#823): wraps the Behavioral beacon detector, and
|
||||
// for each beacon runs the FP gate + corroborating signals + candidate
|
||||
// lifecycle. A confirmed host joins the learned set (persisted, report-only)
|
||||
// and re-contact with a learned host yields a botnet_c2 report verdict. All
|
||||
// report-only.
|
||||
//
|
||||
// Window-advance design note: Behavioral.checkBeaconing latches — it fires a
|
||||
// ClassBotnetC2 "beaconing" verdict at most ONCE per (mac,host) key. Relying
|
||||
// on repeated Behavioral fires to accumulate C2Cand's c2MinWindows would
|
||||
// therefore never promote anything (only the very first window would ever be
|
||||
// recorded). Instead, C2Learner tracks its own per-host "beaconing" state
|
||||
// (interval + last-recorded-window timestamp) once Behavioral's first fire
|
||||
// passes the FP gate and carries >=1 corroborating signal, and advances a
|
||||
// window on its OWN timing: every subsequent flow to that host that arrives
|
||||
// at least ~meanInterval (floored at c2WindowFloorSec) after the last
|
||||
// recorded window counts as one more window. Sustained contact then
|
||||
// accumulates windows across c2MinSpanSec exactly as C2Cand expects, without
|
||||
// depending on Behavioral firing again.
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strconv"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const c2LearnedTTLSec int64 = 30 * 24 * 3600 // 30d; a quiet host ages out
|
||||
|
||||
// c2WindowFloorSec bounds how soon after the last recorded window a
|
||||
// subsequent flow may advance another one. This keeps a degenerate/zero
|
||||
// parsed interval from causing a window-per-message runaway, while still
|
||||
// letting a fast beacon (e.g. every few seconds) accumulate windows at a
|
||||
// sane pace.
|
||||
const c2WindowFloorSec int64 = 60
|
||||
|
||||
// Re-contact throttle bounds: scaled to the learned beacon's own interval
|
||||
// (never faster than c2RecontactMinThrottleSec, never slower than
|
||||
// c2RecontactMaxThrottleSec) rather than a single flat cadence. A learned
|
||||
// host is, by construction, contacted roughly every IntervalS seconds — using
|
||||
// that as the throttle means at most one report per natural check-in cycle
|
||||
// (no spam on a fast beacon) while a slow beacon still gets a timely report
|
||||
// on its very next contact (no waiting a full hour past a 6-minute cycle).
|
||||
const (
|
||||
c2RecontactMinThrottleSec int64 = 60
|
||||
c2RecontactMaxThrottleSec int64 = 3600
|
||||
)
|
||||
|
||||
type LearnedC2 struct {
|
||||
Host string `json:"host"`
|
||||
Signals []string `json:"signals"`
|
||||
IntervalS float64 `json:"interval_s"`
|
||||
Devices int `json:"devices"`
|
||||
FirstSeen int64 `json:"first_seen"`
|
||||
LastSeen int64 `json:"last_seen"`
|
||||
}
|
||||
|
||||
type C2Config struct {
|
||||
AllowFile string
|
||||
BoxFile string
|
||||
CandFile string
|
||||
LearnedFile string
|
||||
BrowserJA4 []string
|
||||
}
|
||||
|
||||
// beaconState is the per-host own-timing tracker used to advance candidate
|
||||
// windows independently of Behavioral's latched beacon verdict.
|
||||
type beaconState struct {
|
||||
intervalS float64
|
||||
lastWindow int64
|
||||
}
|
||||
|
||||
type C2Learner struct {
|
||||
behavioral *Behavioral
|
||||
allow *C2Allow
|
||||
signals *C2Signals
|
||||
cand *C2Cand
|
||||
learnedFN string
|
||||
|
||||
mu sync.Mutex
|
||||
learned map[string]*LearnedC2
|
||||
reported map[string]int64 // host → last re-contact verdict TS (throttle)
|
||||
|
||||
beaconMu sync.Mutex
|
||||
beaconing map[string]*beaconState // host → own-timing window tracker
|
||||
}
|
||||
|
||||
func NewC2Learner(b *Behavioral, cfg C2Config) *C2Learner {
|
||||
l := &C2Learner{
|
||||
behavioral: b,
|
||||
allow: NewC2Allow(cfg.AllowFile, cfg.BoxFile),
|
||||
signals: NewC2Signals(cfg.BrowserJA4),
|
||||
cand: NewC2Cand(cfg.CandFile),
|
||||
learnedFN: cfg.LearnedFile,
|
||||
learned: make(map[string]*LearnedC2),
|
||||
reported: make(map[string]int64),
|
||||
beaconing: make(map[string]*beaconState),
|
||||
}
|
||||
l.loadLearned()
|
||||
return l
|
||||
}
|
||||
|
||||
// Analyze satisfies the daemon Analyzer interface. It always runs Behavioral,
|
||||
// updates the rarity estimate, learns from beacons (both the initial
|
||||
// Behavioral fire and, on its own timing, sustained subsequent contact), and
|
||||
// re-emits for learned hosts. Never blocks; never auto-blocks.
|
||||
func (l *C2Learner) Analyze(m MirrorMsg) []*Verdict {
|
||||
// rarity reflects ALL traffic, not only beacons.
|
||||
l.signals.Observe(m.Meta.Host)
|
||||
|
||||
verdicts := l.behavioral.Analyze(m)
|
||||
|
||||
// learned re-contact → throttled report verdict
|
||||
if v := l.recontact(m); v != nil {
|
||||
verdicts = append(verdicts, v)
|
||||
}
|
||||
|
||||
// The FIRST beacon verdict Behavioral produces for a (mac,host) key
|
||||
// starts our own-timing window tracker (and records candidate window 1).
|
||||
for _, v := range verdicts {
|
||||
if v == nil || v.Class != ClassBotnetC2 || v.Evidence["pattern"] != "beaconing" {
|
||||
continue
|
||||
}
|
||||
l.startBeaconing(m, v)
|
||||
}
|
||||
|
||||
// Because Behavioral's beacon verdict latches (fires only once per key),
|
||||
// sustained contact afterwards is tracked here: advance a window when
|
||||
// enough real time has elapsed since the last one recorded for this
|
||||
// host. No-op for hosts never started above (allowlisted / no signal /
|
||||
// not yet beaconing).
|
||||
l.tickWindow(m)
|
||||
|
||||
return verdicts
|
||||
}
|
||||
|
||||
// startBeaconing runs the FP gate + corroborating-signals check on the FIRST
|
||||
// beacon verdict for a host and, if it passes both, begins this learner's own
|
||||
// per-host window tracking and records candidate window 1.
|
||||
func (l *C2Learner) startBeaconing(m MirrorMsg, beacon *Verdict) {
|
||||
host := m.Meta.Host
|
||||
if host == "" || l.allow.Allowed(host) {
|
||||
return
|
||||
}
|
||||
fired := l.signals.Fired(m.Meta)
|
||||
if len(fired) == 0 {
|
||||
return // no corroboration — periodicity alone never promotes
|
||||
}
|
||||
interval := parseIntervalSec(beacon.Evidence["interval_s"])
|
||||
|
||||
l.beaconMu.Lock()
|
||||
if _, exists := l.beaconing[host]; !exists && len(l.beaconing) >= c2MaxEntries {
|
||||
// Bound l.beaconing the same way C2Cand bounds candidates: evict the
|
||||
// oldest-tracked entry (by last recorded window) before inserting a
|
||||
// new one. Simple linear scan — the map is small.
|
||||
var victim string
|
||||
var oldest int64
|
||||
first := true
|
||||
for h, st := range l.beaconing {
|
||||
if first || st.lastWindow < oldest {
|
||||
victim, oldest = h, st.lastWindow
|
||||
first = false
|
||||
}
|
||||
}
|
||||
if victim != "" {
|
||||
delete(l.beaconing, victim)
|
||||
}
|
||||
}
|
||||
l.beaconing[host] = &beaconState{intervalS: interval, lastWindow: m.TS}
|
||||
l.beaconMu.Unlock()
|
||||
|
||||
l.recordWindow(host, m.Meta.MacHash, m.TS, interval, fired)
|
||||
}
|
||||
|
||||
// tickWindow advances host's window tracker, on the learner's own timing,
|
||||
// once at least ~intervalS (floored) has elapsed since the last recorded
|
||||
// window. It is a no-op for any host not already being tracked (i.e. one
|
||||
// whose first beacon never passed the FP gate / signal check).
|
||||
func (l *C2Learner) tickWindow(m MirrorMsg) {
|
||||
host := m.Meta.Host
|
||||
if host == "" {
|
||||
return
|
||||
}
|
||||
// A learned host is handled by recontact — don't keep re-recording /
|
||||
// re-persisting candidate windows for it. Check under mu and release
|
||||
// before ever touching beaconMu (sequential, never nested) so lock
|
||||
// ordering stays identical to the rest of this file.
|
||||
l.mu.Lock()
|
||||
_, isLearned := l.learned[host]
|
||||
l.mu.Unlock()
|
||||
if isLearned {
|
||||
return
|
||||
}
|
||||
l.beaconMu.Lock()
|
||||
st, ok := l.beaconing[host]
|
||||
if !ok {
|
||||
l.beaconMu.Unlock()
|
||||
return
|
||||
}
|
||||
floor := int64(st.intervalS)
|
||||
if floor < c2WindowFloorSec {
|
||||
floor = c2WindowFloorSec
|
||||
}
|
||||
elapsed := m.TS - st.lastWindow
|
||||
if elapsed < floor {
|
||||
l.beaconMu.Unlock()
|
||||
return
|
||||
}
|
||||
// Pace the next check regardless of outcome below, so a host that keeps
|
||||
// losing corroboration doesn't get re-evaluated on every single message.
|
||||
st.lastWindow = m.TS
|
||||
interval := st.intervalS
|
||||
l.beaconMu.Unlock()
|
||||
|
||||
fired := l.signals.Fired(m.Meta)
|
||||
if len(fired) == 0 {
|
||||
// No corroboration on THIS contact — do not advance the candidate
|
||||
// window. C2Cand unions signals across windows permanently (once
|
||||
// fired, a signal name stays), so recording a window here would let
|
||||
// a signal that only ever fired transiently (e.g. "rare" during an
|
||||
// initial low-hit-count burst that later becomes common) silently
|
||||
// carry a common, browser-fingerprinted host across c2MinWindows on
|
||||
// periodicity alone — exactly the false positive the signal gate
|
||||
// exists to prevent.
|
||||
return
|
||||
}
|
||||
l.recordWindow(host, m.Meta.MacHash, m.TS, interval, fired)
|
||||
}
|
||||
|
||||
// recordWindow folds one window observation into the candidate store and
|
||||
// promotes to the learned set on the (fail-safe, latched) first call where
|
||||
// C2Cand's promotion criteria are met.
|
||||
func (l *C2Learner) recordWindow(host, mac string, ts int64, intervalS float64, signals []string) {
|
||||
promote, cd := l.cand.Record(host, mac, ts, intervalS, signals)
|
||||
_ = l.cand.Persist()
|
||||
if promote {
|
||||
l.promote(cd)
|
||||
}
|
||||
}
|
||||
|
||||
func (l *C2Learner) promote(cd C2Candidate) {
|
||||
sigs := make([]string, 0, len(cd.Signals))
|
||||
for s := range cd.Signals {
|
||||
sigs = append(sigs, s)
|
||||
}
|
||||
l.mu.Lock()
|
||||
l.learned[cd.Host] = &LearnedC2{
|
||||
Host: cd.Host, Signals: sigs, IntervalS: cd.IntervalS,
|
||||
Devices: len(cd.Devices), FirstSeen: cd.FirstSeen, LastSeen: cd.LastSeen,
|
||||
}
|
||||
// Hard cap: evict the oldest-contacted entry (by LastSeen) if promotion
|
||||
// pushed the learned set over budget.
|
||||
if len(l.learned) > c2MaxEntries {
|
||||
var victim string
|
||||
var oldest int64
|
||||
first := true
|
||||
for h, le := range l.learned {
|
||||
if first || le.LastSeen < oldest {
|
||||
victim, oldest = h, le.LastSeen
|
||||
first = false
|
||||
}
|
||||
}
|
||||
if victim != "" {
|
||||
delete(l.learned, victim)
|
||||
}
|
||||
}
|
||||
l.mu.Unlock()
|
||||
|
||||
// A promoted host is covered by recontact from here on — its beacon-
|
||||
// window tracker is dead weight. Acquire/release beaconMu separately
|
||||
// (never nested inside mu) to preserve the existing lock ordering.
|
||||
l.beaconMu.Lock()
|
||||
delete(l.beaconing, cd.Host)
|
||||
l.beaconMu.Unlock()
|
||||
|
||||
// The promoted host leaves the candidate store too, so it stops showing
|
||||
// in both the learned AND candidate WebUI rows.
|
||||
l.cand.Remove(cd.Host)
|
||||
|
||||
l.persistLearned()
|
||||
}
|
||||
|
||||
// recontact returns a throttled report-only botnet_c2 verdict when m hits a
|
||||
// host already in the learned set.
|
||||
func (l *C2Learner) recontact(m MirrorMsg) *Verdict {
|
||||
host := m.Meta.Host
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
l.mu.Lock()
|
||||
le, ok := l.learned[host]
|
||||
if !ok {
|
||||
l.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
throttle := int64(le.IntervalS)
|
||||
if throttle < c2RecontactMinThrottleSec {
|
||||
throttle = c2RecontactMinThrottleSec
|
||||
}
|
||||
if throttle > c2RecontactMaxThrottleSec {
|
||||
throttle = c2RecontactMaxThrottleSec
|
||||
}
|
||||
if m.TS-l.reported[host] < throttle {
|
||||
l.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
l.reported[host] = m.TS
|
||||
le.LastSeen = m.TS
|
||||
l.mu.Unlock()
|
||||
|
||||
return &Verdict{
|
||||
Class: ClassBotnetC2,
|
||||
Severity: 75,
|
||||
Confidence: 75,
|
||||
Action: ActionReport, // learned behavioral — NEVER auto-block
|
||||
Evidence: map[string]string{
|
||||
"pattern": "learned_c2",
|
||||
"host": host,
|
||||
"source": "autolearn",
|
||||
},
|
||||
MacHash: m.Meta.MacHash,
|
||||
TS: m.TS,
|
||||
}
|
||||
}
|
||||
|
||||
func (l *C2Learner) Learned() []LearnedC2 {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
out := make([]LearnedC2, 0, len(l.learned))
|
||||
for _, le := range l.learned {
|
||||
out = append(out, *le)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (l *C2Learner) Candidates() []C2Candidate { return l.cand.Snapshot() }
|
||||
|
||||
// Allow adds host to the operator allowlist and removes it from the learned
|
||||
// set, the candidate store, and this learner's own-timing tracker (the
|
||||
// "Ignorer" safety valve).
|
||||
func (l *C2Learner) Allow(host string) error {
|
||||
if err := l.allow.Add(host); err != nil {
|
||||
return err
|
||||
}
|
||||
l.allow.Reload()
|
||||
l.mu.Lock()
|
||||
delete(l.learned, host)
|
||||
l.mu.Unlock()
|
||||
l.beaconMu.Lock()
|
||||
delete(l.beaconing, host)
|
||||
l.beaconMu.Unlock()
|
||||
l.cand.Remove(host)
|
||||
l.persistLearned()
|
||||
_ = l.cand.Persist()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *C2Learner) loadLearned() {
|
||||
b, err := os.ReadFile(l.learnedFN)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var list []*LearnedC2
|
||||
if json.Unmarshal(b, &list) != nil {
|
||||
return
|
||||
}
|
||||
l.mu.Lock()
|
||||
for _, le := range list {
|
||||
if le != nil && le.Host != "" {
|
||||
l.learned[le.Host] = le
|
||||
}
|
||||
}
|
||||
l.mu.Unlock()
|
||||
}
|
||||
|
||||
// persistLearned atomically writes the learned set, dropping TTL-expired
|
||||
// hosts. TTL is evaluated against the newest LastSeen in the set
|
||||
// (monotonic-free — the daemon has no wall clock guarantee across restarts,
|
||||
// so decay is relative).
|
||||
func (l *C2Learner) persistLearned() {
|
||||
if l.learnedFN == "" {
|
||||
return
|
||||
}
|
||||
l.mu.Lock()
|
||||
var newest int64
|
||||
for _, le := range l.learned {
|
||||
if le.LastSeen > newest {
|
||||
newest = le.LastSeen
|
||||
}
|
||||
}
|
||||
list := make([]*LearnedC2, 0, len(l.learned))
|
||||
for h, le := range l.learned {
|
||||
if newest-le.LastSeen > c2LearnedTTLSec {
|
||||
delete(l.learned, h)
|
||||
delete(l.reported, h) // drop the paired throttle entry too
|
||||
continue
|
||||
}
|
||||
list = append(list, le)
|
||||
}
|
||||
l.mu.Unlock()
|
||||
if b, err := json.Marshal(list); err == nil {
|
||||
_ = atomicWriteFile(l.learnedFN, b, 0o640)
|
||||
}
|
||||
}
|
||||
|
||||
// parseIntervalSec is a fail-safe best-effort parse of Behavioral's
|
||||
// "interval_s" evidence string; 0 on any parse failure (non-fatal — a zero
|
||||
// interval just falls back to the c2WindowFloorSec floor in tickWindow).
|
||||
func parseIntervalSec(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
195
packages/secubox-toolbox-ng/internal/sentinel/c2learn_test.go
Normal file
195
packages/secubox-toolbox-ng/internal/sentinel/c2learn_test.go
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func c2TestLearner(t *testing.T) *C2Learner {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
return NewC2Learner(NewBehavioral(), C2Config{
|
||||
AllowFile: filepath.Join(dir, "allow.txt"),
|
||||
CandFile: filepath.Join(dir, "cand.json"),
|
||||
LearnedFile: filepath.Join(dir, "learned.json"),
|
||||
BrowserJA4: []string{"t13d1516h2_browserfp"},
|
||||
})
|
||||
}
|
||||
|
||||
// A DGA host with a non-browser JA4, beaconed at a steady interval across
|
||||
// enough windows/span, is learned; and re-contact then yields a botnet_c2
|
||||
// verdict.
|
||||
func TestC2LearnerPromotesRealC2(t *testing.T) {
|
||||
l := c2TestLearner(t)
|
||||
host := "x7f3q9zk2vw8plmn.example"
|
||||
mac := "devhashaa"
|
||||
// feed >= beaconMinHits at a constant 300s interval to trip Behavioral,
|
||||
// repeated across >= c2MinWindows spanning >= c2MinSpanSec.
|
||||
ts := int64(1_000_000)
|
||||
learned := false
|
||||
for w := 0; w < 4; w++ {
|
||||
for i := 0; i < 7; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: mac, JA4: "botfp99"}, TS: ts})
|
||||
ts += 300
|
||||
}
|
||||
if len(l.Learned()) > 0 {
|
||||
learned = true
|
||||
}
|
||||
// reset Behavioral's latch is not needed: new windows advance ts; the
|
||||
// learner records a candidate window each time Behavioral fires.
|
||||
}
|
||||
if !learned {
|
||||
t.Fatalf("expected host to be learned; learned=%v candidates=%v", l.Learned(), l.Candidates())
|
||||
}
|
||||
// re-contact a learned host → botnet_c2 report verdict
|
||||
vs := l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: mac, JA4: "botfp99"}, TS: ts})
|
||||
found := false
|
||||
for _, v := range vs {
|
||||
if v.Class == ClassBotnetC2 && v.Action == ActionReport {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("expected a report-only botnet_c2 verdict on re-contact, got %+v", vs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestC2LearnerPromotedHostLeavesCandidates proves a promoted host is moved
|
||||
// wholly out of the candidate store: it must appear in Learned() and NOT in
|
||||
// Candidates() (no double-render in the WebUI), and continued flows to the
|
||||
// now-learned host must not keep growing/re-persisting the candidate
|
||||
// snapshot (tickWindow's early-return on a learned host).
|
||||
func TestC2LearnerPromotedHostLeavesCandidates(t *testing.T) {
|
||||
l := c2TestLearner(t)
|
||||
host := "x7f3q9zk2vw8plmn.example"
|
||||
mac := "devhashaa"
|
||||
ts := int64(1_000_000)
|
||||
learned := false
|
||||
for w := 0; w < 4 && !learned; w++ {
|
||||
for i := 0; i < 7; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: mac, JA4: "botfp99"}, TS: ts})
|
||||
ts += 300
|
||||
}
|
||||
if len(l.Learned()) > 0 {
|
||||
learned = true
|
||||
}
|
||||
}
|
||||
if !learned {
|
||||
t.Fatalf("expected host to be learned; candidates=%v", l.Candidates())
|
||||
}
|
||||
|
||||
for _, cd := range l.Candidates() {
|
||||
if cd.Host == host {
|
||||
t.Fatalf("promoted host %q must not remain in Candidates(), got %+v", host, cd)
|
||||
}
|
||||
}
|
||||
foundLearned := false
|
||||
for _, le := range l.Learned() {
|
||||
if le.Host == host {
|
||||
foundLearned = true
|
||||
}
|
||||
}
|
||||
if !foundLearned {
|
||||
t.Fatalf("promoted host %q must be present in Learned()", host)
|
||||
}
|
||||
|
||||
snapshotBefore := len(l.Candidates())
|
||||
for i := 0; i < 20; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: mac, JA4: "botfp99"}, TS: ts})
|
||||
ts += 300
|
||||
}
|
||||
if got := len(l.Candidates()); got != snapshotBefore {
|
||||
t.Errorf("continued flows to a learned host must not grow candidates: before=%d after=%d", snapshotBefore, got)
|
||||
}
|
||||
}
|
||||
|
||||
// An allowlisted host (box vhost / mail) beaconing with a browser JA4 is never
|
||||
// learned.
|
||||
func TestC2LearnerSuppressesAllowlisted(t *testing.T) {
|
||||
l := c2TestLearner(t)
|
||||
_ = l.Allow("mail.example.com") // operator/seed allow
|
||||
ts := int64(3_000_000)
|
||||
for w := 0; w < 5; w++ {
|
||||
for i := 0; i < 7; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: "imap.mail.example.com", MacHash: "devB", JA4: "t13d1516h2_browserfp"}, TS: ts})
|
||||
ts += 300
|
||||
}
|
||||
}
|
||||
if len(l.Learned()) != 0 {
|
||||
t.Errorf("allowlisted host must never be learned, got %v", l.Learned())
|
||||
}
|
||||
}
|
||||
|
||||
// A periodic host with a BROWSER JA4 and a common word domain (no corroborating
|
||||
// signal) is never learned — periodicity alone is insufficient.
|
||||
func TestC2LearnerNoSignalNoPromote(t *testing.T) {
|
||||
l := c2TestLearner(t)
|
||||
host := "dashboard.example.com"
|
||||
// make it "not rare": observe many times first
|
||||
for i := 0; i < 60; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: "devC", JA4: "t13d1516h2_browserfp"}, TS: int64(4_000_000 + i)})
|
||||
}
|
||||
ts := int64(5_000_000)
|
||||
for w := 0; w < 5; w++ {
|
||||
for i := 0; i < 7; i++ {
|
||||
l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: "devC", JA4: "t13d1516h2_browserfp"}, TS: ts})
|
||||
ts += 300
|
||||
}
|
||||
}
|
||||
if len(l.Learned()) != 0 {
|
||||
t.Errorf("browser-JA4 common-word host must not be learned, got %v", l.Learned())
|
||||
}
|
||||
}
|
||||
|
||||
// TestC2LearnerWindowAdvanceOwnTiming proves the core design point directly:
|
||||
// Behavioral.checkBeaconing fires a ClassBotnetC2 "beaconing" verdict at most
|
||||
// ONCE per (mac,host) — after the first fire, every later Analyze call for
|
||||
// that host returns no NEW beacon verdict from Behavioral (the latch). Yet
|
||||
// C2Learner must still keep accumulating candidate windows on ITS OWN timing
|
||||
// (tickWindow), reaching c2MinWindows across c2MinSpanSec and promoting —
|
||||
// proving promotion does not depend on Behavioral firing again.
|
||||
func TestC2LearnerWindowAdvanceOwnTiming(t *testing.T) {
|
||||
l := c2TestLearner(t)
|
||||
host := "q9zx4kmw7plr2vnh.example" // DGA-shaped, non-browser JA4 → 2 signals
|
||||
mac := "devhashbb"
|
||||
const interval = int64(300)
|
||||
ts := int64(2_000_000)
|
||||
|
||||
beaconFires := 0
|
||||
firstFireSeen := false
|
||||
for i := 0; i < 40; i++ {
|
||||
vs := l.Analyze(MirrorMsg{Meta: FlowMeta{Host: host, MacHash: mac, JA4: "botfp77"}, TS: ts})
|
||||
for _, v := range vs {
|
||||
if v.Class == ClassBotnetC2 && v.Evidence["pattern"] == "beaconing" {
|
||||
beaconFires++
|
||||
firstFireSeen = true
|
||||
}
|
||||
}
|
||||
ts += interval
|
||||
|
||||
if firstFireSeen && len(l.Learned()) == 0 {
|
||||
// After the first Behavioral fire (and before promotion),
|
||||
// candidate windows must still be climbing on the learner's own
|
||||
// timing even though Behavioral itself will never fire
|
||||
// "beaconing" for this key again. Once promoted, the host
|
||||
// deliberately leaves the candidate store (M4) — recontact
|
||||
// takes over — so the check no longer applies past that point.
|
||||
cds := l.Candidates()
|
||||
if len(cds) == 0 {
|
||||
t.Fatalf("iteration %d: expected a tracked candidate after first beacon fire", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if beaconFires != 1 {
|
||||
t.Fatalf("expected Behavioral's beacon verdict to latch (fire exactly once), got %d fires", beaconFires)
|
||||
}
|
||||
if len(l.Learned()) == 0 {
|
||||
t.Fatalf("expected sustained own-timing window-advance to promote the host; candidates=%v", l.Candidates())
|
||||
}
|
||||
}
|
||||
131
packages/secubox-toolbox-ng/internal/sentinel/c2signal.go
Normal file
131
packages/secubox-toolbox-ng/internal/sentinel/c2signal.go
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
// Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
// Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
// See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
package sentinel
|
||||
|
||||
// C2 auto-learn corroborating signals (#823): beaconing alone is not enough to
|
||||
// learn a host — at least one of these independent signals must also fire, so a
|
||||
// browser-driven periodic poll (admin dashboard) is not mistaken for a beacon.
|
||||
|
||||
import (
|
||||
"container/list"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
// c2RareMaxHits: a host seen at most this many times across the daemon's
|
||||
// window counts as "rare". A real C2 destination stays rare; a CDN/portal
|
||||
// the user browses climbs past it quickly.
|
||||
c2RareMaxHits = 20
|
||||
// c2FreqCap bounds the rarity map (LRU).
|
||||
// keep in sync with c2MaxEntries (c2cand.go)
|
||||
c2FreqCap = 2000
|
||||
// c2DGAMinEntropy: Shannon entropy (bits/char) of the most-significant
|
||||
// label above which the domain looks algorithmically generated. Ordinary
|
||||
// words sit well below; a random 16-char label sits near log2(distinct).
|
||||
c2DGAMinEntropy = 3.6
|
||||
// c2DGAMinLen: don't call a short label DGA (too little signal).
|
||||
c2DGAMinLen = 10
|
||||
)
|
||||
|
||||
type C2Signals struct {
|
||||
browser map[string]bool // known browser JA4/JA3 fingerprints
|
||||
|
||||
mu sync.Mutex
|
||||
freq map[string]*list.Element // host → LRU element
|
||||
order *list.List // front = most-recent
|
||||
}
|
||||
|
||||
type freqEntry struct {
|
||||
host string
|
||||
hits int
|
||||
}
|
||||
|
||||
func NewC2Signals(browserJA4 []string) *C2Signals {
|
||||
b := make(map[string]bool, len(browserJA4))
|
||||
for _, f := range browserJA4 {
|
||||
if f = strings.TrimSpace(f); f != "" {
|
||||
b[f] = true
|
||||
}
|
||||
}
|
||||
return &C2Signals{browser: b, freq: make(map[string]*list.Element), order: list.New()}
|
||||
}
|
||||
|
||||
// Observe records one contact with host for the rarity estimate. Call on every
|
||||
// flow (not only beacons) so "rare" reflects true global frequency.
|
||||
func (s *C2Signals) Observe(host string) {
|
||||
if host == "" {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if el, ok := s.freq[host]; ok {
|
||||
el.Value.(*freqEntry).hits++
|
||||
s.order.MoveToFront(el)
|
||||
return
|
||||
}
|
||||
el := s.order.PushFront(&freqEntry{host: host, hits: 1})
|
||||
s.freq[host] = el
|
||||
for s.order.Len() > c2FreqCap {
|
||||
back := s.order.Back()
|
||||
if back == nil {
|
||||
break
|
||||
}
|
||||
delete(s.freq, back.Value.(*freqEntry).host)
|
||||
s.order.Remove(back)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *C2Signals) hits(host string) int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if el, ok := s.freq[host]; ok {
|
||||
return el.Value.(*freqEntry).hits
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Fired returns the corroborating signal names present for m. Order-stable,
|
||||
// deduped. Never panics on missing fields.
|
||||
func (s *C2Signals) Fired(m FlowMeta) []string {
|
||||
var out []string
|
||||
if s.hits(m.Host) <= c2RareMaxHits {
|
||||
out = append(out, "rare")
|
||||
}
|
||||
// non-browser: only when a fingerprint is PRESENT and not a known browser.
|
||||
fp := m.JA4
|
||||
if fp == "" {
|
||||
fp = m.JA3
|
||||
}
|
||||
if fp != "" && !s.browser[fp] {
|
||||
out = append(out, "non_browser_ja")
|
||||
}
|
||||
if isDGA(m.Host) {
|
||||
out = append(out, "dga")
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// isDGA reports whether host's most-significant label looks algorithmically
|
||||
// generated (long + high own-entropy). Fail-safe on empty/short input.
|
||||
func isDGA(host string) bool {
|
||||
host = strings.TrimSpace(strings.ToLower(host))
|
||||
if host == "" {
|
||||
return false
|
||||
}
|
||||
labels := strings.Split(host, ".")
|
||||
// pick the longest label (the registrable label is usually the signal)
|
||||
cand := ""
|
||||
for _, l := range labels {
|
||||
if len(l) > len(cand) {
|
||||
cand = l
|
||||
}
|
||||
}
|
||||
if len(cand) < c2DGAMinLen {
|
||||
return false
|
||||
}
|
||||
return shannonEntropy(cand) >= c2DGAMinEntropy
|
||||
}
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
package sentinel
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestC2SignalsDGA(t *testing.T) {
|
||||
s := NewC2Signals([]string{"t13d1516h2_browserfp"})
|
||||
// high-entropy random-looking label → dga fires
|
||||
fired := s.Fired(FlowMeta{Host: "x7f3q9zk2vw8plmn.example", JA4: "t13d1516h2_browserfp"})
|
||||
if !contains(fired, "dga") {
|
||||
t.Errorf("dga signal expected, got %v", fired)
|
||||
}
|
||||
// ordinary word domain, browser JA4, and seen-often → no signals
|
||||
for i := 0; i < 60; i++ {
|
||||
s.Observe("news.example.com")
|
||||
}
|
||||
fired = s.Fired(FlowMeta{Host: "news.example.com", JA4: "t13d1516h2_browserfp"})
|
||||
if len(fired) != 0 {
|
||||
t.Errorf("no signals expected for common browser-JA4 word-domain, got %v", fired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2SignalsNonBrowserJA(t *testing.T) {
|
||||
s := NewC2Signals([]string{"t13d1516h2_browserfp"})
|
||||
for i := 0; i < 60; i++ {
|
||||
s.Observe("cdn.example.com")
|
||||
}
|
||||
// non-browser JA4 → non_browser_ja fires (host common, low entropy)
|
||||
fired := s.Fired(FlowMeta{Host: "cdn.example.com", JA4: "q99xxbotfp"})
|
||||
if !contains(fired, "non_browser_ja") {
|
||||
t.Errorf("non_browser_ja expected, got %v", fired)
|
||||
}
|
||||
// empty JA4/JA3 (unknown) must NOT count as non-browser (avoid FP on missing data)
|
||||
fired = s.Fired(FlowMeta{Host: "cdn.example.com"})
|
||||
if contains(fired, "non_browser_ja") {
|
||||
t.Errorf("empty JA must not fire non_browser_ja, got %v", fired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestC2SignalsRare(t *testing.T) {
|
||||
s := NewC2Signals(nil)
|
||||
// first-ever contact with a low-entropy word host, no JA → rare only
|
||||
fired := s.Fired(FlowMeta{Host: "portal.example.com"})
|
||||
if !contains(fired, "rare") {
|
||||
t.Errorf("rare expected for never-seen host, got %v", fired)
|
||||
}
|
||||
}
|
||||
|
||||
func contains(ss []string, v string) bool {
|
||||
for _, s := range ss {
|
||||
if s == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
|
@ -11,7 +11,7 @@ import time
|
|||
from pathlib import Path
|
||||
|
||||
import jinja2
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi import APIRouter, Form, HTTPException, Query, Request
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
|
||||
|
||||
|
|
@ -3719,6 +3719,42 @@ async def admin_sentinel_verdicts(limit: int = 50) -> dict:
|
|||
}
|
||||
|
||||
|
||||
def _c2_signals(s):
|
||||
"""Normalize Signals to a list: Go map[string]bool -> JSON object; []string -> JSON array."""
|
||||
if isinstance(s, dict):
|
||||
return sorted(k for k, v in s.items() if v)
|
||||
if isinstance(s, list):
|
||||
return s
|
||||
return []
|
||||
|
||||
|
||||
def _c2_norm_learned(h):
|
||||
return {"host": h.get("host", ""), "signals": _c2_signals(h.get("signals")),
|
||||
"interval_s": h.get("interval_s"), "devices": h.get("devices")}
|
||||
|
||||
|
||||
def _c2_norm_cand(c):
|
||||
return {"host": c.get("host", ""), "signals": _c2_signals(c.get("signals")),
|
||||
"interval_s": c.get("interval_s"), "windows": c.get("windows")}
|
||||
|
||||
|
||||
@router.get("/admin/sentinel/c2")
|
||||
async def admin_sentinel_c2() -> dict:
|
||||
"""Learned + candidate C2 hosts for the WebUI 'C2 appris' view. Fail-safe."""
|
||||
data = sentinel_link.fetch_c2()
|
||||
if not data:
|
||||
return {"active": False, "learned": [], "candidates": []}
|
||||
return {"active": True,
|
||||
"learned": [_c2_norm_learned(h) for h in data.get("learned", [])],
|
||||
"candidates": [_c2_norm_cand(c) for c in data.get("candidates", [])]}
|
||||
|
||||
|
||||
@router.post("/admin/sentinel/c2/allow")
|
||||
async def admin_sentinel_c2_allow(host: str = Form(...)) -> dict:
|
||||
"""Operator 'Ignorer' — allowlist a host so it is never learned as C2."""
|
||||
return {"ok": sentinel_link.c2_allow(host)}
|
||||
|
||||
|
||||
@router.get("/admin/filters/ui", response_class=HTMLResponse)
|
||||
async def admin_filters_ui() -> HTMLResponse:
|
||||
"""#566 — minimal filter toggle panel for the toolbox WebUI."""
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import json
|
|||
import logging
|
||||
import os
|
||||
import re
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from collections import Counter
|
||||
|
||||
|
|
@ -125,6 +126,36 @@ def _is_confirmed_compromise(d: dict) -> bool:
|
|||
)
|
||||
|
||||
|
||||
def fetch_c2() -> dict:
|
||||
"""Fetch learned + candidate C2 sets from the daemon. Fail-safe → {}."""
|
||||
learned = _get_json("/c2/learned")
|
||||
cands = _get_json("/c2/candidates")
|
||||
if learned is None and cands is None:
|
||||
return {}
|
||||
return {
|
||||
"learned": learned if isinstance(learned, list) else [],
|
||||
"candidates": cands if isinstance(cands, list) else [],
|
||||
}
|
||||
|
||||
|
||||
def c2_allow(host: str) -> bool:
|
||||
"""POST an allowlist request to the daemon. Fail-safe → False."""
|
||||
if not host:
|
||||
return False
|
||||
base = daemon_base()
|
||||
if not base:
|
||||
return False
|
||||
try:
|
||||
data = urllib.parse.urlencode({"host": host}).encode()
|
||||
req = urllib.request.Request(base + "/c2/allow", data=data,
|
||||
headers={"Content-Type": "application/x-www-form-urlencoded"})
|
||||
with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
|
||||
return resp.status == 200
|
||||
except Exception as exc:
|
||||
log.debug("sentinel c2_allow failed: %s", exc)
|
||||
return False
|
||||
|
||||
|
||||
def assess(detections: list[dict]) -> dict:
|
||||
"""Compromise/evaluation summary over one device's (or the fleet's) detections.
|
||||
|
||||
|
|
|
|||
52
packages/secubox-toolbox/tests/test_sentinel_c2.py
Normal file
52
packages/secubox-toolbox/tests/test_sentinel_c2.py
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
from fastapi.testclient import TestClient
|
||||
from secubox_toolbox.app import app
|
||||
from secubox_toolbox import sentinel_link as sl
|
||||
|
||||
client = TestClient(app)
|
||||
|
||||
|
||||
def test_c2_route_active(monkeypatch):
|
||||
monkeypatch.setattr(sl, "fetch_c2", lambda: {
|
||||
"learned": [{"host": "x7f3q9zk2vw8plmn.example", "signals": ["dga", "rare"],
|
||||
"interval_s": 300.0, "devices": 1}],
|
||||
"candidates": [],
|
||||
})
|
||||
r = client.get("/admin/sentinel/c2")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["active"] is True
|
||||
assert body["learned"][0]["host"] == "x7f3q9zk2vw8plmn.example"
|
||||
|
||||
|
||||
def test_c2_route_failsafe(monkeypatch):
|
||||
monkeypatch.setattr(sl, "fetch_c2", lambda: {})
|
||||
r = client.get("/admin/sentinel/c2")
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"active": False, "learned": [], "candidates": []}
|
||||
|
||||
|
||||
def test_c2_allow_route(monkeypatch):
|
||||
called = {}
|
||||
|
||||
def _fake_allow(host):
|
||||
called["host"] = host
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(sl, "c2_allow", _fake_allow)
|
||||
r = client.post("/admin/sentinel/c2/allow", data={"host": "fp.example"})
|
||||
assert r.status_code == 200
|
||||
assert r.json()["ok"] is True
|
||||
assert called["host"] == "fp.example"
|
||||
|
||||
|
||||
def test_c2_route_normalizes_candidate_maps(monkeypatch):
|
||||
monkeypatch.setattr(sl, "fetch_c2", lambda: {
|
||||
"learned": [{"host": "l.example", "signals": ["dga"], "interval_s": 300.0, "devices": 2}],
|
||||
"candidates": [{"host": "c.example", "signals": {"dga": True, "rare": True},
|
||||
"interval_s": 120.0, "windows": 2}],
|
||||
})
|
||||
r = client.get("/admin/sentinel/c2")
|
||||
body = r.json()
|
||||
assert body["candidates"][0]["signals"] == ["dga", "rare"] # object → sorted list
|
||||
assert body["candidates"][0]["windows"] == 2
|
||||
assert body["learned"][0]["signals"] == ["dga"] # list passthrough
|
||||
|
|
@ -257,6 +257,7 @@
|
|||
</div>
|
||||
<div id="sentinel-verdict" class="card" style="margin-bottom:.8rem"></div>
|
||||
<div id="sentinel-detections"></div>
|
||||
<div id="sentinel-c2" style="margin-top:1rem"></div>
|
||||
</section>
|
||||
|
||||
<!-- Config -->
|
||||
|
|
@ -640,6 +641,38 @@ async function loadSentinel() {
|
|||
dEl.innerHTML = rows
|
||||
? `<table><thead><tr><th>Menace</th><th>Sév/Conf</th><th>Disposition</th><th>Vu</th><th>Détail</th></tr></thead><tbody>${rows}</tbody></table>`
|
||||
: '<span class="v">Aucune détection récente.</span>';
|
||||
|
||||
if (!stats.__error && data.active) loadSentinelC2();
|
||||
}
|
||||
|
||||
async function loadSentinelC2() {
|
||||
const el = document.getElementById('sentinel-c2');
|
||||
if (!el) return;
|
||||
const d = await J('/admin/sentinel/c2');
|
||||
if (d.__error || !d.active) { el.innerHTML = ''; return; }
|
||||
const row = (h, kind) => `<tr>
|
||||
<td>${esc(h.host)}</td>
|
||||
<td>${esc((h.signals || []).join(', '))}</td>
|
||||
<td>${h.interval_s ? Math.round(h.interval_s) + 's' : '—'}</td>
|
||||
<td>${h.devices || (h.windows ? 'w' + h.windows : '')}</td>
|
||||
<td>${kind === 'learned'
|
||||
? `<button class="c2-ign" data-host="${esc(h.host)}">Ignorer</button>`
|
||||
: '<span class="v">candidat</span>'}</td></tr>`;
|
||||
const learned = (d.learned || []).map(h => row(h, 'learned')).join('');
|
||||
const cand = (d.candidates || []).map(h => row(h, 'candidate')).join('');
|
||||
el.innerHTML = `<h3>🕸️ C2 appris</h3>` + ((learned || cand)
|
||||
? `<table><thead><tr><th>Hôte</th><th>Signaux</th><th>Intervalle</th><th>Vu</th><th></th></tr></thead><tbody>${learned}${cand}</tbody></table>`
|
||||
: '<span class="v">Aucun C2 appris ni candidat.</span>');
|
||||
el.querySelectorAll('.c2-ign').forEach(b => b.addEventListener('click', () => c2Ignore(b.dataset.host)));
|
||||
}
|
||||
|
||||
async function c2Ignore(host) {
|
||||
await fetch(`${API}/admin/sentinel/c2/allow`, {
|
||||
method: 'POST', credentials: 'same-origin',
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
||||
body: 'host=' + encodeURIComponent(host),
|
||||
});
|
||||
loadSentinelC2();
|
||||
}
|
||||
|
||||
async function loadSocial() {
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user