Commit Graph

2916 Commits

Author SHA1 Message Date
3d8207816e fix(vhost): parse certs in-process (cryptography) off-loop + cache — no openssl storm
The /certificates handler shelled out to openssl once per PEM (84 blocking
subprocesses) inside an async handler on the shared aggregator loop → froze the
board when the certs tab was viewed. Now: parse in-process via cryptography
(84 certs in 0.1s vs 12s), run via asyncio.to_thread, and cache 60s. Reads the
real HAProxy PEM store (/data/haproxy/certs), matching the certs module.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-15 07:33:07 +02:00
e66d3e5066 fix(vhost): read certs from /data/haproxy/certs PEM store, not empty /etc/acme (ref #858)
The cert list + cert_count read the legacy acme.sh /etc/acme layout (empty on
this platform) → 0 certs shown. Point them at the combined HAProxy PEM store
(/data/haproxy/certs/*.pem) — the same source the certs module uses. Wildcard
filenames (_wildcard_.x.pem) map back to *.x.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-15 07:15:47 +02:00
83d2da96ea fix(vhost): reskin webui to certs/WebUI-Panel-Guidelines look + enrich certificate metrics (closes #858)
- single clean stylesheet on /shared/hybrid-skin.css + Courier Prime + certs
  palette/component vocab (removed crt-light/sidebar-light/hybrid-dark + the
  duplicate conflicting .btn/.card/.stat-card blocks)
- Certificates tab: parse notAfter -> days-left, colour rows by status
  (expired/critical/warning/healthy), status-count stat grid; add esc() helper
- all tabs/modals/actions + JS element IDs preserved

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-15 07:13:48 +02:00
1e3772b6ec feat(podcaster): 'Add from YouTube/URL' button — full mirror (podcaster + PeerTube + billets) (closes #857)
Productises the one-off importer into the module:
- api/importer.py: yt-dlp URL importer (playlist or single video, or any yt-dlp
  site). Per item: download video<=720p + extract audio, optional PeerTube upload
  (unlisted, creds from root-only /etc/secubox/secrets/peertube-import.json),
  podcaster episode (local audio, done), optional published billet (description +
  source + both stream links). Off-loop via asyncio.to_thread; single job with
  live JOB progress. Optional yt-cookies.txt to defeat YouTube's bot throttle;
  yt-dlp errors surfaced in the progress log.
- endpoints: POST /import/url (guarded), GET /import/status.
- webui: ' Add from URL' button + modal (URL + PeerTube/billets toggles) +
  live progress (current, done/total, bar, errors, log tail).
- Recommends: yt-dlp, ffmpeg.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 11:41:11 +02:00
dae1991309 feat(podcaster): resolve Apple Podcasts links to their RSS feed (ref #856)
Detect podcasts.apple.com / itunes.apple.com links, extract the numeric id, and
resolve the underlying feedUrl via the iTunes Lookup API before ingesting as a
normal RSS feed. Best-effort: falls back to the original URL on any error.
Declare python3-mutagen + python3-pil (used by the #855 cover extractor).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 09:57:28 +02:00
e818f13cf2 feat(podcaster): extract embedded cover art + display cover (now-playing on portal) (ref #855)
- backend: _extract_cover pulls ID3 APIC / MP4 covr / FLAC picture from the first
  audiobook track via mutagen, downscales with Pillow, stores <feed>/cover.jpg,
  sets feed image to /feeds/<fid>/cover; new public cover endpoint. Best-effort
  (never breaks an upload).
- webui: cover thumbnails on episode cards/rows with graceful emoji fallback on
  missing/404; portal gets a 'Now Playing' cover bar driven by a capture-phase
  play listener reading data-* (does NOT rebuild the list → playback guard intact).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 09:55:29 +02:00
35a125fc1f Merge branch 'fix/853-podcaster-downloads-wedge-with-no-timeou' into fix/855-podcaster-webui-episode-ordering-by-type
# Conflicts:
#	packages/secubox-podcaster/www/podcaster/index.html
2026-07-14 09:47:25 +02:00
ed9d679aba feat(podcaster): order episodes by type + feed-first admin + reskin admin/portal + portal playback fix (ref #855)
- store: audiobook feeds order ASC (chapter 1→last), podcasts DESC (latest first)
- admin webui: require selecting a container before showing episodes; reskin to
  WebUI Panel Guidelines (Courier Prime, hybrid-skin.css, certs .btn/.card/.stat-card)
- portal: fix audio stopping every 15s (guard the innerHTML re-render vs playing
  audio + unchanged signature); funky/emoji/modern reskin

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 09:47:10 +02:00
56d3962309 fix(podcaster): stage uploads on the media filesystem + reject oversized early (ref #853)
The audiobook upload wrote its temp ZIP to a hardcoded /var/lib/secubox/podcaster
(the 15GB eMMC) regardless of where media_path points. On a box whose media
store is a large SSD, a big upload still filled the tiny eMMC and failed with a
confusing 502.

- temp ZIP is now created under MEDIA (media_path), same filesystem as the
  extraction target — so it lands wherever the store lives (the SSD).
- before streaming, if Content-Length * 2 + margin exceeds the media
  filesystem's free space, return a clear 413 ("needs ~X MB, only Y free")
  instead of failing mid-write. (No amount of code makes a 700GB ZIP fit — but
  the user gets an instant honest reason.)

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 09:15:32 +02:00
de5c2f2ebb fix(podcaster): socket-ownership race left podcaster.sock root:root → nginx 502
Some checks failed
License Headers / check (push) Has been cancelled
The unit chowned the socket in an ExecStartPost gated only by `sleep 1`.
uvicorn can take several seconds to bind the UDS, so the chown ran before the
socket existed (silently, via the `-` prefix) and it stayed root:root — which
nginx (not root) cannot open, so every podcaster request 502'd after a restart.
A naive poll didn't help either: it found the STALE socket from the previous
run, chowned that, and uvicorn then re-bound a fresh root:root socket.

Fix: ExecStartPre removes the stale socket first, and ExecStartPost polls up to
~15s for the genuinely-new socket before chmod/chown. Socket is now reliably
secubox:secubox on start. Verified live: owner settles secubox:secubox and
nginx serves /api/v1/podcaster/status 200.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 09:04:52 +02:00
bfce737b1d fix(core): raise shared nginx proxy_read_timeout 30s → 300s
30s was too short for slow module operations behind the reverse proxy — a
podcaster audiobook upload (server-side ZIP extraction of ~120MB) exceeded it,
so nginx returned 502 on an upload that actually succeeded. 300s matches the
per-vhost timeouts already used elsewhere (lyrion, yacy). Applies to every
module served through secubox.d.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:59:01 +02:00
f554ec2f9a docs(webui): message boxes persist by severity — transient progress/success, persistent dismissable errors
Fatal messages (API/HTTP error, failed action, missing) must stay until the
user dismisses them, never a 3s flash. Codify the toast()/errorToast() split
and the "report the true server outcome" rule (a proxy timeout must not turn a
real success into a permanent-looking failure).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:58:25 +02:00
66cf9f4089 fix(podcaster): off-loop audiobook extraction + persistent error toasts (ref #853)
The audiobook ZIP handler extracted synchronously on the single-worker event
loop. A ~120MB/20-track audiobook took longer than nginx's 30s proxy_read_timeout,
so nginx returned 502 WHILE the extraction actually completed server-side — a
false-negative "Upload failed" on a successful upload (and it froze the service
meanwhile).

- extraction moved to asyncio.to_thread (_publish_audiobook_zip); the event
  loop stays responsive and the response is sent as soon as it finishes.
- ClientDisconnect during body streaming is caught → clean 400, not a 500.
- any extraction failure now deletes the partial feed (no orphaned feed; the
  prior code only cleaned up the "no audio found" case).

Webui (per operator guideline): fatal messages (API/HTTP/missing) now persist
until the user dismisses them (red toast with ✕); progress + success stay
transient. Routed all 7 error sites to errorToast.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:57:13 +02:00
ae98a76fe5 fix(billets): communiqué always-light + CSP-safe health-banner opt-out on public blog (ref #851)
Backport of two board-live fixes so a package reinstall can't revert them:

- billets-communique.css: the communiqué permalink extends the blog base
  (billets.css flips dark under prefers-color-scheme:dark), which leaked a dark
  ground into the always-light press-release. Pin color-scheme:light + html/body
  light. Bump billet_communique.html css ref to ?v=2.
- health-banner.js: public pages can now opt out of the operator banner with
  <meta name="sbx-no-health-banner"> (reliable — the WAF injects the loader
  after the page head; and CSP-safe, unlike an inline window flag). base.html
  emits it, so the public blog no longer shows the banner while admin pages
  (standalone templates) keep it. Replaces a board-only hostname hardcode.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:42:00 +02:00
0e41813e57 refactor(podcaster): use asyncio.wait to await cancelled download (review #853)
Replace `try: await cur except BaseException: pass` with `await asyncio.wait({cur})`
so awaiting the cancelled download's unwind no longer swallows cancellation of
the request handler itself (client disconnect). Pure idiom cleanup; behaviour
for the cancel-and-restart path is unchanged.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:35:03 +02:00
e0ec3ab417 fix(podcaster): dedup in-flight downloads + cancel-on-restart (review #853)
Adversarial review found a corruption ship-blocker: nothing prevented two
_download_one tasks for the same episode (manual ↻ button, auto-retry re-queue
and restart-recovery all feed the same queue). Two tasks share deterministic
tmp/dest paths → interleaved writes corrupt the audio and thrash state.

- _inflight registry keyed by episode id; the worker skips a queue entry whose
  download is still running (dedup).
- manual /episodes/{id}/download now cancels any live task and awaits its
  unwind before re-queuing, so ↻ genuinely restarts a wedged download instead
  of colliding or being deduped away.
- _download_one treats CancelledError as a clean abort (drop .part, no attempt
  charged), distinct from a real failure.

Test: worker runs one download for two same-id puts, and a cancel+re-put
starts a fresh attempt.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:29:36 +02:00
4e18347d7a fix(podcaster): timeout+auto-retry+restart-recovery for wedged downloads; live upload progress (ref #853)
Downloads no longer wedge a queue slot forever:
- real httpx.Timeout (connect/read/write) so a stalled stream aborts to error
  instead of hanging on the previous timeout=None
- bounded auto-retry (3 tries, 10/30/60s backoff) via a new attempts column
- restart-recovery: episodes orphaned in downloading/queued are re-queued when
  the worker (re)starts
- webui: manual "↻" restart button on downloading/queued episodes; a manual
  (re)download resets the retry budget

Audiobook ZIP upload now uses XHR with upload.onprogress: a persistent
"Uploading N%" indicator (was a fetch with a 3s toast that hid mid-upload),
then a persistent published/error result.

Store: idempotent attempts-column migration + requeue_stuck + bump_attempts,
with tests.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:23:46 +02:00
CyberMind
37688ebc60
Merge pull request #852 from CyberMind-FR/feature/851-billets-communique-style-embed-snapshot
feat(billets): communiqué style + embed snapshot vignette + portable HTML archive (#851)
2026-07-14 08:07:24 +02:00
965377388f fix(billets): SSRF — validate embed_url public-IP before Chromium screenshot (ref #851)
Review HIGH: the headless-browser snapshot path drove Chromium to embed_url with
no SSRF check (only the og:image fallback was guarded), so an author could
screenshot an internal service (127.0.0.1/RFC1918/::1/169.254.169.254). Now
ssrf.validate_url() gates the navigation (reject → fall back to og:image), plus a
Playwright route aborts any redirect/sub-resource to a non-public host. 12 tests.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 08:05:35 +02:00
a95f157eb1 feat(billets): communiqué style + embed snapshot vignette + portable HTML archive (ref #851)
- Per-billet 'style' (default|communique). Communiqué permalink = the poster
  mockup (Space Grotesk/JetBrains Mono, ROUTE side-band, TRANSMISSION frame,
  slogans/footer); funky feed unchanged.
- Embed snapshot vignette: api/services/snapshot.py tries headless Chromium
  (guarded/optional), falls back to SSRF-guarded og:image, re-encoded via the
  media service. Captured off-loop on save (communiqué billets, cached per
  embed_url). Public render shows the vignette; click-to-load the live iframe.
- Portable single-HTML archive: GET /admin/billets/{id}/archive.html — self-
  contained (inlined CSS, system fonts, media+snapshot as base64 data-URIs,
  embed = snapshot linking to original, no iframe). Off-loop render.
- migration 0003 (style + embed_snapshot); models/repo/admin/main wired.
- CSP: style-src adds 'unsafe-inline' so the WAF-injected health-banner's
  dynamic <style> renders (was blocked → banner fell unstyled to page bottom);
  script-src stays 'self'. Google-Fonts relaxation page-scoped to communiqué.
- 118 tests (11 new: snapshot fallback/SSRF, communiqué render, archive self-containment).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-14 07:57:54 +02:00
CyberMind
bb3c56c2cd
feat(#817): Device Guardian consolidation — merge 5 device modules into secubox-nac (Project A) (#818)
Some checks are pending
License Headers / check (push) Waiting to run
* docs(device-guardian): Project A consolidation design — merge 5 device modules into nac (ref #817)

* docs(device-guardian): Project A implementation plan — 9 tasks (ref #817)

* feat(nac): SQLite device store + idempotent legacy migration (ref #817)

Add packages/secubox-nac/api/store.py — pure-stdlib sqlite3 module with
canon_mac(), DeviceStore (WAL, best-value upsert, list/get/count,
history), and migrate_legacy() for idempotent, per-source best-effort
import of mac-guard/device-intel/iot-guard legacy shapes. Reserved
Project-B columns (plane/provenance/geo_cc/geo_asn) created but unused.

Task 1 of the Device Guardian consolidation plan (docs/superpowers/plans/
2026-07-05-device-guardian-consolidation.md).

* fix(nac): store — preserve first_seen, skip corrupt legacy sources, no stray db, SPDX (ref #817)

* feat(nac): unified dnsmasq+ISC+ARP discovery merge (ref #817)

* fix(nac): discovery — latest ISC lease wins the IP on renewal (ref #817)

The merge loop only updated ip/source on a strict rank increase
(incoming_rank > existing_rank), so equal-rank sightings for the same
MAC never touched ip/source. Since dnsmasq -> isc -> arp is processed
in non-increasing rank order, multiple ISC dhcpd.leases blocks for one
MAC (a fresh block per renewal) are equal-rank ties, and the first
(oldest) block's ip was frozen instead of the latest one winning, as
the original secubox-device-intel._get_dhcp_leases did.

Change the condition to incoming_rank >= existing_rank so the latest
same-or-higher-rank sighting updates source, and overwrites ip when
the incoming ip is non-empty. Hostname rule and the lower-rank branch
are unchanged, so a later ARP sighting still cannot clobber a
higher-rank ip/hostname/source.

Add test_isc_latest_lease_wins: two ISC lease blocks for one MAC with
ip/hostname renewal, asserting the latest block wins. Confirmed this
test fails against the pre-fix >-only logic.

* feat(nac): absorbed enrichers — OUI vendor + device-type/risk + OpenWrt fingerprint (ref #817)

* fix(nac): enrich — is_router strictly vendor-based; cover classify/oui edge cases (ref #817)

* feat(nac): background collector + double-cache; handlers def, no inline scan (ref #817)

Task 4 of the Device Guardian consolidation: a Collector background
loop now owns all discover -> enrich -> upsert work (replacing
_monitor_clients), publishing an in-memory snapshot(). The status,
clients, and client/{mac} handlers become plain def reading the SQLite
store / collector cache instead of calling _discover_clients() inline
on every request -- the #808 aggregator SPOF.

* fix(nac): finish #808 — def-convert alerts/list_quarantine/summary; first_seen + recent_events (ref #817)

alerts/list_quarantine were still `async def` calling blocking
_discover_clients() inline on the shared aggregator loop; summary
awaited alerts on top of that (double loop-block). Converted all three
to plain `def` reading the SQLite store/collector snapshot instead,
completing the #808 SPOF fix started for status/clients/client.

Also: collector.cycle_once() now stamps first_seen on newly discovered
devices (store.upsert keeps the earliest via COALESCE), and get_client's
recent_events merges the SQLite device_history (client_joined) with the
JSON event log so a device's join event shows up again.

* feat(nac): allow/deny actions + def-convert action handlers (finish #808) (ref #817)

Task 5: fold mac-guard's allow/deny into nac's zones/nft. New
POST /allow/{mac} and /deny/{mac} set allow_state in the SQLite store
and mirror it into nft's existing inet secubox_nac blocked/lan_allowed
sets via _nft_add_element/_nft_delete_element (mac-guard's separate
inet secubox_mac_guard table stays retired; its element migration is
Task 9).

Carried from Task 4 review: zones, add_to_zone, remove_from_zone,
approve_client, ban_client, update_client, quarantine_client and
unquarantine were still async def wrapping blocking subprocess/nft
calls on the shared aggregator loop (#808). Converted to plain def so
FastAPI threadpools them off the loop; add_to_zone/ban_client's webhook
notify now fires via a new _fire_webhook_sync helper
(asyncio.run_coroutine_threadsafe against the loop captured at
startup) since a plain def can no longer await _notify_webhooks
directly.

Tests: tests/test_actions.py mocks nft via monkeypatched
_nft_add_element/_nft_delete_element/_nft_list_set (+ subprocess.run,
since ban_client/unban_client build raw nft argv) into an in-memory
set-membership dict; covers deny/allow/allow-then-deny, zones,
add_to_zone+approve_client, and ban_client+unban_client.

* feat(nac): absorbed endpoints — vendors/scan/probe/mdns/groups/export (ref #817)

* fix(nac): /scan enriches + validates subnet; /probe IP + CSV injection guards (ref #817)

* feat(nac): retire mac-guard/device-intel/iot-guard via 308 redirects; auth-gate network-anomaly (ref #817)

secubox-mac-guard, secubox-device-intel and secubox-iot-guard are
superseded by secubox-nac's consolidated Device Guardian. Their
api/main.py are reduced to thin shims: every route 308-redirects to
its secubox-nac equivalent (/devices->/clients, /device/{mac}->
/client/{mac}, /whitelist|/allow->/allow, /blacklist|/deny->/deny,
/scan, /vendors, /groups, /export/* pass through unchanged, and a
catch-all forwards anything else), with an X-SecuBox-Deprecated
header. /health stays a plain 200 so liveness probes never 308-loop.
Package removal is a separate, gated follow-up.

secubox-network-anomaly stays (separate traffic-anomaly engine) but
its /status endpoint was unauthenticated while every sibling route
already required a JWT; it now matches.

* feat(nac): webui — vendor/type/risk columns, groups view, export; drop retired webuis (ref #817)

- clients.js: render enriched /clients fields — oui_vendor, device_type,
  risk_level (color-coded badge)/risk_score, and OpenWrt/SecuBox/Router
  fingerprint badges. All values are attacker-influenceable (DHCP
  hostname/vendor) and are rendered exclusively via E()/textContent,
  never innerHTML. Null/missing device_type/vendor/risk render as
  "unknown"/"—", never "null". Adds an Export CSV button (direct link to
  GET /export/csv — auth via the existing SSO session cookie, same as
  every other same-origin nav link in this app; no new auth scheme).
- groups.js (new): list/create/delete device groups and assign a device
  to a group, hitting GET/POST/DELETE /groups* — mirrors zones.js's
  structure and auth (sbxFetch).
- nav.js: register the new "Groupes" tab alongside Zones.
- Remove packages/secubox-{mac-guard,device-intel,iot-guard}/www — their
  functionality is now consolidated into nac; secubox-network-anomaly/www
  is untouched (stays standalone per plan).

Note for Task 9: device-intel/iot-guard's debian/rules unconditionally
`cp -r www/<name>/.` — now broken since www/ is gone; mac-guard's rule is
already guarded (`[ -d www ] && ... || true`). Needs a debian/rules fix
when Task 9 retires/redirects these packages.

* chore(nac): packaging — ieee-data dep, nft migration, retirement redirects, changelog 3.0.0; fix retired debian/rules www (ref #817)

* fix(nac): whole-branch — lazy init for in-aggregator mount, collector off-loop, sqlite lock, batched zone lookup, sentinel-preserve (ref #817)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(nac): risk_score split — known router low, unknown router high (rogue-AP signal) (ref #817)

Split the router risk axis out of the generic device-type/open-ports
scoring: a router/gateway already known (allow-listed or already in the
store) is trusted infrastructure -> low risk; a newly-seen/unknown
router is a rogue-AP signal -> high risk, regardless of open ports.
Removes the old flat -10 "trusted router" discount. Non-router devices
keep the existing scoring unchanged.

risk_score() gains an is_known=False parameter (backward compatible).
collector.cycle_once() and main.scan() now compute is_known from the
prior store row (existing row or allow_state == "allow") before
upserting, reusing the row they already fetch for is_new/enrichment.

* feat(nac): mesh-sync (p2p peers as devices) + fingerprinted-router classification for rogue-AP risk (ref #817)

Part A: absorb iot-guard's missed "mesh-sync" capability. GET /mesh/peers
and POST /mesh/sync now live in nac, reading live P2P peers over
/run/secubox/p2p.sock (HTTP-over-UDS GET /peers, fail-safe -> [] on any
error/timeout/absent socket) and upserting one sb:-prefixed synthetic
device per peer (iot-guard's exact md5-hash MAC scheme), device_type
"mesh_node", source "mesh". iot-guard's retired redirect-shim catch-all
already forwards /mesh/* here, so no shim change needed.

Part B: classify_device_type's "router" keyword list is much narrower
than ROUTER_VENDORS (misses tp-link/asus/linksys/d-link/gl.inet/xiaomi/
huawei), so a router-vendor MAC with a generic hostname fingerprinted
is_router=True still classified as device_type="unknown" and skipped
risk scoring entirely (the omit-unknown guard) -- the known-router-vs-
rogue-AP HIGH signal never fired for the most common vendor case. Both
enrichment call sites (Collector.cycle_once and /scan) now reclassify
device_type="unknown"+is_router to "router" before scoring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(nac): postinst ensures secubox_nac table + zone sets (incl lxc_zone)

Nothing shipped created nac's inet secubox_nac table or its zone sets — they
were assumed to pre-exist (they don't: the live board had no such table), so
operator 'move to zone' failed for every zone and the mac-guard element
migration silently no-op'd. postinst now creates the table + all ZONES sets
(lan_allowed, lxc_zone, iot_zone, guest_zone, quarantine_zone) + blocked,
idempotently and best-effort, before the migration. Makes operator moves to
the LXC Containers zone work on a fresh install. nft commands validated on
gk2 (create/idempotent re-add/add-element/list), no live drift left.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>

---------

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 09:58:01 +02:00
b800b624d7 docs: track metalogue OSINT suite (#845) — wiki + WIP + HISTORY + TODO
Some checks are pending
License Headers / check (push) Waiting to run
- wiki/Metalogue.md (use cases: Maigret identity collector, stylized PDF report,
  SpiderFoot automation, OpenCTI deferred) + sidebar entry.
- WIP: metalogue section + follow-ups (OpenCTI P3, ip_forward drift, aggregator
  self-register, SpiderFoot re-enable on fresh image).
- HISTORY: dated 2026-07-12 entry (both modules, install root-causes, PDF report).
- TODO: metalogue follow-ups section.
- Adds the implementation plan doc.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-13 05:44:28 +02:00
CyberMind
ea04f9441a
Merge pull request #850 from CyberMind-FR/feature/845-metalogue-maltego-style-osint-suite-as-l
Some checks are pending
License Headers / check (push) Waiting to run
feat(maigret): stylized PDF report (formal dossier from raw lookup) (#845)
2026-07-12 15:24:05 +02:00
493e2bd2c3 feat(maigret): stylized PDF report from raw lookup (ref #845)
Turn a maigret --json simple lookup into a FORMAL, SecuBox-branded PDF dossier
(api/report.py, fpdf2): dark cyber masthead, a document metadata block
(REFERENCE/SUBJECT/GENERATED/SOURCE/SCOPE/CLASSIFICATION), a numbered §1 Summary
(sentence + stat cards), and §2 Findings as a coherent results TABLE (# /
platform / category / profile URL / details) — zebra rows, category colour-coding,
paginated with a repeating header. Latin-1 sanitised (unicode profile data never
crashes core fonts).

- GET /api/v1/maigret/lookup/{id}/report.pdf (JWT-gated, plain def →
  threadpooled so the fpdf render never blocks the aggregator loop).
- Panel: 📄 PDF button per completed lookup → authed fetch → blob download.
- python3-fpdf2 dependency. 4 report tests (valid PDF, claimed-only, empty/
  malformed safe, unicode safe) + 8 api tests green. Deployed; route live (401 gated).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 15:24:00 +02:00
CyberMind
d35307460a
Merge pull request #849 from CyberMind-FR/feature/845-metalogue-maltego-style-osint-suite-as-l
feat(spiderfoot): public UI vhost spiderfoot.gk2.secubox.in (#845)
2026-07-12 15:05:53 +02:00
b2abaf714f feat(spiderfoot): expose UI at https://spiderfoot.gk2.secubox.in (root proxy) (ref #845)
SpiderFoot emits absolute /static paths, so it can't live under a subpath of the
admin.gk2 portal SPA (/spiderfoot-ui/ returned the SPA's 'module not found').
Fix: serve it AT ROOT on its own vhost (zigbee/lyrion pattern):
- spiderfoot.gk2.conf: public vhost (:9080 server_name spiderfoot.gk2 → proxy
  10.100.0.43:5001 at root, LAN-gated exposure snippet, ACME). Chain: HAProxy
  (wildcard *.gk2 cert) → sbxwaf → nginx → container. Live-wired: sbxwaf route +
  HAProxy ACL (mitmproxy_inspector) + exposure snippet.
- spiderfoot-lan.conf: LAN-direct fallback (host :9043 → container), nft :9043
  allowed + persisted.
- Removed the broken /spiderfoot-ui/ location; panel 'Open SpiderFoot' → the
  public https URL. rules ship both vhosts to sites-available (operator-enabled).

Verified: https://spiderfoot.gk2.secubox.in/ returns 200 (SpiderFoot v4.0.0), assets load.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 15:05:48 +02:00
CyberMind
bc6b5ec2c1
Merge pull request #848 from CyberMind-FR/feature/845-metalogue-maltego-style-osint-suite-as-l
fix(metalogue): arm64 container installs via piwheels + ip_forward note (#845)
2026-07-12 14:53:25 +02:00
e67e393abb fix(metalogue): container installs use piwheels + dev headers (arm64) (ref #845)
Root cause of failed installs on gk2: maigret/SpiderFoot pull C-extension deps
(lxml, pandas, cryptography, netaddr…) with no stock arm64 wheels on PyPI, so a
minbase LXC can't pip-install them. Fix: pip --extra-index-url piwheels (prebuilt
aarch64 wheels — install drops from a 17-min failing compile to ~2 min) + apt
build-essential/dev headers as fallback. Install logs no longer /dev/null'd so
failures are diagnosable.

NOTE: containers also need host ip_forward=1 for egress via the masquerade NAT;
99-secubox-zz-lxc-forward.conf sets it (zz- to win over the hardening drop-in),
but the runtime value had drifted to 0 — re-applied with 'sysctl --system'.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 14:51:20 +02:00
CyberMind
8b472eaaab
Merge pull request #847 from CyberMind-FR/feature/845-metalogue-maltego-style-osint-suite-as-l
fix(spiderfoot): nginx duplicate proxy_read_timeout (deploy fix, #845)
2026-07-12 11:38:27 +02:00
7a931a000a fix(spiderfoot): nginx /spiderfoot-ui/ duplicate proxy_read_timeout (ref #845)
The secubox-proxy snippet already sets proxy_read_timeout, so including it AND
setting 300s in the same location failed nginx -t. Set the proxy headers inline
(like fmrelay's stream location) with a single 300s timeout for long SpiderFoot
scans. Verified nginx -t + reload on gk2.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 11:38:22 +02:00
CyberMind
0f226caee6
Merge pull request #846 from CyberMind-FR/feature/845-metalogue-maltego-style-osint-suite-as-l
feat(metalogue): OSINT LXC modules — secubox-maigret + secubox-spiderfoot (P1+P2 of #845)
2026-07-12 11:33:07 +02:00
f1cbb49937 feat(metalogue): secubox-spiderfoot OSINT automation engine + auth-gate both UIs (ref #845)
- secubox-spiderfoot: SpiderFoot (200+ passive modules, correlation graph =
  interim Maltego-style hub) in an LXC container (10.100.0.43, mem 1536M).
  spiderfootctl lifecycle (debootstrap + git clone + venv + in-container
  systemd unit binding sf.py to the container LAN IP only, never a host port);
  api/main.py aggregator-served, plain-def handlers, detached install, JWT +
  audit on install/start/stop/restart-ui; status probe = bounded curl to the
  container UI port. Panel + menu.d 711 + sudoers least-priv. 4 guard + 10 API tests.
- SECURITY (review HIGH): SpiderFoot has no native auth, so every nginx
  location (panel, API, and the /spiderfoot-ui/ container-UI proxy) now carries
  the shared SecuBox auth gate (auth_request /__sbx_auth_verify + @sbx_auth_login,
  LAN-pass/else-deny), matching the peer pattern (fmrelay/grafana). Applied the
  same gate to secubox-maigret's locations for consistency (defense-in-depth on
  top of its app-layer require_jwt).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 11:31:54 +02:00
4535b5c118 feat(metalogue): secubox-maigret — username/identity OSINT LXC module (ref #845)
Maigret (username -> accounts across 3000+ sites) wrapped as a SecuBox module,
adapted from the openclaw LXC pattern:
- maigretctl privileged control plane (LXC 10.100.0.42, memory.max 1024M);
  username passed ONLY positionally to lxc_attach (no shell interpolation),
  flag-injection guards (_valid_username/_valid_id) + __guard test verb.
- api/main.py aggregator-served, all handlers plain def, detached workers
  (start_new_session) — nothing blocks the shared loop. JWT + append-only audit
  on lookup/delete/install; concurrency cap (max_concurrent_lookups=3, 429) so a
  loop of POSTs can't saturate the board; 15s single-flight status cache.
- nginx secubox.d route, menu.d 710 navbar tile, cyan guideline panel
  (sbx_token bearer, esc() everywhere), sudoers least-priv (only maigretctl).
- Tests: 21 guard assertions + 8 API tests (mocked ctl layer). Passive-only.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 11:13:58 +02:00
CyberMind
6e39c415ac
Merge pull request #844 from CyberMind-FR/feature/billets-microblog
docs(wiki): Billets use-cases page
2026-07-12 09:26:01 +02:00
fbd7e6d19c docs(wiki): Billets page — use cases (micro-blog, media gallery zoom, gateway, backup, operator override)
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 09:24:39 +02:00
CyberMind
810ebb7ae5
Merge pull request #842 from CyberMind-FR/feature/billets-microblog
feat(billets): image gallery + zoom, portable export, operator panel + password override
2026-07-12 08:28:24 +02:00
77435afbc9 fix(billets): review — export off-loop, override bytes-compare, upload feedback
Adversarial review follow-ups:
- /admin/export.sbxsite: base64 + json.dumps now run in a thread (asyncio.to_thread),
  not on the shared event loop — a photo-heavy backup no longer stalls the module.
- /admin/override: compare the operator secret on bytes; a non-ASCII input used to
  raise TypeError in hmac.compare_digest → 500. Now a clean rejection + audit event.
- Uploads: surface the skipped-file count (?media_skipped=N + dashboard flash)
  instead of silently dropping invalid/oversized images. 107 tests.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 08:28:18 +02:00
b80f445629 feat(billets): complete operator panel + generate-password override
- /billets/ operator panel: full Administration card linking every admin
  surface (Dashboard, Nouveau billet, Modération, Backup .sbxsite, Change-pw,
  Flux) instead of a couple of buttons.
- Forgotten-password OVERRIDE for the operator: GET/POST /admin/override — no
  session, gated by the module operator secret (/etc/secubox/secrets/billets,
  root-only, constant-time compare, rate-limited 5/h). Generates a strong
  random password (token_urlsafe), shows it once (rendered in the POST response,
  never in a URL), audit-logged (author.password_overridden / override_failed).
  Linked from the panel; the reset page is same-origin on billets (no CORS,
  no cross-origin secret handling).
- billets-admin.css: flash.good + code tokens. 106 tests (2 new override).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 08:20:02 +02:00
db74e02ffc feat(billets): image upload with zoomable vignette gallery + portable export
- Upload images (png/jpeg/webp/gif ≤5 MiB) alongside the text of a billet:
  multi-image mini-gallery. New api/services/media.py validates + FULLY
  re-encodes every upload via Pillow (drops EXIF/GPS, neutralises polyglots;
  SVG refused), producing a cleaned original + a ≤480px thumbnail.
- Storage under BILLETS_MEDIA_DIR (/var/lib/secubox/billets/media), served by
  nginx /media/ alias (in-process StaticFiles fallback). img-src 'self' already
  covers it — no third party, no CSP relaxation.
- Public: vignette grid in feed + permalink, pure-JS zoomable lightbox with
  ◀▶ keyboard/arrow navigation (graceful degradation: links open the full
  image with JS off). First image drives og:image / twitter:image.
- Admin editor: multipart file input + existing-media thumbnails with delete.
  New media table (migration 0002, ON DELETE CASCADE), repo CRUD, media delete
  route; billet delete removes files. All Pillow work off the event loop
  (asyncio.to_thread).
- Portable backup: GET /admin/export.sbxsite emits every billet + media inlined
  as base64 (single re-importable file).
- nginx client_max_body_size 6m; requirements pillow==11.1.0. 104 tests (11 new).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 08:11:43 +02:00
CyberMind
3c724bb6ff
Merge pull request #841 from CyberMind-FR/feature/billets-microblog
fix(billets): guideline panel reskin + per-client login rate-limit
2026-07-12 07:52:07 +02:00
0d50cf6a51 fix(billets): reskin /billets/ panel to WebUI guideline + per-client login rate-limit
- /billets/ management panel rebuilt to the SecuBox WebUI Panel Guidelines
  (certs reference): restrained header + stat grid + glass cards + dense list
  + toast, dropping the funky hero/blobs. Cyan, Courier Prime, emoji glyphs.
- Login rate-limit now keys on the real client via X-Forwarded-For (admin
  _client_ip) instead of the shared proxy IP, so one operator's attempts no
  longer lock out everyone. nginx preserves the XFF chain (X-Real-IP +
  $proxy_add_x_forwarded_for).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 07:51:44 +02:00
CyberMind
c388266622
Merge pull request #840 from CyberMind-FR/feature/billets-microblog
feat(billets): admin password reset + guideline reskin + doc
2026-07-12 07:15:44 +02:00
0c74eddc0e feat(billets): admin password reset + guideline reskin (cyan hybrid-dark) + doc
- Admin webui: POST /admin/password (verify current, argon2, event-logged,
  forces re-login). Change-password card in the dashboard.
- Reskin billets admin (login/dashboard/editor/moderation) + the /billets/
  management panel to the SecuBox WebUI Panel Guidelines: Courier Prime, cyan
  #00d4ff accent, glass cards, emoji (certs as reference). Public blog stays funky.
- New self-contained api/static/billets-admin.css (module-vhost variant of the
  guideline, no /shared/).
- Docs: WEBUI-PANEL-GUIDELINES.md v1.1 (canonical DEFAULT for webui module
  management, billets added as reference, 2 variants documented) + CLAUDE.md pointer.
- 93 tests (3 new password-reset).

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-12 07:15:15 +02:00
CyberMind
939c14cf09
Merge pull request #838 from CyberMind-FR/feature/billets-microblog
feat(billets): funky public webapp — inline media, link chips, quick-share
2026-07-11 16:36:08 +02:00
ac6df22258 feat(billets): funky/party public webapp — inline media + link chips + quick-share
Public feed + permalink redesigned funky/party (theme-aware gradients, emoji
accents, playful cards). Embeds now render INLINE in the feed too (responsive
16:9-ish media, CSP frame-src augmented for self-hosted hosts). Inline embedded
reference links shown as 🌐 chips. Prominent emoji quick-share row (Bluesky/X/
WhatsApp/Telegram/Reddit/LinkedIn/email + Mastodon/copy via billets.js) on every
billet in the feed and permalink (replaces the hidden Republier details).
Asset URLs versioned (?v=2) to bust the sbxwaf media cache. 90 tests.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-11 16:35:56 +02:00
CyberMind
4a9545b09e
Merge pull request #837 from CyberMind-FR/feature/billets-microblog
feat(billets): SecuBox webui panel + navbar integration
2026-07-11 16:24:09 +02:00
5e77b9ebbd feat(billets): SecuBox webui panel + navbar integration
menu.d/415-billets.json (category mind) surfaces Billets in the SecuBox sidebar.
A funky, emoji-rich hybrid-dark panel at /billets/ (admin webui) with live stat
count-ups, reaction breakdown, latest billets and action buttons — fed by a new
CORS-enabled GET /stats.json (public aggregate counts). Fixed a latent import-time
BILLETS_DB read (now resolved at connect()). 89 tests.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-11 16:19:35 +02:00
CyberMind
0b4e0fe83c
Merge pull request #836 from CyberMind-FR/feature/billets-microblog
fix(billets): live-deploy hotfixes (linkify, ExecStartPre, compat)
2026-07-11 16:04:50 +02:00
084120c858 fix(billets): deploy hotfixes — disable markdown-it linkify (optional dep), drop failing ExecStartPre, remove debian/compat
Found during live deploy to billets.gk2.secubox.in:
- markdown-it linkify=True needs linkify-it-py (not in requirements) -> 500 on any
  body with a URL; spec only needs markdown [links], so linkify is off (bare-URL
  autolink stays for comments via linkify_plain).
- systemd ExecStartPre tried to chown the shared /run/secubox as the secubox user
  -> service failed to start; removed (tmpfiles already provides 1777 /run/secubox).
- debian/compat conflicted with debhelper-compat in control; removed.

Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
2026-07-11 16:04:24 +02:00
CyberMind
e33dc4d1bb
Merge pull request #835 from CyberMind-FR/feature/billets-microblog
feat(#834): billets — micro-blog gateway inter-médias sociaux
2026-07-11 15:55:22 +02:00