Commit Graph

123 Commits

Author SHA1 Message Date
CyberMind
82b0522c5b
Aggregator résilience: sweep async (243 handlers) + threadpool 80 (#738) (#739)
* perf(aggregator): async-sweep — 243 blocking route handlers async def→def (#738)

Mounted in the aggregator's single event loop, an 'async def' route handler
that runs blocking code (subprocess/journalctl/openssl/argon2) freezes the
WHOLE loop -> aggregator.sock Connection refused -> 502 board-wide.

Deterministic AST codemod (scripts/async-sweep.py) converts route handlers
that (a) are decorated with an HTTP verb, (b) contain a known blocking call,
(c) have NO await/async-with/async-for/yield, (d) are never used as a
coroutine elsewhere -> plain 'def'. Starlette then runs them in the AnyIO
threadpool, so the blocking call no longer stalls the gateway. await/stream/
websocket handlers are left untouched. Every file py_compile-checked.

243 handlers across 56 modules (system 17, qos 13, netdiag/hexo 12, hub 11...).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* perf(aggregator): raise AnyIO threadpool to 80 tokens (#738)

The async-sweep moves ~243 blocking handlers to the threadpool. With ~110
modules in one process, the default 40-token pool can queue head-of-line under
concurrent blocking load. Raise to 80 on startup (best-effort, never breaks
boot).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* perf(hub): double-buffer status cache + emojized health page (#738)

Navbar status (menu + health-batch) is now a strict double-buffer cache:
- request handlers NEVER compute on the request path — they return the current
  snapshot instantly (or a 'warming' placeholder), so the sidebar's polling can
  no longer serialize behind a ~3s systemctl walk and starve the loop;
- the background refresher is kicked from the request path (_ensure_bg) because
  mounted sub-apps receive neither startup nor @app.middleware events under the
  aggregator — the previous lazy-start middleware never fired there;
- snapshots are built complete then swapped atomically, so the dashboard never
  shows partial/bad counts.

Served by the dedicated secubox-hub process (:8001, isolated loop) the navbar
stays <50ms and holds 200 under 25+ concurrent polls where the aggregator-
mounted copy wedged (000). health.js: 🟢🟡🔴 emoji status indicators.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style(hub): render health status emoji cleanly (neutralize .led dot) (#738)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* perf(sidebar): kill per-module health storm, use batch endpoint (#738)

The navbar refreshed LEDs by firing ONE /api/v1/<module>/health request per
module — ~119 requests every 30s, in batches of 8 — straight at the aggregator's
single shared event loop. Combined with the in-process module mount this is a
prime driver of the recurring board-wide 502 wedge (user-identified).

checkAllHealth + refreshStaleHealth now call /api/v1/hub/public/health-batch
ONCE (served by the dedicated, double-buffered hub process) and populate every
module's LED from that single response. 119 reqs/cycle -> 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 08:06:15 +02:00
6a557cbe2c fix(image): dearmor apt keyring for signed-by (apt rejects armored) (ref #760)
build-image.sh / build-installer-iso.sh / validate-staged-repo.sh fetched the
ASCII-armored secubox-keyring.gpg and used it directly as
`signed-by=/usr/share/keyrings/secubox.gpg`. apt's signed-by requires a
DEARMORED (binary) keyring, so the chroot apt-get update failed with
"NO_PUBKEY 44E50F0178E8BC7E / repository not signed" even though the repo
signature and key are correct. Pipe the key through `gpg --dearmor`.

Root cause of all image-build failures since the keyring became armored;
not a key desync (repo signer == published keyring == 44E50F0178E8BC7E).
2026-06-27 17:44:13 +02:00
c6d6eb5c75 Merge #744: sbxwaf Go WAF engine + shared internal/ core
Some checks are pending
License Headers / check (push) Waiting to run
# Conflicts:
#	packages/secubox-toolbox-ng/cmd/sbxmitm/compress_test.go
#	packages/secubox-toolbox-ng/cmd/sbxmitm/cosmetic_test.go
#	packages/secubox-toolbox-ng/cmd/sbxmitm/gzip.go
#	packages/secubox-toolbox-ng/cmd/sbxmitm/gzip_test.go
2026-06-26 16:02:35 +02:00
11438e394c docs(sbxwaf): bench harness + cutover/rollback runbook with parity-gap gates (ref #744)
- scripts/sbxwaf-bench.sh: wrk/hey bench harness against legacy mitmproxy
  (10.100.0.60:8080) and shadow sbxwaf (127.0.0.1:8081); captures req/s,
  p99, RSS; prints comparison table with PASS/FAIL for all 3 go/no-go gates
  (>5× req/s·core, p99<1/3, RSS<1/4); shellcheck clean.

- packages/secubox-waf-ng/docs/CUTOVER.md: operator runbook with 6 sections:
  pre-cutover checklist (CA, CrowdSec JWT, COMPLETE log4shell corpus,
  null-byte \x00 fix, goform FP fix, parity green), shadow-run procedure,
  go/no-go gate table, exact HAProxy server re-point + nftables DNAT topology,
  single-edit rollback, and post-cutover monitoring (threat log, cookie-audit,
  RuntimeDirectoryPreserve guarantee, CrowdSec JWT rotation constraint,
  Python WAF unescaped-Host XSS backport note, body URL-decode limitation).
2026-06-26 15:02:18 +02:00
c01f10c474 docs(wiki): refonte 126 modules — snapshots WebUI déterministes + READMEs 4 langues (ref #742)
Capture des 126 dashboards authentifiés (JWT minté serveur, injection localStorage)
avec attente de COMPLÉTION d'affichage déterministe (sentinelles de chargement
purgées + réseau calmé, plafond --delay) au lieu d'un sleep fixe. 126/126 OK.

- capture-screenshots.py: _wait_content_ready déterministe, mode --token (bypass
  login), goto domcontentloaded (plus de stall networkidle sur dashboards live)
- generate-docs.py: 23 modules jusqu'ici non documentés ajoutés (descriptions
  réelles depuis debian/control, EN/FR/DE/ZH), licence MIT→LicenseRef-CMSD-1.0,
  images wiki en URL raw.githubusercontent absolue (relatif = 404 sur GitHub wiki)
- 126 snapshots + thumbnails régénérés
- 126 READMEs paquet succincts + pages wiki MODULES/CATEGORIES ×4 langues

Gap modules documentés vs découverts: 23 → 0.
2026-06-26 10:14:14 +02:00
845683c9c9 fix(metablog): chown site dirs to secubox:secubox after ingest (ref #121)
ingest_site() runs git init/commit inside /srv/metablogizer/sites/<name> over
ssh as root, so a freshly created .git is owned root:root — but the
metablogizer service runs as 'secubox' and cannot write the repo (blocked
sub-E #113 webhook deploys). Add a fix_perms helper that chown -R
secubox:secubox the site dir after every .git-touching ingest path
(ingested-fresh x2 + ingested-with-history). Matches module postinst pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 09:30:13 +02:00
6b7d7f8607 fix(#623): shared /run|/var/lib|/var/cache|/etc/secubox parents stay 1777/0755 in all postinsts
Systemic clobber: the scaffold boilerplate (install -d -m 750 /var/lib/secubox,
/run/secubox) put restrictive modes on SHARED parents in ~56 module postinsts,
reverting them to 0750 on every install/upgrade and breaking traversal for
non-secubox daemons (kbin/toolbox 500). Empirically confirmed install -d -m only
modes the final component, so /parent/leaf forms are harmless — only bare-parent
targets were rewritten. Multi-arg lines (incl. ones making /var/lib world-writable
1777) split per-parent: /run/secubox=1777 root:root, /var/lib|cache|etc=0755
secubox:secubox; module-private leaves keep 0750. Scaffold + PATTERNS.md fixed so
new packages don't reintroduce it.
2026-06-18 10:32:58 +02:00
c64a666fa5 fix(build): build-packages.sh passes -d; docs: apt repo published + signed
Some checks are pending
License Headers / check (push) Waiting to run
build-packages.sh omitted -d, so dpkg-checkbuilddeps silently dropped
arch:all packages (incl. secubox-core) from the repo. All 144 packages now
build + are published to apt.secubox.in, signed with apt@secubox.in.
2026-06-15 15:14:55 +02:00
154872da9f feat(scripts): add idempotent 'harden' subcommand — key-only sshd + nft SSH-guard (ref #529)
After the wg-admin tunnel is confirmed working, 'harden' closes the
public SSH brute-force surface:
  - sshd drop-in: PasswordAuthentication no + PermitRootLogin
    prohibit-password (refuses if no root authorized_keys — no lockout).
  - nft SSH-guard: drop tcp/22 from sources outside the LAN (192.168.1.0/24)
    + 10.x tunnels, inserted BEFORE the blanket 'iif eth1 accept' so the
    router's port-forwarded public SSH (real public src IP, DNAT no-SNAT)
    is dropped while LAN + wg-admin stay reachable. Persisted in
    /etc/nftables.conf + applied live without flushing other tables.

Applied + verified on gk2: tunnel SSH (10.98.0.1) key-only works; public
admin.gk2.secubox.in:22 now times out. Tables (blacklist/wg) intact.
2026-06-12 15:09:00 +02:00
3c289c50b9 fix(scripts): local loop vars in write_server_conf so client $pip isn't clobbered (ref #529)
The while-read in write_server_conf reused the global pip/ppub names, so
after it ran the caller's $pip (the new peer IP for the client config)
was wiped to empty — the emitted client config showed 'Address = /32'.
The provisioned server state was always correct; only the printed config
was wrong. Loop now uses local _pn/_pip/_pub.
2026-06-12 14:49:38 +02:00
8f2a5c1608 feat(scripts): admin WireGuard tunnel setup script (wg-admin, ref #529)
Idempotent provisioner for a dedicated out-of-band admin tunnel,
distinct from wg-toolbox:
  - wg-admin UDP 51821, server 10.98.0.1/24, peers 10.98.0.2+/32.
  - generates server+peer keys once (0600, never in repo), writes
    /etc/wireguard/wg-admin.conf from a peers state file.
  - installs /etc/nftables.d/secubox-admin-wg.nft (allow udp/51821,
    persistent) — ADDITIVE, does not touch sshd or existing rules.
  - 'add <name>' mints a client .conf + QR (split-tunnel, endpoint
    admin.gk2.secubox.in:51821).
sshd hardening (key-only + restrict 22) is a deliberate separate step.
2026-06-12 14:47:21 +02:00
561007fe74 fix(build): scripts/build-packages.sh — auto-discover packages instead of hardcoded list
The PACKAGES array was hardcoded to 30 entries, last updated months
ago. As of master at a28984c1 there are 140 secubox-* package
directories with debian/control; the script silently skipped the 110
that weren't in the list (operators got 0 build output for those).

Surfaced today during the consolidation-deploy: the 9 PRs merged into
master touched 31 packages, but the build script only built 13
because the other 18 weren't in the hardcoded list. The 18 had to be
built manually with direct dpkg-buildpackage calls.

Replace the static array with a glob over packages/secubox-*/ that
keeps only directories with debian/control. Build order: secubox-core
first (universal Depends:), metapackages secubox-full / secubox-lite
last, everything else alphabetical between them. The exact order
doesn't matter for dpkg-buildpackage (each builds independently); the
sort keeps the build log readable and surfaces core/meta failures
early/late respectively.

Verified: bash scripts/build-packages.sh bookworm all --dry-run now
plans 140 builds (vs 30 before) with the expected first/last order.
2026-05-27 10:24:16 +02:00
a85129f5c6 docs(audit): Phase 1 consolidation audit — script + findings (ref #378)
Adds scripts/audit-packages.py to scan packages/secubox-*/ and emit
structured artifacts under out/ (gitignored): packages.json,
packages.dot dependency graph, empty-packages.txt, prefix-based
clusters, and a hand-curated fuzzy-cluster report with per-member
evidence (deps, routes, services).

Findings doc 2026-05-27-secubox-consolidation-audit.md synthesises the
data into Tier 0-4 merge proposals grounded in real package contents,
not the plan stub's illustrative guesses. Key corrections vs the stub:

- 141 real packages (not "100+"); secubox-core is the substrate for 131
- streamlit-idle is already an in-package service variant, not a
  separate package — that consolidation is half-done
- meta{blogizer,bolizer,catalog,oblizer} are real distinct services,
  not typo proliferation (recommend rename for clarity, not merge)
- Two genuine packaging bugs surfaced: secubox-c3box binary name
  collision between Python dashboard and Go daemon (issue #378),
  secubox-smart-strip has no debian/ directory at all

Also imports two pre-existing 2026-05-26 plan stubs (consolidation,
build-scripts-refactor) that had been sitting untracked.

Re-run after any packaging change: python3 scripts/audit-packages.py
2026-05-27 07:14:48 +02:00
65c64411ae fix(led-heartbeat): v2.1.1 — cap brightness at 10 to keep IS31FL319X i2c bus responsive
The IS31FL319X chip on the MOCHAbin's i2c-1 bus errors out with -EIO
above brightness ~50. Pushing 100 or 255 saturates the bus, the chip
starts NACK'ing the next register writes, the LED driver logs
'Setting an LED's brightness failed (-5)' on every retry, and the
LEDs freeze at whatever value last got through.

Observed live 2026-05-24 on gk2: all three RGBs stuck plain green at
218/255/255 with the bash bumper and the python heartbeat both pushing
high values into a saturated chip. After dropping to brightness 10
and adding a 30 ms i2c-write delay in the python set_rgb() loop,
writes succeed and the LEDs reflect the actual health state instead
of a static color.

Files:
  - usr/sbin/secubox-led          BRIGHTNESS 100 -> 10
  - usr/sbin/secubox-healthbump   ACTIVE 100 -> 30, SLEEP 20 -> 5
  - usr/sbin/secubox-led-pulse    BRIGHTNESS 100 -> 10
  - usr/sbin/secubox-led-heartbeat python set_rgb scales any 0-255
    caller down to 0-10 in write_text, then sleeps 30 ms before the
    next channel — matches the bash scripts' I2C_DELAY and lets the
    mv64xxx_i2c controller drain between calls.
  - scripts/secubox-healthbump (the duplicate script kept in repo root
    for ad-hoc dev use) gets the same BRIGHTNESS=10 cap.

The misleading 'BRIGHTNESS=100 works reliably' comment that the code
contradicted is replaced with an incident note pointing at the EIO
ceiling.
2026-05-24 08:35:22 +02:00
CyberMind
ff990ba112
docs(mochabin): EP06 mPCIe GPIO investigation runbook + hardened probe script (closes #345) (#359)
* docs(mochabin): add EP06 mPCIe GPIO investigation runbook (ref #345)

scripts/probe-mpcie-gpios.sh: empirical CP0 GPIO sweep that drives
each unrequested line HIGH for a few seconds and watches dmesg/lsusb
for new USB device. Skips [used] lines, uses libgpiod gpioset
--mode=time so the line auto-restores to input on release. Modes:
--baseline (snapshot only, safe), --line gpiochipN K (single line),
no arg (full sweep gpiochip1+gpiochip2).

docs/hardware/mochabin-mpcie-ep06-runbook.md: hypothesis, procedure,
and DTS rfkill-gpio template patterned after cn9132-clearfog.dts
lines 69-122. Apply once a candidate GPIO is identified.

--baseline smoke-tested on gk2 — sweep deferred until spare EP06
arrives (no mPCIe device in J5 = no signal regardless of which line
drives W_DISABLE#).

* docs(mochabin): drop EP06 spare assumption from runbook (ref #345)

No spare EP06 on order; the existing EP06-E will be used for the
sweep when re-seated in J5. Replaces "deferred until spare" framing
with a multimeter-scoping fallback for the inconclusive case.

* fix(mpcie): v0.2.0 probe-mpcie-gpios.sh — drop blanket sweep, danger-list, dry-run default (ref #345)

The v0.1.0 "drive every unused-input CP0 GPIO HIGH for 3s one-at-a-time"
sweep took gk2 hard-down within seconds (host became unreachable; ARP
stopped responding). gpioinfo's [used] tag only reflects lines the
kernel *requested* — several unrequested CP0 GPIOs are physically
wired to critical board functions (eth switch reset, PCIe2 PERST#,
pca9554 IRQ) and driving them HIGH wedged the board.

v0.2.0:
- No blanket-sweep mode. Removed.
- Default mode = dry-run candidate listing only. No GPIO is driven.
- --commit --line CHIP N required to actually drive a single line.
- Hardcoded DANGER_LINES (gpiochip1:1, gpiochip1:9, gpiochip1:27)
  skipped unconditionally, even with --commit.
- gpiochip2 entirely held off without --allow-chip2 (no per-line
  schematic mapping for that bank).
- Hard refusals are vocal (REFUSED: ... reason) before exit.

Runbook updated to walk through the new dry-run → --commit flow.
HISTORY documents the incident + the v0.2.0 mitigations.
Lesson saved in memory feedback_gpio_blanket_sweep_crashed_board.md.

---------

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
2026-05-22 17:50:33 +02:00
CyberMind
b6923cd051
fix(image): replace emojis in /etc/issue with VT102-safe ASCII tags (#339)
Symptom: pre-login banner on ttyS0 (Minicom 2.9 VT102) rendered as
trailing `�` after every emoji line. The console driver couldn't
decode the UTF-8 multi-byte sequences for  🔐 🌐 📡 and emitted the
replacement character instead.

The `█` box-block characters (U+2588) DO render correctly in VT102+
UTF-8 — they're a single-byte-class glyph in most modern fonts. We
keep those so the SECUBOX wordmart still looks like SecuBox.

Replacements (compatible with VT102 minicom AND SSH UTF-8 alike):
     ->  >>
  🔐   ->  [user]
  🌐   ->  [web]
  📡   ->  [ssh]

The /etc/motd block (shown after SSH login) is left untouched — SSH
sessions have proper UTF-8 terminals so the rich emojis are fine
there.

Also add scripts/build-kernel-local.sh (new): the local kernel build
script that I authored today for #255. Documented; runs the same
defconfig + merge_config + scripts/config + olddefconfig + sed ZRAM
override + bindeb-pkg flow as .github/workflows/build-kernel.yml,
without the 20-min GitHub Actions ping-pong.

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
2026-05-22 10:07:15 +02:00
CyberMind
3435cfa069
feat(scripts): add check-dashboard-cache.py lint + CI (closes #147) (#148)
Anti-regression for the double-cache pattern documented in CLAUDE.md
(« Performance Patterns — Double Caching »). Catches the bug class that
caused PR #146 (secubox-waf 17 % sustained CPU under SOC polling)
before it lands.

What's added
============
- scripts/check-dashboard-cache.py — AST-based audit. Walks every
  packages/secubox-*/api/main.py, flags hot dashboard routes
  (/stats, /alerts, /summary, /dashboard, /metrics, /overview,
  /events, /history, /recent, /bans, /sessions, /connections)
  whose handler body does per-request I/O (open, read_text,
  read_bytes, subprocess.run, Popen) AND doesn't go through a cache
  (module-level _cache instance + .get() call, OR
  asyncio.create_task(refresh_*) at startup).
    * --check : exit 1 on unjustified violations (CI mode)
    * --report : human-readable table (default)
    * --json : machine-readable

- .cache-lint-allowlist.toml — TOML allowlist for justified
  exceptions. Each entry needs a daemon + route + justification
  (issue link expected). Seeded with the 4 current legacy cases:
  secubox-waf (covered by PR #146 — will drop the entries when
  merged), secubox-config-advisor /history (admin drill-in, not
  polled), secubox-netdiag /connections (real-time TCP state),
  secubox-tor /summary (small file read, follow-up cleanup).

- .github/workflows/dashboard-cache-check.yml — mirror of
  license-check.yml. Runs the lint + the self-tests on PRs that
  touch packages/**/api/main.py, the lint script, the allowlist,
  or the workflow itself.

- docs/CACHE-PATTERN.md — short remediation guide. Three accepted
  shapes (read-through, background-refresh, pure in-memory), code
  example, allowlist syntax, references to the canonical
  implementations (secubox-crowdsec for read-through, secubox-system
  for background-refresh).

Tests (14/14 pass)
==================
- StatsCache pattern detection (compliant cases)
- Non-compliant detection: open, subprocess.run, multiple I/O ops
- Pure in-memory handler = compliant
- Background-refresh task = compliant
- Non-hot route (/health) not audited
- @router.get treated like @app.get
- Allowlist suppresses by exact (daemon, route) key
- Allowlist doesn't cross-pollinate across routes
- Missing allowlist file = no error
- CLI --check exits 1 on unjustified
- CLI --check exits 0 when clean
- CLI --json output well-formed
- **Real repo audit with seeded allowlist passes --check**
  (sanity gate: catches regression if anyone removes an allowlist
  entry without fixing the daemon)

Baseline against the real packages/ tree
========================================
122 daemons audited, 118 compliant, 4 with findings (all
allowlisted). When PR #146 merges, the secubox-waf entries can be
removed from the allowlist — the next lint run will then prove the
fix is wired correctly.

Follow-up
=========
- Drop secubox-waf allowlist entries when #146 merges.
- Per-daemon follow-up issues for the remaining 3 (config-advisor,
  netdiag, tor) — not blocking, allowlist documents the rationale.
- Consider extending HOT_ROUTES to /health / /status once the small
  daemons that call `systemctl is-active` adopt a 10 s cache.

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 08:21:20 +02:00
531fd878a6 fix(sync): drop 10.100.0.10 from DEAD_CONTAINER_IPS — mail LXC lives there
The mail Phase 1 LXC went live at 10.100.0.10 (see #136, PR #141,
35ba4c2c). The 'dead container fix' was still rewriting routes that
target 10.100.0.10 to 10.100.0.1:9080, which would have clobbered
mail.gk2 / webmail.gk2 / rspamd.gk2 / autoconfig.gk2 routes. The
production gk2 board already had this correction in its local copy
(diffed 2026-05-17); bringing the repo back in sync so deploying
the script doesn't regress.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 10:42:07 +02:00
4a73cc245c fix(sync): mitmproxy-routes regex anchored at \s|$ instead of space-only
The previous regex `host_\K[a-z0-9_]+(?= )` required a trailing SPACE
after the hostname. HAProxy `use_backend mitmproxy_inspector if
host_<dotted_domain>` lines end at the hostname with no trailing space,
so the regex matched ZERO domains. Sync silently never auto-populated
mitmproxy routes for any haproxyctl-added vhost.

Caught 2026-05-17 when ckwa.gk2.secubox.in returned 502: the vhost was
added via `haproxyctl vhost add ckwa.gk2.secubox.in mitmproxy_inspector
ssl` (HAProxy ACL created correctly), but sync didn't write the
mitmproxy route entry, so the request landed at mitmproxy's default
(127.0.0.1:9080 placeholder) which doesn't exist inside the mitmproxy
LXC.

Fix: change lookahead to `(?=\s|$)` to also match end-of-line. Verified:

  $ printf '    use_backend ... if host_x_y_z\n' | \
    grep -oP 'host_\K[a-z0-9_]+(?=\s|$)' | sed 's/_/./g'
  x.y.z

Live patch applied on gk2 (mitmproxy LXC) to restore ckwa to 200.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 10:39:44 +02:00
d54b711fda feat(metablog-webhook): install/uninstall scripts via Gitea API (ref #113)
- metablog-webhook-install.sh: pages through GITEA_URL/users/<owner>/repos,
  filters to metablog-*, checks existing hooks for the target URL,
  POSTs a new hook with content_type=json + secret + events=[push].
  Idempotent.
- metablog-webhook-uninstall.sh: reverse, DELETE on matching hook IDs.
- Both support --dry-run.

Exit non-zero if any operation failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 07:31:20 +02:00
426b0b1fea feat(metablog): site.json backfill orchestrator (ref #101)
For each site under /srv/metablogizer/sites/:
- missing site.json -> create complete (name, domain, published, version, streamlit_app)
- present + --force -> merge missing fields, preserve existing keys/values
- present, no --force -> skip

streamlit_app auto-detected via Gitea ls-remote probe (5s timeout).
version from git describe --tags, else v1.0.0.
JSON report at output/metablog-backfill-report.json.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 17:25:59 +02:00
2f440c6194 Merge remote-tracking branch 'origin/master' into feature/95-streamlit-per-site-version-pinning-conta
# Conflicts:
#	.claude/HISTORY.md
#	.claude/WIP.md
#	.gitignore
2026-05-12 16:59:00 +02:00
e9192d3d32 fix(streamlit): Also fix _saved_source_args empty-array expansion (ref #95)
Same bug pattern as 34a4760e but on the source-helpers line.
Both empty-array restores now use "${var[@]}" without :-.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
9e614bd0f7 fix(streamlit): saved_args empty-array expansion (ref #95)
set -- "${saved_args[@]:-}" produces a single empty positional
argument when the array is empty, which the case-statement then
sees as Unknown flag "". Use "${saved_args[@]}" (no :-) which
gives zero positional args correctly on empty.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
343ebf8628 fix(streamlit): Also exclude __pycache__ from ingest scan (ref #95)
After excluding dot-dirs, __pycache__ still appeared as an app
candidate. Add explicit !-name '__pycache__' to the find filter.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
8d7a24d233 fix(streamlit): Exclude dot-directories from ingest scan (ref #95)
/srv/streamlit/apps/.claude exists (Claude tracking artifacts:
HISTORY.md, TODO.md, WIP.md) and was being picked up as an app
candidate. Filter out dot-prefixed directory names so only real
Streamlit app directories are considered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
f8cd25bb26 feat(streamlit): Orchestrator for Gitea ingest (preflights + report) (ref #95)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
2ecd96e688 feat(metablog): SSH preflight + Gitea key enrolment helper (ref #94)
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).

Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
c85d0d82a0 feat(streamlit): Per-app ingest function (idempotent, history-preserving) (ref #95)
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user 'gitea', not 'git'). Mirrors scripts/lib/metablog-ingest-site.sh
from sub-project B (PR #97) with the streamlit- prefix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
a0e9ba8beb test(metablog): Smoke test for 3-site Gitea ingest (ref #94)
fix(metablog-ingest-site): Handle .git repos with unborn branch (no commits):
- Use git rev-parse --verify HEAD (exits non-zero on unborn branch) instead of
  rev-parse HEAD (which returns "HEAD" string even with no commits)
- Add fall-through path: .git exists + no commits → git symbolic-ref HEAD
  refs/heads/main + commit + push (same as fresh-init path)

test: Add tests/scripts/test-metablog-ingest.sh smoke test:
- Step 1: dry-run --limit 3 verifies report total=3
- Step 2: live --limit 3 accepts any ok+skip combo >= 3
- Step 3: idempotent re-run asserts all 3 skip-already-current
- Step 4: git ls-remote via MOCHAbin SSH verifies remote HEAD exists
- Step 5: clone+diff via MOCHAbin SSH (diff --exclude=.git) verifies
  clone == source

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
557d85b27e feat(metablog): Orchestrator for Gitea ingest (preflights + report) (ref #94)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
2900e25da9 feat(metablog): Per-site ingest function (idempotent, history-preserving) (ref #94)
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user it runs as, which is 'gitea' in this LXC, not 'git').

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
a38011fca6 feat(metablog): SSH preflight + Gitea key enrolment helper (ref #94)
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).

Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
08c998512f fix(metablog): awk INI patcher no longer duplicates [repository] section (ref #94)
The section-header rule flushed missing keys but did not set
saw_push/saw_branch to 1, so the END block then appended a second
[repository] block at EOF. Set the flags inline with the flush.

Also cleaned the live MOCHAbin app.ini which had been doubly
patched by the original buggy version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
ba972d1a8d feat(metablog): One-time Gitea config patch for push-create (ref #94)
- app.ini at /var/lib/gitea/custom/conf/app.ini (not /etc/gitea/)
- awk-based INI patcher (python3 not installed in gitea LXC)
- ENABLE_PUSH_CREATE_USER=true, DEFAULT_BRANCH=main applied
- Gitea restarted and confirmed active

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
CyberMind
b0b6e78cab
Merge pull request #84 from CyberMind-FR/feature/license-headers-phase-a
feat(license): CMSD-1.0 header tool, CI, and conventions (Phase A)
2026-05-12 11:41:43 +02:00
b88b8ada95 fix(license): tighten detect_existing + missing-file = repo-wide (ref #81)
Two bug fixes to the CMSD-1.0 license-header tool:

1. detect_existing() previously matched the SPDX token anywhere in the
   first 10 lines, including inside docstrings and prose comments.
   Tightened the matcher to require comment markers (#, //, *, <!--)
   and whitespace before the SPDX token.

   Regression tests:
   - test_detect_existing_no_false_match_in_docstring
   - test_detect_existing_no_false_match_inline_comment_prose

2. _read_enrollment() now returns ["**"] when the allowlist file is
   absent, per spec §5.2 "missing file → repo-wide enforcement".
   Previously it returned [] (nothing enforced), making Phase C closure
   impossible without an additional file.

   Empty allowlist file still returns [] (Phase A initial), so
   test_main_empty_allowlist_passes_check is unaffected.

   New tests:
   - test_read_enrollment_missing_file_means_repo_wide
   - test_main_check_missing_allowlist_enforces_repo_wide

Suite: 49 → 53 passing. --check still exits 0 repo-wide.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:39:57 +02:00
ba9f6b20c1 fix(scripts): re-apply secubox-haproxy-regen-safe sub-command fix
A merge from feature/83 reverted the fix from commit 22014865, which had
already replaced `haproxyctl regen` (non-existent sub-command, printed
help text and exited 0) with the correct `haproxyctl generate`.

This commit re-applies the same fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:17:34 +02:00
2763009fad fix(license): walk() accepts repo_root for subdir invocations (ref #81)
Discovered during Phase B pilot: running `--fix packages/secubox-hub`
yielded zero files because walk() computed `rel` relative to the walk
root (the subdir) while allowlist patterns are repo-relative.

Add optional `repo_root` parameter to walk(); when present, allowlist
matching uses paths relative to it instead of the walk root.
Backwards-compatible (defaults to None, preserving existing tests).
main() passes the discovered repo_root through.

Regression test added: test_walk_subdir_with_repo_root_arg.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:03:32 +02:00
e31c1fa99b fix(sync-routes): preserve streamlit & custom routes from clobbering
After Session 156 fix, secubox-haproxy.service was safely restarted but
cpf.gk2.secubox.in (and other streamlit instances) immediately fell back
to "Wrong Domain" because sync-mitmproxy-routes.sh was clobbering their
container routes.

Two compounding bugs:
1. DEAD_CONTAINER_IPS included 10.100.0.50 — but that's the streamlit
   LXC's actual IP (RUNNING). fix_dead_container_routes() rewrote every
   route pointing to 10.100.0.50 (cpf:8523, etc.) to 10.100.0.1:9080.
2. The update conditional reapplied routes when the existing port didn't
   match the computed port. The computed port logic only knows metablog
   vs webui — it has no streamlit awareness, so it always tried to force
   streamlit domains to port 9080 (webui default).

Fixes:
- Drop 10.100.0.50 from DEAD_CONTAINER_IPS. The streamlit LXC at that
  address is alive; routes pointing to it are valid.
- Change "update if existing missing OR port mismatch" to "update only
  if existing is missing." The script now only ADDS routes for ACL/
  metablog domains; it never overwrites existing routes set by other
  sources (sync-all-routes.sh, manual edits, etc.). fix_dead_container_
  routes() still handles truly dead IPs separately.

Verified: cpf.gk2.secubox.in → HTTP 200 "Streamlit" persists across
sync-mitmproxy-routes.service runs. arm/admin still 200 with correct
content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:02:42 +02:00
c911e8192f fix(haproxy): generator emits mitmproxy_inspector + safe regen wrapper
Triggered by cpf.gk2.secubox.in returning "Wrong Domain". Root cause: the
secubox-haproxy generators (both bash haproxyctl and Python FastAPI) emit
config rules that point to dead/wrong backends, leading to HTTPS-wide
metablog/streamlit outages when a regen runs.

Generator fixes:
- packages/secubox-haproxy/api/main.py: use_backend waf_inspector
  → use_backend mitmproxy_inspector (2 occurrences). waf_inspector
  pointed to 127.0.0.1:8890 which is not listening; mitmproxy_inspector
  is the actual healthy WAF backend at 10.100.0.60:8080.
- packages/secubox-haproxy/sbin/haproxyctl: default_backend fallback
  → default_backend mitmproxy_inspector (2 occurrences). The fallback
  backend deny_status 503's unknown hosts; switching to mitmproxy_inspector
  lets mitmproxy dispatch all hosts via its routes JSON (245 routes vs
  93 declared in haproxy.toml).

New safe regen wrapper:
- scripts/secubox-haproxy-regen-safe: snapshot → regen via haproxyctl →
  validate via haproxy -c -f → atomic swap → reload. On validation
  failure, current cfg is preserved and the broken candidate is saved
  for forensics. Prevents another silent breakage like Session 156.

Board-side state (not in repo): /etc/haproxy/haproxy.cfg patched in
place to mirror these fixes, secubox-haproxy.service stopped until
user confirms safe to re-enable. All 245 routes verified live (cpf,
arm, lldh, admin, pub, werdl, 3d, 42, zkp all HTTP 200 with correct
content).

See .claude/HISTORY.md Session 156 for full symptom chain and the
6-layer root cause analysis.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:52:44 +02:00
c661c4f053 fix(scripts): use issue title for PR, surface push errors as exit 4 (ref #83)
Addresses code review:
- Important #4: finish now fetches issue title via `gh issue view --json title`
  and uses it as PR title (was using the branch name). Falls back to branch
  name if the fetch fails or returns empty.
- Minor #8: explicit `return 4` on `git push` failure for consistency with
  the documented exit-code contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:22:14 +02:00
b77a49fc2f docs(scripts): document agent-worktree in scripts/README (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:05:48 +02:00
4ff4830df4 feat(scripts): implement agent-worktree clean sub-command (ref #83)
Also patches gh-mock to sanitize dashes in branch-derived env var names,
and fixes --head two-arg parsing in the pr view mock handler.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:03:47 +02:00
9eb75676d5 docs(license): document license-headers.py in scripts/README (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:01:46 +02:00
4d937995ba ci(license): add License Headers workflow (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:54 +02:00
eba35ff5b5 feat(scripts): implement agent-worktree finish sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:24 +02:00
e90a07a578 chore(license): self-host header on tool and tests + initial allowlist (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:09 +02:00
4554cb991e feat(scripts): implement agent-worktree sync sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:59:01 +02:00
c93cfdb639 feat(license): CLI dispatch with --check/--fix/--list/--diff (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:57:19 +02:00