Adds scripts/audit-packages.py to scan packages/secubox-*/ and emit
structured artifacts under out/ (gitignored): packages.json,
packages.dot dependency graph, empty-packages.txt, prefix-based
clusters, and a hand-curated fuzzy-cluster report with per-member
evidence (deps, routes, services).
Findings doc 2026-05-27-secubox-consolidation-audit.md synthesises the
data into Tier 0-4 merge proposals grounded in real package contents,
not the plan stub's illustrative guesses. Key corrections vs the stub:
- 141 real packages (not "100+"); secubox-core is the substrate for 131
- streamlit-idle is already an in-package service variant, not a
separate package — that consolidation is half-done
- meta{blogizer,bolizer,catalog,oblizer} are real distinct services,
not typo proliferation (recommend rename for clarity, not merge)
- Two genuine packaging bugs surfaced: secubox-c3box binary name
collision between Python dashboard and Go daemon (issue #378),
secubox-smart-strip has no debian/ directory at all
Also imports two pre-existing 2026-05-26 plan stubs (consolidation,
build-scripts-refactor) that had been sitting untracked.
Re-run after any packaging change: python3 scripts/audit-packages.py
The IS31FL319X chip on the MOCHAbin's i2c-1 bus errors out with -EIO
above brightness ~50. Pushing 100 or 255 saturates the bus, the chip
starts NACK'ing the next register writes, the LED driver logs
'Setting an LED's brightness failed (-5)' on every retry, and the
LEDs freeze at whatever value last got through.
Observed live 2026-05-24 on gk2: all three RGBs stuck plain green at
218/255/255 with the bash bumper and the python heartbeat both pushing
high values into a saturated chip. After dropping to brightness 10
and adding a 30 ms i2c-write delay in the python set_rgb() loop,
writes succeed and the LEDs reflect the actual health state instead
of a static color.
Files:
- usr/sbin/secubox-led BRIGHTNESS 100 -> 10
- usr/sbin/secubox-healthbump ACTIVE 100 -> 30, SLEEP 20 -> 5
- usr/sbin/secubox-led-pulse BRIGHTNESS 100 -> 10
- usr/sbin/secubox-led-heartbeat python set_rgb scales any 0-255
caller down to 0-10 in write_text, then sleeps 30 ms before the
next channel — matches the bash scripts' I2C_DELAY and lets the
mv64xxx_i2c controller drain between calls.
- scripts/secubox-healthbump (the duplicate script kept in repo root
for ad-hoc dev use) gets the same BRIGHTNESS=10 cap.
The misleading 'BRIGHTNESS=100 works reliably' comment that the code
contradicted is replaced with an incident note pointing at the EIO
ceiling.
* docs(mochabin): add EP06 mPCIe GPIO investigation runbook (ref #345)
scripts/probe-mpcie-gpios.sh: empirical CP0 GPIO sweep that drives
each unrequested line HIGH for a few seconds and watches dmesg/lsusb
for new USB device. Skips [used] lines, uses libgpiod gpioset
--mode=time so the line auto-restores to input on release. Modes:
--baseline (snapshot only, safe), --line gpiochipN K (single line),
no arg (full sweep gpiochip1+gpiochip2).
docs/hardware/mochabin-mpcie-ep06-runbook.md: hypothesis, procedure,
and DTS rfkill-gpio template patterned after cn9132-clearfog.dts
lines 69-122. Apply once a candidate GPIO is identified.
--baseline smoke-tested on gk2 — sweep deferred until spare EP06
arrives (no mPCIe device in J5 = no signal regardless of which line
drives W_DISABLE#).
* docs(mochabin): drop EP06 spare assumption from runbook (ref #345)
No spare EP06 on order; the existing EP06-E will be used for the
sweep when re-seated in J5. Replaces "deferred until spare" framing
with a multimeter-scoping fallback for the inconclusive case.
* fix(mpcie): v0.2.0 probe-mpcie-gpios.sh — drop blanket sweep, danger-list, dry-run default (ref #345)
The v0.1.0 "drive every unused-input CP0 GPIO HIGH for 3s one-at-a-time"
sweep took gk2 hard-down within seconds (host became unreachable; ARP
stopped responding). gpioinfo's [used] tag only reflects lines the
kernel *requested* — several unrequested CP0 GPIOs are physically
wired to critical board functions (eth switch reset, PCIe2 PERST#,
pca9554 IRQ) and driving them HIGH wedged the board.
v0.2.0:
- No blanket-sweep mode. Removed.
- Default mode = dry-run candidate listing only. No GPIO is driven.
- --commit --line CHIP N required to actually drive a single line.
- Hardcoded DANGER_LINES (gpiochip1:1, gpiochip1:9, gpiochip1:27)
skipped unconditionally, even with --commit.
- gpiochip2 entirely held off without --allow-chip2 (no per-line
schematic mapping for that bank).
- Hard refusals are vocal (REFUSED: ... reason) before exit.
Runbook updated to walk through the new dry-run → --commit flow.
HISTORY documents the incident + the v0.2.0 mitigations.
Lesson saved in memory feedback_gpio_blanket_sweep_crashed_board.md.
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Symptom: pre-login banner on ttyS0 (Minicom 2.9 VT102) rendered as
trailing `�` after every emoji line. The console driver couldn't
decode the UTF-8 multi-byte sequences for ⚡🔐🌐📡 and emitted the
replacement character instead.
The `█` box-block characters (U+2588) DO render correctly in VT102+
UTF-8 — they're a single-byte-class glyph in most modern fonts. We
keep those so the SECUBOX wordmart still looks like SecuBox.
Replacements (compatible with VT102 minicom AND SSH UTF-8 alike):
⚡ -> >>
🔐 -> [user]
🌐 -> [web]
📡 -> [ssh]
The /etc/motd block (shown after SSH login) is left untouched — SSH
sessions have proper UTF-8 terminals so the rich emojis are fine
there.
Also add scripts/build-kernel-local.sh (new): the local kernel build
script that I authored today for #255. Documented; runs the same
defconfig + merge_config + scripts/config + olddefconfig + sed ZRAM
override + bindeb-pkg flow as .github/workflows/build-kernel.yml,
without the 20-min GitHub Actions ping-pong.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Anti-regression for the double-cache pattern documented in CLAUDE.md
(« Performance Patterns — Double Caching »). Catches the bug class that
caused PR #146 (secubox-waf 17 % sustained CPU under SOC polling)
before it lands.
What's added
============
- scripts/check-dashboard-cache.py — AST-based audit. Walks every
packages/secubox-*/api/main.py, flags hot dashboard routes
(/stats, /alerts, /summary, /dashboard, /metrics, /overview,
/events, /history, /recent, /bans, /sessions, /connections)
whose handler body does per-request I/O (open, read_text,
read_bytes, subprocess.run, Popen) AND doesn't go through a cache
(module-level _cache instance + .get() call, OR
asyncio.create_task(refresh_*) at startup).
* --check : exit 1 on unjustified violations (CI mode)
* --report : human-readable table (default)
* --json : machine-readable
- .cache-lint-allowlist.toml — TOML allowlist for justified
exceptions. Each entry needs a daemon + route + justification
(issue link expected). Seeded with the 4 current legacy cases:
secubox-waf (covered by PR #146 — will drop the entries when
merged), secubox-config-advisor /history (admin drill-in, not
polled), secubox-netdiag /connections (real-time TCP state),
secubox-tor /summary (small file read, follow-up cleanup).
- .github/workflows/dashboard-cache-check.yml — mirror of
license-check.yml. Runs the lint + the self-tests on PRs that
touch packages/**/api/main.py, the lint script, the allowlist,
or the workflow itself.
- docs/CACHE-PATTERN.md — short remediation guide. Three accepted
shapes (read-through, background-refresh, pure in-memory), code
example, allowlist syntax, references to the canonical
implementations (secubox-crowdsec for read-through, secubox-system
for background-refresh).
Tests (14/14 pass)
==================
- StatsCache pattern detection (compliant cases)
- Non-compliant detection: open, subprocess.run, multiple I/O ops
- Pure in-memory handler = compliant
- Background-refresh task = compliant
- Non-hot route (/health) not audited
- @router.get treated like @app.get
- Allowlist suppresses by exact (daemon, route) key
- Allowlist doesn't cross-pollinate across routes
- Missing allowlist file = no error
- CLI --check exits 1 on unjustified
- CLI --check exits 0 when clean
- CLI --json output well-formed
- **Real repo audit with seeded allowlist passes --check**
(sanity gate: catches regression if anyone removes an allowlist
entry without fixing the daemon)
Baseline against the real packages/ tree
========================================
122 daemons audited, 118 compliant, 4 with findings (all
allowlisted). When PR #146 merges, the secubox-waf entries can be
removed from the allowlist — the next lint run will then prove the
fix is wired correctly.
Follow-up
=========
- Drop secubox-waf allowlist entries when #146 merges.
- Per-daemon follow-up issues for the remaining 3 (config-advisor,
netdiag, tor) — not blocking, allowlist documents the rationale.
- Consider extending HOT_ROUTES to /health / /status once the small
daemons that call `systemctl is-active` adopt a 10 s cache.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The mail Phase 1 LXC went live at 10.100.0.10 (see #136, PR #141,
35ba4c2c). The 'dead container fix' was still rewriting routes that
target 10.100.0.10 to 10.100.0.1:9080, which would have clobbered
mail.gk2 / webmail.gk2 / rspamd.gk2 / autoconfig.gk2 routes. The
production gk2 board already had this correction in its local copy
(diffed 2026-05-17); bringing the repo back in sync so deploying
the script doesn't regress.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous regex `host_\K[a-z0-9_]+(?= )` required a trailing SPACE
after the hostname. HAProxy `use_backend mitmproxy_inspector if
host_<dotted_domain>` lines end at the hostname with no trailing space,
so the regex matched ZERO domains. Sync silently never auto-populated
mitmproxy routes for any haproxyctl-added vhost.
Caught 2026-05-17 when ckwa.gk2.secubox.in returned 502: the vhost was
added via `haproxyctl vhost add ckwa.gk2.secubox.in mitmproxy_inspector
ssl` (HAProxy ACL created correctly), but sync didn't write the
mitmproxy route entry, so the request landed at mitmproxy's default
(127.0.0.1:9080 placeholder) which doesn't exist inside the mitmproxy
LXC.
Fix: change lookahead to `(?=\s|$)` to also match end-of-line. Verified:
$ printf ' use_backend ... if host_x_y_z\n' | \
grep -oP 'host_\K[a-z0-9_]+(?=\s|$)' | sed 's/_/./g'
x.y.z
Live patch applied on gk2 (mitmproxy LXC) to restore ckwa to 200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- metablog-webhook-install.sh: pages through GITEA_URL/users/<owner>/repos,
filters to metablog-*, checks existing hooks for the target URL,
POSTs a new hook with content_type=json + secret + events=[push].
Idempotent.
- metablog-webhook-uninstall.sh: reverse, DELETE on matching hook IDs.
- Both support --dry-run.
Exit non-zero if any operation failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same bug pattern as 34a4760e but on the source-helpers line.
Both empty-array restores now use "${var[@]}" without :-.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
set -- "${saved_args[@]:-}" produces a single empty positional
argument when the array is empty, which the case-statement then
sees as Unknown flag "". Use "${saved_args[@]}" (no :-) which
gives zero positional args correctly on empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After excluding dot-dirs, __pycache__ still appeared as an app
candidate. Add explicit !-name '__pycache__' to the find filter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/srv/streamlit/apps/.claude exists (Claude tracking artifacts:
HISTORY.md, TODO.md, WIP.md) and was being picked up as an app
candidate. Filter out dot-prefixed directory names so only real
Streamlit app directories are considered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).
Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user 'gitea', not 'git'). Mirrors scripts/lib/metablog-ingest-site.sh
from sub-project B (PR #97) with the streamlit- prefix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user it runs as, which is 'gitea' in this LXC, not 'git').
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).
Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The section-header rule flushed missing keys but did not set
saw_push/saw_branch to 1, so the END block then appended a second
[repository] block at EOF. Set the flags inline with the flush.
Also cleaned the live MOCHAbin app.ini which had been doubly
patched by the original buggy version.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- app.ini at /var/lib/gitea/custom/conf/app.ini (not /etc/gitea/)
- awk-based INI patcher (python3 not installed in gitea LXC)
- ENABLE_PUSH_CREATE_USER=true, DEFAULT_BRANCH=main applied
- Gitea restarted and confirmed active
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two bug fixes to the CMSD-1.0 license-header tool:
1. detect_existing() previously matched the SPDX token anywhere in the
first 10 lines, including inside docstrings and prose comments.
Tightened the matcher to require comment markers (#, //, *, <!--)
and whitespace before the SPDX token.
Regression tests:
- test_detect_existing_no_false_match_in_docstring
- test_detect_existing_no_false_match_inline_comment_prose
2. _read_enrollment() now returns ["**"] when the allowlist file is
absent, per spec §5.2 "missing file → repo-wide enforcement".
Previously it returned [] (nothing enforced), making Phase C closure
impossible without an additional file.
Empty allowlist file still returns [] (Phase A initial), so
test_main_empty_allowlist_passes_check is unaffected.
New tests:
- test_read_enrollment_missing_file_means_repo_wide
- test_main_check_missing_allowlist_enforces_repo_wide
Suite: 49 → 53 passing. --check still exits 0 repo-wide.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A merge from feature/83 reverted the fix from commit 22014865, which had
already replaced `haproxyctl regen` (non-existent sub-command, printed
help text and exited 0) with the correct `haproxyctl generate`.
This commit re-applies the same fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discovered during Phase B pilot: running `--fix packages/secubox-hub`
yielded zero files because walk() computed `rel` relative to the walk
root (the subdir) while allowlist patterns are repo-relative.
Add optional `repo_root` parameter to walk(); when present, allowlist
matching uses paths relative to it instead of the walk root.
Backwards-compatible (defaults to None, preserving existing tests).
main() passes the discovered repo_root through.
Regression test added: test_walk_subdir_with_repo_root_arg.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After Session 156 fix, secubox-haproxy.service was safely restarted but
cpf.gk2.secubox.in (and other streamlit instances) immediately fell back
to "Wrong Domain" because sync-mitmproxy-routes.sh was clobbering their
container routes.
Two compounding bugs:
1. DEAD_CONTAINER_IPS included 10.100.0.50 — but that's the streamlit
LXC's actual IP (RUNNING). fix_dead_container_routes() rewrote every
route pointing to 10.100.0.50 (cpf:8523, etc.) to 10.100.0.1:9080.
2. The update conditional reapplied routes when the existing port didn't
match the computed port. The computed port logic only knows metablog
vs webui — it has no streamlit awareness, so it always tried to force
streamlit domains to port 9080 (webui default).
Fixes:
- Drop 10.100.0.50 from DEAD_CONTAINER_IPS. The streamlit LXC at that
address is alive; routes pointing to it are valid.
- Change "update if existing missing OR port mismatch" to "update only
if existing is missing." The script now only ADDS routes for ACL/
metablog domains; it never overwrites existing routes set by other
sources (sync-all-routes.sh, manual edits, etc.). fix_dead_container_
routes() still handles truly dead IPs separately.
Verified: cpf.gk2.secubox.in → HTTP 200 "Streamlit" persists across
sync-mitmproxy-routes.service runs. arm/admin still 200 with correct
content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Triggered by cpf.gk2.secubox.in returning "Wrong Domain". Root cause: the
secubox-haproxy generators (both bash haproxyctl and Python FastAPI) emit
config rules that point to dead/wrong backends, leading to HTTPS-wide
metablog/streamlit outages when a regen runs.
Generator fixes:
- packages/secubox-haproxy/api/main.py: use_backend waf_inspector
→ use_backend mitmproxy_inspector (2 occurrences). waf_inspector
pointed to 127.0.0.1:8890 which is not listening; mitmproxy_inspector
is the actual healthy WAF backend at 10.100.0.60:8080.
- packages/secubox-haproxy/sbin/haproxyctl: default_backend fallback
→ default_backend mitmproxy_inspector (2 occurrences). The fallback
backend deny_status 503's unknown hosts; switching to mitmproxy_inspector
lets mitmproxy dispatch all hosts via its routes JSON (245 routes vs
93 declared in haproxy.toml).
New safe regen wrapper:
- scripts/secubox-haproxy-regen-safe: snapshot → regen via haproxyctl →
validate via haproxy -c -f → atomic swap → reload. On validation
failure, current cfg is preserved and the broken candidate is saved
for forensics. Prevents another silent breakage like Session 156.
Board-side state (not in repo): /etc/haproxy/haproxy.cfg patched in
place to mirror these fixes, secubox-haproxy.service stopped until
user confirms safe to re-enable. All 245 routes verified live (cpf,
arm, lldh, admin, pub, werdl, 3d, 42, zkp all HTTP 200 with correct
content).
See .claude/HISTORY.md Session 156 for full symptom chain and the
6-layer root cause analysis.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses code review:
- Important #4: finish now fetches issue title via `gh issue view --json title`
and uses it as PR title (was using the branch name). Falls back to branch
name if the fetch fails or returns empty.
- Minor #8: explicit `return 4` on `git push` failure for consistency with
the documented exit-code contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Also patches gh-mock to sanitize dashes in branch-derived env var names,
and fixes --head two-arg parsing in the pr view mock handler.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: log() in sync-mitmproxy-routes.sh wrote to stdout, polluting
$(fix_dead_container_routes ...) capture → corrupted JSON → jq parse
errors → set -e abort → bad routes pushed to mitmproxy container →
mitmproxy restart-loop → HAProxy backend DOWN → HTTP 503 on every
*.gk2.secubox.in metablog vhost (arm, zkp, 3d, ~160 sites).
Fixes:
- log() now writes to stderr (>&2) — preserves stdout for command-sub
- fix_dead_container_routes returns 0 (was returning $fixed count, set -e
killed the caller's assignment)
- sync-all-routes.sh routes metablog domains to port 8900 (was 9080,
which is webui.conf default_server → "Wrong Domain")
- defensive 2>/dev/null || true on jq reads
- fallback to old routes_json on jq write failure (preserves last valid
state instead of crashing)
- flock guard prevents concurrent runs on /run/sync-mitmproxy-routes.lock
Verified: systemctl start sync-mitmproxy-routes.service exits 0/SUCCESS,
244 routes valid in mitmproxy container, all four spot-checked domains
(admin.gk2, arm.gk2, zkp.gk2, 3d.gk2) return HTTP 200 with correct titles.
Note: also disabled duplicate timer secubox-route-sync.timer on the
board (executed the same script, racing on same file). systemd-level
change is on board only, not in repo.
See .claude/HISTORY.md Session 153 for full symptom chain and verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
