secubox-core.service is a Type=oneshot (mkdir+chown) that RemainAfterExit=yes. A hard
Requires= on ~108 units cascade-stops them all if core is restarted/fails (e.g. a
secubox-core package upgrade) — a thundering-herd outage. After= keeps the ordering;
Wants= keeps the soft dependency without the cascade. Prereq for mass native apply
(Phase 3). Scaffolds (new-module.sh/new-package.sh) updated so future units use Wants=.
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
Live gk2 findings: haproxyctl vhost-add is blocked by its drift-guard and is
redundant with default_backend mitmproxy_inspector, so route_ok now reflects the
WAF route (the operative mechanism), vhost stays advisory. And the board runs
Python 3.11.2 (no tarfile filter= kwarg) — replaced filter=data with a portable
manual member check (rejects traversal/absolute/links).
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
secubox-publish's infra provisioner ran as unprivileged `secubox` and tried
to write /etc/nginx/sites-available/<domain>.conf, rewrite
/etc/haproxy/haproxy.cfg, write /etc/secubox/waf/haproxy-routes.json, and
reload nginx/haproxy/secubox-waf directly. All of these silently failed
(permission denied), which was the root cause of published sites answering
421/never routing. Metablogizer already owns routing through the audited
secubox-publishctl root helper (apply_route + provision_cert), so add a
POST /publish/route endpoint there and have the hub call it instead of
touching root config itself.
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
Adds the Task-7 5-step Publish wizard (Content -> Version -> Route ->
Cert -> Backup) to www/metablogizer/index.html, wired to the Task-6
endpoints (POST /publish/wizard multipart, GET /publish/export/{name}).
Reuses the page's existing token()/API helpers and the
uploadSiteContent() multipart-fetch/401-retry pattern for consistency;
JSON result is rendered via textContent only.
Co-Authored-By: Gerald KERMA <devel@cybermind.fr>
Add /shared/hybrid-dark.css: a shared overlay that remaps the design tokens
(consumed by crt-light.css, sidebar-light.css and every module's inline styles)
to the cyan hybrid-dark palette from /certs/ + /wireguard/. body.hybrid-dark
(specificity 0,1,1) beats :root and inline :root, so one token remap flips the
whole tree — no per-panel rewrite. Each panel gains the overlay <link> + the
hybrid-dark body class (2-line additive swap; crt-light.css kept as base).
See .claude/WEBUI-PANEL-GUIDELINES.md.
Panels with fully bespoke styles (certs/wireguard/users/nac already hybrid;
~14 non-crt-light custom panels) are untouched.
- git_commit_push(): stage-all + commit + push origin HEAD, best-effort
(no repo / no changes / push-rejected never raise; commit kept local)
- /site/{name}/upload wires it under the per-site lock, off the event loop
- postinst rewrites the push URL to Gitea's internal LXC endpoint for the
service user (public host NAT-hairpins → pack transfer stalls); seeds
known_hosts. Verified live on gk2: upload → commit → push = ok
The upload endpoint unconditionally created and wrote into <site>/public/, but
a site's served root follows the convention 'public/ if it exists, else the
site dir root' (same as regenerate_nginx_config/publish). For a site published
from its dir root (e.g. 'status', root .../status), writing to public/ meant
nginx never served the uploaded content — the target site appeared unchanged.
Now target = public/ only when it already exists, else the site dir root, and
we no longer force-create public/ (which would have silently moved the served
root). Response includes the resolved target for clarity.
Adds two per-row actions in the sites list:
- 🔄 deploySite → POST /site/{name}/deploy : manual git pull + redeploy, the same
operation as the Gitea webhook (shared per-site lock, records a deploy entry).
- ⬆ uploadSiteContent → POST /site/{name}/upload : upgrade content from an
uploaded archive; upload now also accepts a bare .html (written as index.html)
in addition to .zip / .tar.gz.
Both reuse the existing auth (require_jwt) with the Bearer→cookie retry pattern.
Deployed + served on gk2.
Previous fix removeItem'd sbx_token/secubox_token on any 401, which destroyed
the auth the shared sidebar + sibling components rely on, cascading the whole
page to /login even for a transient/single 401. Now retries cookie-only
(credentials:same-origin) WITHOUT deleting the token, then redirects only if
that also 401s.
Root cause of the login bounce that other modules didn't have: metablogizer's
token()/headers() read ONLY sbx_token, but the fleet (hub, portal) reads
sbx_token||secubox_token — a valid token stored under secubox_token was invisible
to metablogizer, so it sent no Bearer and 401'd. Now reads both keys. Also: the
401 cookie-retry no longer gates on token() still being set (parallel dashboard
calls raced — first cleared the token, others bounced), and clears both keys.
require_jwt prefers the Bearer token when present (else the secubox_session
cookie). A stale sbx_token in localStorage was sent as Bearer, shadowing the
user's VALID session cookie → 401 → bounce to login → (with the return-redirect)
an endless login↔metablogizer loop, even though the user was authenticated.
Proven live: session-cookie-only=200, but stale-bearer+valid-cookie=401.
Fix: on 401 with a token, drop the stale token and retry cookie-only (SSO
fallback) before ever redirecting to login. Self-healing; no loop.
login.html hardcoded window.location='/' after auth, ignoring where the user
came from — so any module that 401-bounced to login landed on the hub root
instead of coming back. Add _sbxRedirect(): honor a same-origin ?redirect=
path (open-redirect-safe, falls back to '/'). metablogizer now sends
?redirect=<pathname> on 401 (index.html + site.html), so /metablogizer/ ->
login -> back to the /metablogizer/ dashboard. Fixes the 'ko' auth/dashboard
switch; benefits every module that already sends ?redirect=.
* perf(aggregator): async-sweep — 243 blocking route handlers async def→def (#738)
Mounted in the aggregator's single event loop, an 'async def' route handler
that runs blocking code (subprocess/journalctl/openssl/argon2) freezes the
WHOLE loop -> aggregator.sock Connection refused -> 502 board-wide.
Deterministic AST codemod (scripts/async-sweep.py) converts route handlers
that (a) are decorated with an HTTP verb, (b) contain a known blocking call,
(c) have NO await/async-with/async-for/yield, (d) are never used as a
coroutine elsewhere -> plain 'def'. Starlette then runs them in the AnyIO
threadpool, so the blocking call no longer stalls the gateway. await/stream/
websocket handlers are left untouched. Every file py_compile-checked.
243 handlers across 56 modules (system 17, qos 13, netdiag/hexo 12, hub 11...).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(aggregator): raise AnyIO threadpool to 80 tokens (#738)
The async-sweep moves ~243 blocking handlers to the threadpool. With ~110
modules in one process, the default 40-token pool can queue head-of-line under
concurrent blocking load. Raise to 80 on startup (best-effort, never breaks
boot).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(hub): double-buffer status cache + emojized health page (#738)
Navbar status (menu + health-batch) is now a strict double-buffer cache:
- request handlers NEVER compute on the request path — they return the current
snapshot instantly (or a 'warming' placeholder), so the sidebar's polling can
no longer serialize behind a ~3s systemctl walk and starve the loop;
- the background refresher is kicked from the request path (_ensure_bg) because
mounted sub-apps receive neither startup nor @app.middleware events under the
aggregator — the previous lazy-start middleware never fired there;
- snapshots are built complete then swapped atomically, so the dashboard never
shows partial/bad counts.
Served by the dedicated secubox-hub process (:8001, isolated loop) the navbar
stays <50ms and holds 200 under 25+ concurrent polls where the aggregator-
mounted copy wedged (000). health.js: 🟢🟡🔴 emoji status indicators.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style(hub): render health status emoji cleanly (neutralize .led dot) (#738)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(sidebar): kill per-module health storm, use batch endpoint (#738)
The navbar refreshed LEDs by firing ONE /api/v1/<module>/health request per
module — ~119 requests every 30s, in batches of 8 — straight at the aggregator's
single shared event loop. Combined with the in-process module mount this is a
prime driver of the recurring board-wide 502 wedge (user-identified).
checkAllHealth + refreshStaleHealth now call /api/v1/hub/public/health-batch
ONCE (served by the dedicated, double-buffered hub process) and populate every
module's LED from that single response. 119 reqs/cycle -> 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
/run/secubox parent to a module user, clobbering the canonical tmpfiles rule
(1777 root root) and 502'ing cross-user socket traversal:
- secubox-hub.service ExecStartPre (the active cascading culprit — fired on
every (re)start, right after secubox-runtime re-applied 1777 root root)
- secubox-eye-remote / metrics / metablogizer postinsts (chown parent)
- secubox-eye-square postinst (chowned BOTH /run/secubox AND /var/log/secubox
to secubox-eye-square — worst, #511 class)
- secubox-p2p postinst (chown parent + /var/log/secubox)
All now keep only mkdir as a fallback; the 1777 sticky parent lets each daemon
create its own socket, and module logs go to own subdirs (eye-square/p2p) instead
of owning the shared /var/log/secubox. Parent lifecycle is owned solely by
tmpfiles.d + secubox-runtime.service. Bumped: hub 1.4.4, eye-remote 1.0.1,
eye-square 1.0.4, metablogizer 1.2.2, metrics 1.0.4, p2p 1.7.1 (core 1.1.7
in prior commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 68d98bf1eb)
debian/secubox-metablogizer.service: lower uvicorn --log-level from
warning to info so the webhook deploy success line
`logger.info("deploy site=…")` reaches journald. Previously only
error paths were visible, making successful auto-deploys silent.
The 8 info-level log sites in api/main.py are all event-driven
(startup + webhook + sync) — no flood concern.
Closes#119.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
The hub's `_compute_menu_sync()` drops menu entries lacking an `id`
field. 6+ packages (lyrion, yacy, zigbee, rustdesk, grafana, authelia)
shipped a different schema (`title/url/section/module`) so they NEVER
appeared in the navbar. Plus the existing CATEGORY_META declared 12
sections (dashboard/security/network/system/core/users/services/
privacy/monitoring/publishing/apps/admin), not the 6 SecuBox charter
modules.
Changes:
* All 121 packages/secubox-*/menu.d/*.json normalised to the canonical
schema (id, name, path, category, icon, order, description). Legacy
schema aliases (title/url/module/section) preserved as fallbacks then
dropped from the file.
* Each menu entry's category remapped to one of the 6 charter modules:
AUTH (auth/users/identity/zkp/nac/openclaw),
WALL (crowdsec/waf/mitmproxy/hardening/threat-* /cve-triage/...),
BOOT (kernel-build/eye-remote/master-link/droplet/cloner/backup/...),
MIND (ai-gateway/mcp-server/grafana/ndpid/netifyd/glances/...),
ROOT (system/hub/portal/console/admin/vault/vm/rtty/...),
MESH (wireguard/dns/tor/matrix/gitea/nextcloud/mail/lyrion/yacy/
zigbee/dns-provider/rustdesk/... — all network + comms apps).
* secubox-hub v1.4.0 — CATEGORY_META rewritten with the 6 charter
modules carrying their official color from DESIGN-CHARTER.md and
the complementary-pair order. DEFAULT_MENU fallback remapped too.
* secubox-zigbee v2.5.3 — www/zigbee/index.html rewritten with the
canonical SecuBox scaffold (body display:flex, sidebar 220px fixed,
.main reserving 48px for the global-menu-bar). MESH palette.
Browser side: operators need to clear localStorage sbx_menu_cache (or
hard-refresh after the 1h TTL) to see the new sections after deploy.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
metablogizerctl 1.2.0:
site publish now accepts --public-domain <fqdn>. When given:
* Inject the FQDN into the nginx vhost server_name (so the
same nginx instance answers Host: <fqdn> coming back through
HAProxy → mitmproxy).
* Persist public_domain in site.json (consumed by unpublish).
* Run `haproxyctl vhost add <fqdn> mitmproxy_inspector true`.
* Trigger sync via `systemctl start sync-mitmproxy-routes.service`
(falls back to `sync-mitmproxy-routes.sh` in PATH; warns if
neither is available).
* Without the flag the behaviour is unchanged from 1.1.x.
site unpublish reads public_domain from site.json and auto-reverses
the HAProxy add + re-triggers the route sync. No-op if the site
was never published with --public-domain.
--public-domain is validated against ^[a-zA-Z0-9.-]+$ as defence-
in-depth against shell/TOML injection from API callers.
dropletctl 1.1.1:
cmd_publish and cmd_rename now pass --public-domain "$vhost" to
metablogizerctl so the full HAProxy → mitmproxy chain is wired
automatically. Closes the end-to-end gap reported by the gk2
smoke test of 1.1.0 (curl https://smoke.gk2.secubox.in/ → 502
because metablogizerctl publish only configured nginx).
Bump Depends: secubox-metablogizer (>= 1.2) — required for the
--public-domain flag.
Tests tightened to assert the new flag is passed on both publish
and rename. 17/17 bats green.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per CLAUDE.md the Punk Exposure Engine has three verbs (Peek/Poke/
Emancipate) and Tor is one of the three exposure channels. metablogizerctl
already handled site create/delete/publish/unpublish/list (the Poke verb
at the publishing layer); this adds the Emancipate verb:
tor expose <site> Publish site via Tor hidden service
tor revoke <site> Stop publishing via Tor
tor list List Tor-exposed sites + onion addresses
tor status <site> Stanza + onion + tor service state
When secubox-exposure is installed, the verbs delegate to it for
consistency with other exposure channels (DNS+SSL, Mesh) — one
orchestrator across all three channels. When unavailable, falls back
to direct /etc/tor/secubox-metablogizer.d/<site>.conf stanza writes
and `systemctl reload tor`, reading the resulting .onion hostname from
/var/lib/tor/secubox-metablogizer/<site>/hostname.
The onion address is persisted back into the site's site.json under
exposure.tor so the Peek verb (later, separate ticket) can surface it
without re-running tor status.
Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove Access and Actions tabs (duplicate / obsolete) along with the
migrate-from-OpenWrt modal and dead JS handlers. Make each row's Domain
column a clickable link to the live site and attach an emoji-rich title
tooltip (name, status, version, title, description, category, tags,
Streamlit app, port, size, last updated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Site dirs at /srv/metablogizer/sites/* are owned by root (from sub-B
ingest), but the FastAPI service runs as secubox. Git 2.35+ refuses
to operate with `fatal: detected dubious ownership in repository`,
turning every webhook-triggered deploy into a 500.
Pass `-c safe.directory=<site_dir>` per-invocation. Scoped to each
git call — no global git config change.
Existing git_pull test updated for the new arg layout
([git, -c, safe.directory=…, -C, <site>, <op>, ...]) and asserts the
override is on every call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Wires webhook.py into the FastAPI app: HMAC verify → classify →
per-site lock → git_pull → cache invalidate → conditional nginx
reload if site.json:domain changed → record + return.
- Git ops dispatched via loop.run_in_executor so the single uvicorn
worker stays responsive during concurrent deploys.
- GET /deploys (JWT-gated) exposes the last 50 deploys for ops review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- classify_payload(): pure dispatcher returning accept/skip/malformed
with a structured info dict. Covers non-metablog repos, non-default
refs (incl. tags + feature branches), and missing fields.
- site_lock(name): per-site asyncio.Lock pool with a master lock around
dict access so two concurrent first-creates don't race.
8 new pytest cases. 21 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- git_pull(site_dir, branch) -> (old_sha, new_sha) via 4 git ops
(rev-parse, fetch, reset, rev-parse) with timeouts (60s fetch,
10s local ops).
- Ring buffer holds the last 50 deploy records; oldest evicted
on overflow. list_deploys() returns newest-first.
5 new pytest cases cover the buffer and git ops (with subprocess
stubbed via monkeypatch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After deploying sub-C runtime (#110), /status and /sites became
extremely slow (3-30 s per request) because load_sites() walks
all 166 sites and runs enrich+validate+du -sh per entry. The new
60s polling on the dashboard amplified the load via concurrent
queued requests → browser-side NetworkError.
Two fixes here:
1. In-memory cache of load_sites() with a 30 s TTL. Hot reads
now return in microseconds. Invalidated explicitly from every
write path:
- POST /site (create)
- DELETE /site/<name>
- POST /site/<name>/publish
- POST /site/<name>/unpublish
- POST /republish-all
2. /sites no longer calls load_sites() twice. Was:
return {"sites": load_sites(), "count": len(load_sites())}
Now:
sites = load_sites()
return {"sites": sites, "count": len(sites)}
Tests in api/tests/test_sites_cache.py (5 cases) mirror the
contract in isolation — TTL-based caching, explicit invalidation,
identity stable for same-tick reads. All 15 metablogizer api tests
pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`uvicorn api.main:app` runs with WorkingDirectory=/usr/lib/secubox/metablogizer,
which puts the parent dir on sys.path but not api/ itself. Sub-C's
`from site_schema import …` (PR #102) and #106's `from rmtree import …`
both fail in production with ModuleNotFoundError, sending the service
into a restart loop and nginx into 502.
Insert `sys.path.insert(0, str(Path(__file__).resolve().parent))` at the
top of main.py so api/ is discoverable regardless of invocation style.
The pytest path (PYTHONPATH=api) is unaffected: the insert is a no-op
when api/ is already on sys.path.
All 10 metablogizer api tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sites cloned from Gitea (sub-B of #49 — 166 sites) ship a .git tree
whose pack files are 0444 and whose directories are 0500.
`shutil.rmtree` then fails on both `os.open(dir, O_RDONLY, ...)` (for
restricted directories) and `os.unlink(file)` (for files inside a
non-writable parent).
Extract _rmtree_force into api/rmtree.py with an onerror that chmods
both the parent dir AND the entry to 0700 before retrying the failing
op. The new helper handles both classes of EACCES that rmtree raises.
Tests in api/tests/test_rmtree_force.py:
- locked Gitea-style .git subtree (0500 dirs + 0444 packs) is cleared
- plain directory still cleared (no regression on non-locked sites)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `return res.json()` pattern in both index.html and site.html
returned the parse promise unawaited, so a JSON.parse rejection
escaped the surrounding try/catch and surfaced as
"Uncaught (in promise) SyntaxError" in the browser console.
Changes per file (index.html and site.html):
- `return await res.json()` so rejections enter the catch block
- Check `res.ok` and the `content-type` header before parsing
- `console.warn(path, status, content-type)` on any non-OK or
non-JSON response, so future failures are diagnosable in the
browser console without server-side digging
- Symmetric catch: log the thrown error with path context
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Match the rest of the SecuBox Hub codebase (index.html, login.html,
shared/api-utils.js, every other module) which reads localStorage
under 'sbx_token'. The plan's heredoc had 'jwt' which would have
caused an infinite login redirect on every drill-in page load.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces every site.json field (domain, version, last_updated, title,
description, category, tags) plus three external links: live site,
Gitea repo, Streamlit app (hidden when streamlit_app is null).
Same CRT P31 phosphor theme as index.html. Single fetch to
/api/v1/metablogizer/site/<name>; renders or shows a clear error
('site not found' / 'missing name').
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 3 new columns: Version (links to Gitea releases), Streamlit (icon link
when site has a streamlit_app), Updated (relative time, full ISO tooltip)
- Filter box above the table: live substring match on name + domain
- Sortable headers (Name, Domain, Version, Updated) with ▲/▼ indicator
- 60-second auto-refresh, paused when tab is hidden (Page Visibility API)
- Row name now links to site.html?name=<X> (drill-in, Task 2)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- _load_site_json() reads site.json, enriches from git, runs the
validator in warn-only mode (log violations, never reject)
- load_sites() overlays the enriched fields (version, last_updated,
streamlit_app, etc.) onto the per-site response
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
