Commit Graph

103 Commits

Author SHA1 Message Date
426b0b1fea feat(metablog): site.json backfill orchestrator (ref #101)
For each site under /srv/metablogizer/sites/:
- missing site.json -> create complete (name, domain, published, version, streamlit_app)
- present + --force -> merge missing fields, preserve existing keys/values
- present, no --force -> skip

streamlit_app auto-detected via Gitea ls-remote probe (5s timeout).
version from git describe --tags, else v1.0.0.
JSON report at output/metablog-backfill-report.json.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 17:25:59 +02:00
2f440c6194 Merge remote-tracking branch 'origin/master' into feature/95-streamlit-per-site-version-pinning-conta
# Conflicts:
#	.claude/HISTORY.md
#	.claude/WIP.md
#	.gitignore
2026-05-12 16:59:00 +02:00
e9192d3d32 fix(streamlit): Also fix _saved_source_args empty-array expansion (ref #95)
Same bug pattern as 34a4760e but on the source-helpers line.
Both empty-array restores now use "${var[@]}" without :-.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
9e614bd0f7 fix(streamlit): saved_args empty-array expansion (ref #95)
set -- "${saved_args[@]:-}" produces a single empty positional
argument when the array is empty, which the case-statement then
sees as Unknown flag "". Use "${saved_args[@]}" (no :-) which
gives zero positional args correctly on empty.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
343ebf8628 fix(streamlit): Also exclude __pycache__ from ingest scan (ref #95)
After excluding dot-dirs, __pycache__ still appeared as an app
candidate. Add explicit !-name '__pycache__' to the find filter.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
8d7a24d233 fix(streamlit): Exclude dot-directories from ingest scan (ref #95)
/srv/streamlit/apps/.claude exists (Claude tracking artifacts:
HISTORY.md, TODO.md, WIP.md) and was being picked up as an app
candidate. Filter out dot-prefixed directory names so only real
Streamlit app directories are considered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
f8cd25bb26 feat(streamlit): Orchestrator for Gitea ingest (preflights + report) (ref #95)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
2ecd96e688 feat(metablog): SSH preflight + Gitea key enrolment helper (ref #94)
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).

Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
c85d0d82a0 feat(streamlit): Per-app ingest function (idempotent, history-preserving) (ref #95)
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user 'gitea', not 'git'). Mirrors scripts/lib/metablog-ingest-site.sh
from sub-project B (PR #97) with the streamlit- prefix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:29:34 +02:00
a0e9ba8beb test(metablog): Smoke test for 3-site Gitea ingest (ref #94)
fix(metablog-ingest-site): Handle .git repos with unborn branch (no commits):
- Use git rev-parse --verify HEAD (exits non-zero on unborn branch) instead of
  rev-parse HEAD (which returns "HEAD" string even with no commits)
- Add fall-through path: .git exists + no commits → git symbolic-ref HEAD
  refs/heads/main + commit + push (same as fresh-init path)

test: Add tests/scripts/test-metablog-ingest.sh smoke test:
- Step 1: dry-run --limit 3 verifies report total=3
- Step 2: live --limit 3 accepts any ok+skip combo >= 3
- Step 3: idempotent re-run asserts all 3 skip-already-current
- Step 4: git ls-remote via MOCHAbin SSH verifies remote HEAD exists
- Step 5: clone+diff via MOCHAbin SSH (diff --exclude=.git) verifies
  clone == source

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
557d85b27e feat(metablog): Orchestrator for Gitea ingest (preflights + report) (ref #94)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
2900e25da9 feat(metablog): Per-site ingest function (idempotent, history-preserving) (ref #94)
URL uses gitea@ (Gitea built-in SSH server validates against the
OS user it runs as, which is 'gitea' in this LXC, not 'git').

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
a38011fca6 feat(metablog): SSH preflight + Gitea key enrolment helper (ref #94)
gitea CLI in 1.22 lacks 'user keys add', so enrolment goes:
generate-access-token (--config app.ini) -> POST /api/v1/user/keys via
LXC loopback -> delete token via sqlite3 (token list/delete API requires
basic auth in 1.22, not token auth).

Key finding: Gitea builtin SSH server (START_SSH_SERVER=true) rejects
the conventional "git@" username — the connecting username must match
the OS user Gitea runs as ("gitea" here). GITEA_SSH_USER var controls
this, defaulting to "gitea".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
08c998512f fix(metablog): awk INI patcher no longer duplicates [repository] section (ref #94)
The section-header rule flushed missing keys but did not set
saw_push/saw_branch to 1, so the END block then appended a second
[repository] block at EOF. Set the flags inline with the flush.

Also cleaned the live MOCHAbin app.ini which had been doubly
patched by the original buggy version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
ba972d1a8d feat(metablog): One-time Gitea config patch for push-create (ref #94)
- app.ini at /var/lib/gitea/custom/conf/app.ini (not /etc/gitea/)
- awk-based INI patcher (python3 not installed in gitea LXC)
- ENABLE_PUSH_CREATE_USER=true, DEFAULT_BRANCH=main applied
- Gitea restarted and confirmed active

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:10:06 +02:00
CyberMind
b0b6e78cab
Merge pull request #84 from CyberMind-FR/feature/license-headers-phase-a
feat(license): CMSD-1.0 header tool, CI, and conventions (Phase A)
2026-05-12 11:41:43 +02:00
b88b8ada95 fix(license): tighten detect_existing + missing-file = repo-wide (ref #81)
Two bug fixes to the CMSD-1.0 license-header tool:

1. detect_existing() previously matched the SPDX token anywhere in the
   first 10 lines, including inside docstrings and prose comments.
   Tightened the matcher to require comment markers (#, //, *, <!--)
   and whitespace before the SPDX token.

   Regression tests:
   - test_detect_existing_no_false_match_in_docstring
   - test_detect_existing_no_false_match_inline_comment_prose

2. _read_enrollment() now returns ["**"] when the allowlist file is
   absent, per spec §5.2 "missing file → repo-wide enforcement".
   Previously it returned [] (nothing enforced), making Phase C closure
   impossible without an additional file.

   Empty allowlist file still returns [] (Phase A initial), so
   test_main_empty_allowlist_passes_check is unaffected.

   New tests:
   - test_read_enrollment_missing_file_means_repo_wide
   - test_main_check_missing_allowlist_enforces_repo_wide

Suite: 49 → 53 passing. --check still exits 0 repo-wide.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:39:57 +02:00
ba9f6b20c1 fix(scripts): re-apply secubox-haproxy-regen-safe sub-command fix
A merge from feature/83 reverted the fix from commit 22014865, which had
already replaced `haproxyctl regen` (non-existent sub-command, printed
help text and exited 0) with the correct `haproxyctl generate`.

This commit re-applies the same fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:17:34 +02:00
2763009fad fix(license): walk() accepts repo_root for subdir invocations (ref #81)
Discovered during Phase B pilot: running `--fix packages/secubox-hub`
yielded zero files because walk() computed `rel` relative to the walk
root (the subdir) while allowlist patterns are repo-relative.

Add optional `repo_root` parameter to walk(); when present, allowlist
matching uses paths relative to it instead of the walk root.
Backwards-compatible (defaults to None, preserving existing tests).
main() passes the discovered repo_root through.

Regression test added: test_walk_subdir_with_repo_root_arg.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:03:32 +02:00
e31c1fa99b fix(sync-routes): preserve streamlit & custom routes from clobbering
After Session 156 fix, secubox-haproxy.service was safely restarted but
cpf.gk2.secubox.in (and other streamlit instances) immediately fell back
to "Wrong Domain" because sync-mitmproxy-routes.sh was clobbering their
container routes.

Two compounding bugs:
1. DEAD_CONTAINER_IPS included 10.100.0.50 — but that's the streamlit
   LXC's actual IP (RUNNING). fix_dead_container_routes() rewrote every
   route pointing to 10.100.0.50 (cpf:8523, etc.) to 10.100.0.1:9080.
2. The update conditional reapplied routes when the existing port didn't
   match the computed port. The computed port logic only knows metablog
   vs webui — it has no streamlit awareness, so it always tried to force
   streamlit domains to port 9080 (webui default).

Fixes:
- Drop 10.100.0.50 from DEAD_CONTAINER_IPS. The streamlit LXC at that
  address is alive; routes pointing to it are valid.
- Change "update if existing missing OR port mismatch" to "update only
  if existing is missing." The script now only ADDS routes for ACL/
  metablog domains; it never overwrites existing routes set by other
  sources (sync-all-routes.sh, manual edits, etc.). fix_dead_container_
  routes() still handles truly dead IPs separately.

Verified: cpf.gk2.secubox.in → HTTP 200 "Streamlit" persists across
sync-mitmproxy-routes.service runs. arm/admin still 200 with correct
content.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 11:02:42 +02:00
c911e8192f fix(haproxy): generator emits mitmproxy_inspector + safe regen wrapper
Triggered by cpf.gk2.secubox.in returning "Wrong Domain". Root cause: the
secubox-haproxy generators (both bash haproxyctl and Python FastAPI) emit
config rules that point to dead/wrong backends, leading to HTTPS-wide
metablog/streamlit outages when a regen runs.

Generator fixes:
- packages/secubox-haproxy/api/main.py: use_backend waf_inspector
  → use_backend mitmproxy_inspector (2 occurrences). waf_inspector
  pointed to 127.0.0.1:8890 which is not listening; mitmproxy_inspector
  is the actual healthy WAF backend at 10.100.0.60:8080.
- packages/secubox-haproxy/sbin/haproxyctl: default_backend fallback
  → default_backend mitmproxy_inspector (2 occurrences). The fallback
  backend deny_status 503's unknown hosts; switching to mitmproxy_inspector
  lets mitmproxy dispatch all hosts via its routes JSON (245 routes vs
  93 declared in haproxy.toml).

New safe regen wrapper:
- scripts/secubox-haproxy-regen-safe: snapshot → regen via haproxyctl →
  validate via haproxy -c -f → atomic swap → reload. On validation
  failure, current cfg is preserved and the broken candidate is saved
  for forensics. Prevents another silent breakage like Session 156.

Board-side state (not in repo): /etc/haproxy/haproxy.cfg patched in
place to mirror these fixes, secubox-haproxy.service stopped until
user confirms safe to re-enable. All 245 routes verified live (cpf,
arm, lldh, admin, pub, werdl, 3d, 42, zkp all HTTP 200 with correct
content).

See .claude/HISTORY.md Session 156 for full symptom chain and the
6-layer root cause analysis.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:52:44 +02:00
c661c4f053 fix(scripts): use issue title for PR, surface push errors as exit 4 (ref #83)
Addresses code review:
- Important #4: finish now fetches issue title via `gh issue view --json title`
  and uses it as PR title (was using the branch name). Falls back to branch
  name if the fetch fails or returns empty.
- Minor #8: explicit `return 4` on `git push` failure for consistency with
  the documented exit-code contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:22:14 +02:00
b77a49fc2f docs(scripts): document agent-worktree in scripts/README (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:05:48 +02:00
4ff4830df4 feat(scripts): implement agent-worktree clean sub-command (ref #83)
Also patches gh-mock to sanitize dashes in branch-derived env var names,
and fixes --head two-arg parsing in the pr view mock handler.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:03:47 +02:00
9eb75676d5 docs(license): document license-headers.py in scripts/README (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:01:46 +02:00
4d937995ba ci(license): add License Headers workflow (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:54 +02:00
eba35ff5b5 feat(scripts): implement agent-worktree finish sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:24 +02:00
e90a07a578 chore(license): self-host header on tool and tests + initial allowlist (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:00:09 +02:00
4554cb991e feat(scripts): implement agent-worktree sync sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:59:01 +02:00
c93cfdb639 feat(license): CLI dispatch with --check/--fix/--list/--diff (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:57:19 +02:00
70e145ba62 feat(scripts): implement agent-worktree list sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:57:00 +02:00
149304911d test(scripts): cover agent-worktree start edge cases (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:55:17 +02:00
d5df48c5d8 fix(infra): mitmproxy route sync stability — restore metablogizer sites
Root cause: log() in sync-mitmproxy-routes.sh wrote to stdout, polluting
$(fix_dead_container_routes ...) capture → corrupted JSON → jq parse
errors → set -e abort → bad routes pushed to mitmproxy container →
mitmproxy restart-loop → HAProxy backend DOWN → HTTP 503 on every
*.gk2.secubox.in metablog vhost (arm, zkp, 3d, ~160 sites).

Fixes:
- log() now writes to stderr (>&2) — preserves stdout for command-sub
- fix_dead_container_routes returns 0 (was returning $fixed count, set -e
  killed the caller's assignment)
- sync-all-routes.sh routes metablog domains to port 8900 (was 9080,
  which is webui.conf default_server → "Wrong Domain")
- defensive 2>/dev/null || true on jq reads
- fallback to old routes_json on jq write failure (preserves last valid
  state instead of crashing)
- flock guard prevents concurrent runs on /run/sync-mitmproxy-routes.lock

Verified: systemctl start sync-mitmproxy-routes.service exits 0/SUCCESS,
244 routes valid in mitmproxy container, all four spot-checked domains
(admin.gk2, arm.gk2, zkp.gk2, 3d.gk2) return HTTP 200 with correct titles.

Note: also disabled duplicate timer secubox-route-sync.timer on the
board (executed the same script, racing on same file). systemd-level
change is on board only, not in repo.

See .claude/HISTORY.md Session 153 for full symptom chain and verification.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:54:52 +02:00
72733385eb feat(license): walk() with skip-list and enrollment allowlist (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:54:35 +02:00
496592913b feat(scripts): implement agent-worktree start sub-command (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:54:12 +02:00
33768c7854 feat(scripts): add agent-worktree CLI skeleton with help and dispatch (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:51:38 +02:00
2bd965f481 feat(license): apply() for HTML (doctype) and Markdown (frontmatter), plus TOML/YAML/conf (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:50:48 +02:00
a67c4c6b6e feat(scripts): add prefix_from_labels helper (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:50:35 +02:00
ada5040693 feat(license): apply() for JS/TS/CSS/C/H with idempotence (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:49:28 +02:00
c74e71d890 feat(scripts): add agent-worktree library with derive_slug (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:49:02 +02:00
6474728dc4 feat(license): apply() for Bash with shebang-aware placement (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:48:43 +02:00
6f4bd948a3 test(scripts): add gh CLI mock for agent-worktree tests (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:46:20 +02:00
63e62c1d54 feat(license): apply() for plain Python, with idempotence and foreign-skip (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:45:34 +02:00
72771702e9 chore(scripts): scaffold tests directory for agent-worktree (ref #83)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:44:34 +02:00
5b4d3e25fb feat(license): detect_existing distinguishes CMSD/foreign/none (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:41:04 +02:00
0f1907df15 fix(scripts): Decouple chroot apt-update grep from pipefail (ref #80)
apt-get update may exit non-zero on unsigned/file:// warnings while
the SecuBox repo is correctly discovered. Capture the log first
(via tee, so root-owned chroot output is writeable), then grep
separately so pipefail can't mask a successful discovery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:39:14 +02:00
70ac535e68 feat(license): render_header for slash/block/html comment styles (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:38:20 +02:00
f08eb38073 feat(license): render_header for hash-comment languages (ref #81)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:34:43 +02:00
bd7975d12f chore(license): fix NotImplementedError task reference (ref #81)
Per code review: main() is implemented in Task 11 (CLI dispatch), not Task 9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:33:03 +02:00
bb58789b58 feat(scripts): Add validate-staged-repo.sh (ref #80)
Four-step validation gate: reprepro check, gpg verify on InRelease,
license byte-match against project root, optional chroot apt-update
smoke test (SKIP_CHROOT=1 to disable).

Accepts French or English "Good signature" / "Bonne signature" so
the gate works under any locale.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:32:46 +02:00