mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-29 13:59:40 +00:00
fix(infra): mitmproxy route sync stability — restore metablogizer sites
Root cause: log() in sync-mitmproxy-routes.sh wrote to stdout, polluting $(fix_dead_container_routes ...) capture → corrupted JSON → jq parse errors → set -e abort → bad routes pushed to mitmproxy container → mitmproxy restart-loop → HAProxy backend DOWN → HTTP 503 on every *.gk2.secubox.in metablog vhost (arm, zkp, 3d, ~160 sites). Fixes: - log() now writes to stderr (>&2) — preserves stdout for command-sub - fix_dead_container_routes returns 0 (was returning $fixed count, set -e killed the caller's assignment) - sync-all-routes.sh routes metablog domains to port 8900 (was 9080, which is webui.conf default_server → "Wrong Domain") - defensive 2>/dev/null || true on jq reads - fallback to old routes_json on jq write failure (preserves last valid state instead of crashing) - flock guard prevents concurrent runs on /run/sync-mitmproxy-routes.lock Verified: systemctl start sync-mitmproxy-routes.service exits 0/SUCCESS, 244 routes valid in mitmproxy container, all four spot-checked domains (admin.gk2, arm.gk2, zkp.gk2, 3d.gk2) return HTTP 200 with correct titles. Note: also disabled duplicate timer secubox-route-sync.timer on the board (executed the same script, racing on same file). systemd-level change is on board only, not in repo. See .claude/HISTORY.md Session 153 for full symptom chain and verification. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
496592913b
commit
d5df48c5d8
|
|
@ -4,6 +4,70 @@
|
|||
---
|
||||
## 2026-05-12
|
||||
|
||||
### Session 153 — Mitmproxy Route Sync Stability Fix
|
||||
|
||||
**Goal:** All `*.gk2.secubox.in` metablogizer sites (arm, zkp, 3d, …) returned "Wrong Domain" page on HTTPS. Investigate, restore service, and harden the auto-sync infrastructure.
|
||||
|
||||
**Symptom chain:**
|
||||
|
||||
1. User report: `https://arm.gk2.secubox.in/` → "Wrong Domain - SecuBox" landing page.
|
||||
2. `~160` metablogizer sites affected (all routed via mitmproxy WAF).
|
||||
3. Two systemd timers (`sync-mitmproxy-routes.timer` + `secubox-route-sync.timer`) firing at the same second on the same routes file.
|
||||
4. `sync-mitmproxy-routes.service` had been failing with exit code 30/4 for >30 minutes (chronic).
|
||||
|
||||
**Root cause (deepest):**
|
||||
|
||||
`log()` function in `/usr/local/bin/sync-mitmproxy-routes.sh` wrote to **stdout**. `fix_dead_container_routes()` calls `log "Fixing dead route: …"` and is itself captured via `routes_json=$(fix_dead_container_routes "$routes_json")` — every log line was concatenated into the JSON variable, producing `[date] Fixing dead route: …{"255.gk2..."`. jq then complained `Invalid numeric literal at line 1, column 12`, `set -e` killed the script, and the corrupted JSON got pushed to the mitmproxy container (`lxc-attach … tee /srv/mitmproxy/haproxy-routes.json`). mitmproxy restart-looped on `Failed to load routes: Expecting ',' delimiter`, HAProxy backend `mitmproxy_inspector` went DOWN, every public domain returned **HTTP 503**.
|
||||
|
||||
**Additional contributing bugs:**
|
||||
|
||||
- `fix_dead_container_routes` returned `$fixed` (a count) instead of `0` — non-zero return tripped `set -e` in the caller's command substitution.
|
||||
- `sync-all-routes.sh` step 2 wrote metablogizer routes to port **9080** (nginx default_server → "Wrong Domain") instead of **8900** (where nginx actually listens with `server_name` per metablog site).
|
||||
- jq read calls had no error tolerance — any malformed JSON fed in via `$routes_json` aborted the whole script via `set -euo pipefail`.
|
||||
- Two systemd services bound to the **same script** (`sync-mitmproxy-routes.service` + `secubox-route-sync.service`) racing on the same file.
|
||||
|
||||
**Fixes applied:**
|
||||
|
||||
| # | Fix | Cible | Mechanism |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Routes patchées 9080→8900 (165 sites metablog) | `/srv/mitmproxy/haproxy-routes.json` (host + container) | Python script (extract `server_name` from nginx, rewrite port) |
|
||||
| 2 | `return $fixed` → `return 0` | `sync-mitmproxy-routes.sh:fix_dead_container_routes` | sed |
|
||||
| 3 | Port metablog 9080 → 8900 in step 2 | `sync-all-routes.sh:62` | sed |
|
||||
| 4 | **`log()` writes to stderr** (root-cause fix) | `sync-mitmproxy-routes.sh:log` | adds `>&2` so `$()` capture is clean |
|
||||
| 5 | Defensive `2>/dev/null \|\| true` on jq read | `sync-mitmproxy-routes.sh` (2 occurrences) | tolerate corrupted input |
|
||||
| 6 | Fallback preserving old `routes_json` on jq write failure | `sync-mitmproxy-routes.sh` (3 occurrences) | `new_rj=$(... \|\| true); [[ -n "$new_rj" ]] && routes_json="$new_rj"` |
|
||||
| 7 | `flock -n` guard prevents concurrent runs | `sync-mitmproxy-routes.sh` (head) | `/run/sync-mitmproxy-routes.lock` |
|
||||
| 8 | Disable duplicate timer | `systemctl disable --now secubox-route-sync.timer` | systemd |
|
||||
|
||||
**Verification (post-fix):**
|
||||
|
||||
- `sync-mitmproxy-routes.service` via systemd → exit 0/SUCCESS, "Sync complete".
|
||||
- Container routes JSON valid: 244 keys, all metablogizer domains → `[10.100.0.1, 8900]`.
|
||||
- mitmproxy: `active`, listening on `0.0.0.0:8080`.
|
||||
- HAProxy backend `mitmproxy_inspector srv0 10.100.0.60:8080` op_state=UP.
|
||||
- Live tests: `admin.gk2`, `arm.gk2`, `zkp.gk2`, `3d.gk2` all HTTP 200 with correct titles.
|
||||
|
||||
**Files modified (versioned in repo):**
|
||||
|
||||
- `scripts/sync-mitmproxy-routes.sh` (synced from board `/usr/local/bin/`)
|
||||
- `scripts/sync-all-routes.sh` (new in repo, synced from board)
|
||||
|
||||
**Backups created on board (timestamped, kept for rollback):**
|
||||
|
||||
- `/srv/mitmproxy/haproxy-routes.json.bak.<epoch>` (pre-patch JSON)
|
||||
- `/usr/local/bin/sync-mitmproxy-routes.sh.bak.<epoch>` and `.bak.<epoch>-preflock`
|
||||
- `/usr/local/bin/sync-all-routes.sh.bak.<epoch>`
|
||||
|
||||
**Topology note discovered:**
|
||||
|
||||
- Canonical Hub vhosts (nginx `sites-available/secubox-local`): `admin.gk2.secubox.in`, `gk2.secubox.in`, `secubox.maegia.tv`, `c3box.maegia.tv` + LAN aliases.
|
||||
- `~165` metablogizer sites listed in `/etc/nginx/sites-enabled/metablogizer`, each `listen 0.0.0.0:8900` with per-site `server_name`, `root /srv/metablogizer/sites/<name>/`.
|
||||
- Public flow: HAProxy `https-in` (443) → ACL host match → backend `mitmproxy_inspector` (LXC `10.100.0.60:8080`) → mitmproxy looks up host in `haproxy-routes.json` → upstream `[10.100.0.1, 8900]` → nginx vhost matches `server_name` → serves static site.
|
||||
|
||||
**Open follow-up:** the `sync-streamlit-routes.timer` last fired 2026-05-10 (1d 15h ago) and didn't fire since — needs separate investigation. Not blocking metablog/Hub stability.
|
||||
|
||||
---
|
||||
|
||||
### Session 152 — APT Public Repo Staging Pipeline (Issue #80)
|
||||
|
||||
**Goal:** Stage a complete signed APT repo at `output/repo/` for `bookworm` × {arm64, amd64}, validated end-to-end. User pushes to `apt.secubox.in` out-of-band.
|
||||
|
|
|
|||
91
scripts/sync-all-routes.sh
Executable file
91
scripts/sync-all-routes.sh
Executable file
|
|
@ -0,0 +1,91 @@
|
|||
#!/bin/bash
|
||||
# SecuBox Routes Auto-Sync - All Services
|
||||
# Syncs routes from streamlit, metablogizer, and other services to mitmproxy
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROUTES_FILE="/srv/mitmproxy/haproxy-routes.json"
|
||||
LOG_TAG="routes-sync"
|
||||
|
||||
log() { logger -t "$LOG_TAG" "$1"; echo "[$(date +%H:%M:%S)] $1"; }
|
||||
|
||||
log "Starting full routes sync..."
|
||||
|
||||
# Ensure mitmproxy container is running
|
||||
if ! lxc-info -n mitmproxy 2>/dev/null | grep -q "RUNNING"; then
|
||||
log "Starting mitmproxy container..."
|
||||
lxc-start -n mitmproxy 2>/dev/null || true
|
||||
sleep 3
|
||||
fi
|
||||
|
||||
# Get container IPs
|
||||
STREAMLIT_IP=$(lxc-attach -n streamlit -- ip -4 addr show eth0 2>/dev/null | grep -oP "inet \K[\d.]+" || echo "10.100.0.50")
|
||||
GITEA_IP=$(lxc-attach -n gitea -- ip -4 addr show eth0 2>/dev/null | grep -oP "inet \K[\d.]+" || echo "10.100.0.40")
|
||||
|
||||
python3 << PYEOF
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
routes_file = "$ROUTES_FILE"
|
||||
streamlit_ip = "$STREAMLIT_IP"
|
||||
gitea_ip = "$GITEA_IP"
|
||||
host_ip = "192.168.1.200"
|
||||
|
||||
# Load existing routes
|
||||
routes = {}
|
||||
if Path(routes_file).exists():
|
||||
routes = json.loads(Path(routes_file).read_text())
|
||||
|
||||
updated = 0
|
||||
|
||||
# 1. Streamlit instances
|
||||
try:
|
||||
import tomllib
|
||||
with open("/etc/secubox/streamlit.toml", "rb") as f:
|
||||
cfg = tomllib.load(f)
|
||||
for name, inst in cfg.get("instances", {}).items():
|
||||
domain = inst.get("domain", f"{name}.gk2.secubox.in")
|
||||
port = inst.get("port")
|
||||
if port:
|
||||
routes[domain] = [streamlit_ip, int(port)]
|
||||
updated += 1
|
||||
print(f"Streamlit: {updated} routes")
|
||||
except Exception as e:
|
||||
print(f"Streamlit error: {e}")
|
||||
|
||||
# 2. Metablogizer sites (static sites served by nginx on 9080)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["curl", "-s", "--unix-socket", "/run/secubox/metablogizer.sock", "http://localhost/access"],
|
||||
capture_output=True, text=True, timeout=5
|
||||
)
|
||||
data = json.loads(result.stdout)
|
||||
mb_count = 0
|
||||
for site in data.get("sites", []):
|
||||
domain = site.get("domain")
|
||||
if domain and site.get("published"):
|
||||
routes[domain] = [host_ip, 8900]
|
||||
mb_count += 1
|
||||
print(f"Metablogizer: {mb_count} routes")
|
||||
updated += mb_count
|
||||
except Exception as e:
|
||||
print(f"Metablogizer error: {e}")
|
||||
|
||||
# 3. Fixed routes
|
||||
routes["admin.gk2.secubox.in"] = [host_ip, 9080]
|
||||
routes["git.gk2.secubox.in"] = [gitea_ip, 3000]
|
||||
|
||||
# Save routes
|
||||
Path(routes_file).write_text(json.dumps(routes, indent=2, sort_keys=True))
|
||||
print(f"Total: {len(routes)} routes saved")
|
||||
PYEOF
|
||||
|
||||
# Copy routes to mitmproxy container
|
||||
log "Syncing to mitmproxy container..."
|
||||
cat "$ROUTES_FILE" | lxc-attach -n mitmproxy -- tee /srv/mitmproxy/haproxy-routes.json > /dev/null
|
||||
|
||||
# Reload mitmproxy
|
||||
lxc-attach -n mitmproxy -- pkill -HUP mitmdump 2>/dev/null || true
|
||||
|
||||
log "Routes sync complete"
|
||||
|
|
@ -1,4 +1,10 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Re-exec self under flock to prevent concurrent runs (added 2026-05-12)
|
||||
LOCK=/run/sync-mitmproxy-routes.lock
|
||||
exec 9>"$LOCK"
|
||||
flock -n 9 || { echo "[$(date "+%F %T")] another instance running, skipping"; exit 0; }
|
||||
|
||||
# SecuBox Route Sync - Ensures HAProxy vhosts are synced to mitmproxy routes
|
||||
# Run periodically via cron or systemd timer
|
||||
#
|
||||
|
|
@ -23,7 +29,7 @@ BACKEND_IP="10.100.0.1"
|
|||
DEAD_CONTAINER_IPS="10.100.0.10 10.100.0.20 10.100.0.30 10.100.0.40 10.100.0.50"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >&2
|
||||
}
|
||||
|
||||
# Get current routes from container
|
||||
|
|
@ -71,13 +77,14 @@ fix_dead_container_routes() {
|
|||
for domain in $domains; do
|
||||
[[ -z "$domain" ]] && continue
|
||||
log "Fixing dead route: $domain ($dead_ip -> $BACKEND_IP:$WEBUI_PORT)"
|
||||
routes_json=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$WEBUI_PORT" '.[$d] = [$ip, $p]')
|
||||
new_rj=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$WEBUI_PORT" '.[$d] = [$ip, $p]' 2>/dev/null || true)
|
||||
if [[ -n "$new_rj" ]]; then routes_json="$new_rj"; fi
|
||||
fixed=$((fixed + 1))
|
||||
done
|
||||
done
|
||||
|
||||
echo "$routes_json"
|
||||
return $fixed
|
||||
return 0
|
||||
}
|
||||
|
||||
# Main sync
|
||||
|
|
@ -101,11 +108,12 @@ main() {
|
|||
port=$(get_port_for_domain "$domain")
|
||||
|
||||
# Check if route exists with correct port
|
||||
existing=$(echo "$routes_json" | jq -r --arg d "$domain" '.[$d] // empty')
|
||||
existing=$(echo "$routes_json" | jq -r --arg d "$domain" '.[$d] // empty' 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$existing" ]] || ! echo "$existing" | jq -e ".[1] == $port" >/dev/null 2>&1; then
|
||||
log "Updating route: $domain -> $BACKEND_IP:$port"
|
||||
routes_json=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$port" '.[$d] = [$ip, $p]')
|
||||
new_rj=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$port" '.[$d] = [$ip, $p]' 2>/dev/null || true)
|
||||
if [[ -n "$new_rj" ]]; then routes_json="$new_rj"; fi
|
||||
updated=$((updated + 1))
|
||||
fi
|
||||
done < <(get_haproxy_domains)
|
||||
|
|
@ -114,11 +122,12 @@ main() {
|
|||
while IFS= read -r domain; do
|
||||
[[ -z "$domain" ]] && continue
|
||||
|
||||
existing=$(echo "$routes_json" | jq -r --arg d "$domain" '.[$d] // empty')
|
||||
existing=$(echo "$routes_json" | jq -r --arg d "$domain" '.[$d] // empty' 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$existing" ]] || ! echo "$existing" | jq -e ".[1] == $METABLOG_PORT" >/dev/null 2>&1; then
|
||||
log "Updating metablog route: $domain -> $BACKEND_IP:$METABLOG_PORT"
|
||||
routes_json=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$METABLOG_PORT" '.[$d] = [$ip, $p]')
|
||||
new_rj=$(echo "$routes_json" | jq --arg d "$domain" --arg ip "$BACKEND_IP" --argjson p "$METABLOG_PORT" '.[$d] = [$ip, $p]' 2>/dev/null || true)
|
||||
if [[ -n "$new_rj" ]]; then routes_json="$new_rj"; fi
|
||||
updated=$((updated + 1))
|
||||
fi
|
||||
done < <(get_metablog_domains)
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user