mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-29 11:12:29 +00:00
fix(health): VHost health prober placeholder categorization + Eye Remote metrics
- VHost Health: Add "placeholder" status category to prober
- Placeholders no longer counted as "down"
- Health % only reflects real services
- Dashboard shows ⬜ indicator for placeholders
- Eye Remote API: Enhanced metrics exposure
- Proper CPU calculation (not load average)
- Memory free/total in MB
- Disk free/total in GB
- Active connections and sessions
- Services running count (86 sockets)
- VHost health summary
- HAProxy workflow script (scripts/haproxy-workflow.sh)
- rehealth: Reload HAProxy, invalidate caches, restart probers
- waf-sync: Sync vhost→backend routes to mitmproxy
- certs: Check certificate status for all vhosts
- mitmproxy routing: Fixed 127.0.0.1 → 10.100.0.1 (LXC bridge IP)
to prevent routing loop causing CPU spike
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
2454abd233
commit
f42ff0fd9c
103
.claude/WIP.md
103
.claude/WIP.md
|
|
@ -1,5 +1,106 @@
|
|||
# WIP — Work In Progress
|
||||
*Mis à jour : 2026-05-07 (Session 106)*
|
||||
*Mis à jour : 2026-05-06 (Session 108)*
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Session 108 (Continued): VHost Health Fixes + Services Restore
|
||||
|
||||
### VHost Health Prober
|
||||
- [x] Updated prober to treat placeholders as "placeholder" status (not "down")
|
||||
- [x] Added ⬜ placeholder indicator to dashboard
|
||||
- [x] Health % now only counts real vhosts (excludes placeholders)
|
||||
- [x] Current stats: 🟢 0 🟡 5 🔴 29 ⬜ 58 (14.7% health)
|
||||
|
||||
### HAProxy Routing Fixes
|
||||
- [x] Fixed c3box.maegia.tv → metablog_gandalf (was nginx_vhosts placeholder)
|
||||
- [x] mitmproxy backend confirmed UP at 10.100.0.60:8080
|
||||
|
||||
### Service Socket Restoration
|
||||
- [x] Restarted all secubox-* services
|
||||
- [x] 86 API sockets now active
|
||||
- [x] Verified hub, crowdsec, system, haproxy all working
|
||||
|
||||
### System Load Issues (Ongoing)
|
||||
- [ ] mitmdump using 95% CPU (821MB RAM) - investigate
|
||||
- [ ] Memory at 91% (7.1GB/7.7GB)
|
||||
- [ ] Load average: 11-17 (very high)
|
||||
|
||||
### Metrics Dashboard (User Request)
|
||||
- [ ] Add precaching + double-buffer with threaded updating
|
||||
- [ ] Currently 1.6s response time due to high load
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Session 108: Dashboard Health Widgets + HAProxy Workflow
|
||||
|
||||
### HAProxy Workflow Script
|
||||
- [x] Created `scripts/haproxy-workflow.sh` with 3 commands:
|
||||
- `rehealth` — Auto-rehealth HAProxy (reload, invalidate caches, restart probers)
|
||||
- `waf-sync` — Sync vhost→backend routes through WAF (mitmproxy)
|
||||
- `certs` — Check/list certificate status for all vhosts
|
||||
- `all` — Run complete workflow
|
||||
|
||||
### Dashboard Health Widgets (Upper Position)
|
||||
- [x] Added Module Health widget to dashboard index.html
|
||||
- Progress bar + health percentage
|
||||
- Emoji counters 🟢🟡🔴 (healthy/degraded/down)
|
||||
- Calls `/api/v1/hub/module-health/summary`
|
||||
- [x] Added VHost Health widget to dashboard index.html
|
||||
- Progress bar + health percentage
|
||||
- Emoji counters 🟢🟡🔴 (ok/slow/down)
|
||||
- Calls `/api/v1/hub/health-monitor/summary`
|
||||
- [x] Positioned both widgets upper in grid (before main content)
|
||||
|
||||
### Dashboard Widget Limits
|
||||
- [x] Modules Overview: Reduced from 10 to 4 items
|
||||
- [x] Alert Timeline: Reduced from 5 to 3 items
|
||||
|
||||
### Hub API Health Endpoints
|
||||
- [x] Added `/module-health/summary` — Module health counts
|
||||
- [x] Added `/module-health/status` — Detailed module status
|
||||
- [x] Added `/module-health/alerts` — Degraded/down modules
|
||||
- [x] Added `/health-monitor/summary` — VHost health counts
|
||||
- [x] Added `/health-monitor/status` — Detailed VHost status
|
||||
- [x] Added `/health-monitor/alerts` — Slow/down VHosts
|
||||
|
||||
---
|
||||
|
||||
## ✅ Session 107: Module Health Monitor + Placeholder Detection
|
||||
|
||||
### Admin Page Fixes
|
||||
- [x] Fixed "undefined/undefined" bug in System Administration page
|
||||
- [x] Added null coalescing (`?? 0`) for missing API data
|
||||
|
||||
### Module Health Prober System
|
||||
- [x] Created `/usr/lib/secubox/health/module_prober.py`
|
||||
- [x] Multilayer checks: systemd → socket → API
|
||||
- [x] 8 core modules monitored (hub, crowdsec, dpi, haproxy, vhost, wireguard, system, heartbeat)
|
||||
- [x] Results cached to `/var/cache/secubox/health/modules.json`
|
||||
- [x] Created `secubox-module-prober.service` (enabled)
|
||||
- [x] Added API router `/api/v1/hub/module-health/`:
|
||||
- `GET /summary` - Global module health stats
|
||||
- `GET /status` - All modules with layer details
|
||||
- `GET /module/{name}` - Single module status
|
||||
- `GET /alerts` - Degraded/down modules
|
||||
- [x] Dashboard widget "Module Health" with:
|
||||
- Progress bar + health %
|
||||
- Emoji counters 🟢🟡🔴
|
||||
- Alert list for degraded/down modules
|
||||
|
||||
### Sidebar Module Filtering
|
||||
- [x] Menu API now hides inactive modules from sidebar
|
||||
- [x] Only functional modules displayed (systemd active)
|
||||
- [x] Reduces sidebar clutter for partially-deployed systems
|
||||
|
||||
### VHost Health Prober Enhancements
|
||||
- [x] Added placeholder page detection (`SecuBox Domain` / `SecuBox Protected`)
|
||||
- [x] Sites showing placeholder = marked as "down" (not "ok")
|
||||
- [x] Real health: 7 slow, 85 down (mostly placeholder pages)
|
||||
|
||||
### VHost Routing Fixes
|
||||
- [x] Fixed gandalf.maegia.tv routing
|
||||
- [x] Added `metablog_gandalf` backend (port 8901)
|
||||
- [x] Updated HAProxy ACL → metablog_gandalf
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -939,6 +939,157 @@ async def health():
|
|||
return {"status": "ok", "module": "hub", "version": "1.7.0"}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
# Module Health Monitor Endpoints
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
|
||||
MODULE_HEALTH_CACHE = Path("/var/cache/secubox/health/modules.json")
|
||||
VHOST_HEALTH_CACHE = Path("/var/cache/secubox/health/status.json")
|
||||
|
||||
|
||||
@router.get("/module-health/summary")
|
||||
async def module_health_summary(user=Depends(require_jwt)):
|
||||
"""Get module health summary (healthy/degraded/down counts)."""
|
||||
try:
|
||||
if MODULE_HEALTH_CACHE.exists():
|
||||
data = json.loads(MODULE_HEALTH_CACHE.read_text())
|
||||
# Use pre-computed values from cache if available
|
||||
if "ok" in data and "degraded" in data:
|
||||
return {
|
||||
"health_percent": data.get("health_pct", 0),
|
||||
"healthy": data.get("ok", 0),
|
||||
"degraded": data.get("degraded", 0),
|
||||
"down": data.get("down", 0),
|
||||
"total": data.get("total", 0),
|
||||
}
|
||||
# Fallback: compute from modules dict
|
||||
modules = data.get("modules", {})
|
||||
healthy = sum(1 for m in modules.values() if m.get("overall") == "ok")
|
||||
degraded = sum(1 for m in modules.values() if m.get("overall") == "degraded")
|
||||
down = sum(1 for m in modules.values() if m.get("overall") in ("down", "error"))
|
||||
total = len(modules)
|
||||
return {
|
||||
"health_percent": (healthy / max(total, 1)) * 100,
|
||||
"healthy": healthy,
|
||||
"degraded": degraded,
|
||||
"down": down,
|
||||
"total": total,
|
||||
}
|
||||
except Exception as e:
|
||||
log.debug("Module health cache read failed: %s", e)
|
||||
|
||||
# Fallback: compute from menu data
|
||||
menu = _menu_cache or _load_menu_cache_from_file()
|
||||
if menu and menu.get("categories"):
|
||||
items = [i for c in menu["categories"] for i in c.get("items", [])]
|
||||
active = sum(1 for i in items if i.get("active"))
|
||||
total = len(items)
|
||||
return {
|
||||
"health_percent": (active / max(total, 1)) * 100,
|
||||
"healthy": active,
|
||||
"degraded": 0,
|
||||
"down": total - active,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
return {"health_percent": 0, "healthy": 0, "degraded": 0, "down": 0, "total": 0}
|
||||
|
||||
|
||||
@router.get("/module-health/status")
|
||||
async def module_health_status(user=Depends(require_jwt)):
|
||||
"""Get detailed module health status."""
|
||||
try:
|
||||
if MODULE_HEALTH_CACHE.exists():
|
||||
return json.loads(MODULE_HEALTH_CACHE.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return {"modules": {}, "timestamp": time.time()}
|
||||
|
||||
|
||||
@router.get("/module-health/alerts")
|
||||
async def module_health_alerts(user=Depends(require_jwt)):
|
||||
"""Get modules that are degraded or down."""
|
||||
try:
|
||||
if MODULE_HEALTH_CACHE.exists():
|
||||
data = json.loads(MODULE_HEALTH_CACHE.read_text())
|
||||
modules = data.get("modules", {})
|
||||
alerts = [
|
||||
{"name": name, "status": m.get("overall"), "message": m.get("message", "")}
|
||||
for name, m in modules.items()
|
||||
if m.get("overall") in ("degraded", "down", "error")
|
||||
]
|
||||
return {"alerts": alerts}
|
||||
except Exception:
|
||||
pass
|
||||
return {"alerts": []}
|
||||
|
||||
|
||||
@router.get("/health-monitor/summary")
|
||||
async def vhost_health_summary(user=Depends(require_jwt)):
|
||||
"""Get VHost health summary (ok/slow/placeholder/down counts)."""
|
||||
try:
|
||||
if VHOST_HEALTH_CACHE.exists():
|
||||
data = json.loads(VHOST_HEALTH_CACHE.read_text())
|
||||
# Use pre-computed values if available
|
||||
if "ok" in data:
|
||||
return {
|
||||
"health_percent": data.get("health_pct", 0),
|
||||
"ok": data.get("ok", 0),
|
||||
"slow": data.get("slow", 0),
|
||||
"placeholder": data.get("placeholder", 0),
|
||||
"down": data.get("down", 0),
|
||||
"total": data.get("total", 0),
|
||||
}
|
||||
# Fallback: compute from vhosts dict
|
||||
vhosts = data.get("vhosts", {})
|
||||
ok = sum(1 for v in vhosts.values() if v.get("status") == "ok")
|
||||
slow = sum(1 for v in vhosts.values() if v.get("status") == "slow")
|
||||
placeholder = sum(1 for v in vhosts.values() if v.get("status") == "placeholder")
|
||||
down = sum(1 for v in vhosts.values() if v.get("status") in ("down", "error", "timeout"))
|
||||
total = len(vhosts)
|
||||
real_total = total - placeholder
|
||||
return {
|
||||
"health_percent": (ok + slow) / max(real_total, 1) * 100,
|
||||
"ok": ok,
|
||||
"slow": slow,
|
||||
"placeholder": placeholder,
|
||||
"down": down,
|
||||
"total": total,
|
||||
}
|
||||
except Exception as e:
|
||||
log.debug("VHost health cache read failed: %s", e)
|
||||
return {"health_percent": 0, "ok": 0, "slow": 0, "placeholder": 0, "down": 0, "total": 0}
|
||||
|
||||
|
||||
@router.get("/health-monitor/status")
|
||||
async def vhost_health_status(user=Depends(require_jwt)):
|
||||
"""Get detailed VHost health status."""
|
||||
try:
|
||||
if VHOST_HEALTH_CACHE.exists():
|
||||
return json.loads(VHOST_HEALTH_CACHE.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return {"vhosts": {}, "timestamp": time.time()}
|
||||
|
||||
|
||||
@router.get("/health-monitor/alerts")
|
||||
async def vhost_health_alerts(user=Depends(require_jwt)):
|
||||
"""Get VHosts that are down or slow."""
|
||||
try:
|
||||
if VHOST_HEALTH_CACHE.exists():
|
||||
data = json.loads(VHOST_HEALTH_CACHE.read_text())
|
||||
vhosts = data.get("vhosts", {})
|
||||
alerts = [
|
||||
{"domain": domain, "status": v.get("status"), "response_time": v.get("response_time", 0)}
|
||||
for domain, v in vhosts.items()
|
||||
if v.get("status") in ("slow", "down", "error", "timeout")
|
||||
]
|
||||
return {"alerts": alerts[:20]} # Limit to 20
|
||||
except Exception:
|
||||
pass
|
||||
return {"alerts": []}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
# Network Mode Selection (integrates with secubox-netmodes)
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
|
|
|
|||
|
|
@ -480,6 +480,39 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Health Monitor Widgets (Upper) -->
|
||||
<div class="health-row" style="display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin-bottom: 1.5rem;">
|
||||
<div class="card" style="margin-bottom: 0;">
|
||||
<div class="card-header">
|
||||
<h2>Module Health</h2>
|
||||
<span id="module-health-pct" style="font-family: var(--font-mono); color: var(--root-main);">-%</span>
|
||||
</div>
|
||||
<div class="health-bar" style="height: 8px; background: var(--border); border-radius: 4px; margin-bottom: 0.75rem;">
|
||||
<div id="module-health-bar" class="fill" style="width: 0%; height: 100%; background: var(--root-main); border-radius: 4px; transition: width 0.3s;"></div>
|
||||
</div>
|
||||
<div style="display: flex; gap: 1rem; font-size: 0.9rem;">
|
||||
<span id="module-health-ok" style="color: var(--root-main);">🟢 0</span>
|
||||
<span id="module-health-warn" style="color: var(--wall-main);">🟡 0</span>
|
||||
<span id="module-health-down" style="color: var(--boot-main);">🔴 0</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card" style="margin-bottom: 0;">
|
||||
<div class="card-header">
|
||||
<h2>VHost Health</h2>
|
||||
<span id="vhost-health-pct" style="font-family: var(--font-mono); color: var(--mesh-main);">-%</span>
|
||||
</div>
|
||||
<div class="health-bar" style="height: 8px; background: var(--border); border-radius: 4px; margin-bottom: 0.75rem;">
|
||||
<div id="vhost-health-bar" class="fill" style="width: 0%; height: 100%; background: var(--mesh-main); border-radius: 4px; transition: width 0.3s;"></div>
|
||||
</div>
|
||||
<div style="display: flex; gap: 1rem; font-size: 0.9rem;">
|
||||
<span id="vhost-health-ok" style="color: var(--root-main);">🟢 0</span>
|
||||
<span id="vhost-health-slow" style="color: var(--wall-main);">🟡 0</span>
|
||||
<span id="vhost-health-down" style="color: var(--boot-main);">🔴 0</span>
|
||||
<span id="vhost-health-placeholder" style="color: var(--muted);">⬜ 0</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">
|
||||
<div>
|
||||
<div class="card">
|
||||
|
|
@ -725,7 +758,7 @@
|
|||
const tbody = document.getElementById('modules-table');
|
||||
if (menuData && menuData.categories) {
|
||||
const allItems = menuData.categories.flatMap(c => c.items);
|
||||
tbody.innerHTML = allItems.slice(0, 10).map(item => {
|
||||
tbody.innerHTML = allItems.slice(0, 4).map(item => {
|
||||
// Look up version from modules data if available
|
||||
const modData = modules[item.id] || {};
|
||||
const version = modData.version || '-';
|
||||
|
|
@ -756,6 +789,49 @@
|
|||
}
|
||||
}
|
||||
|
||||
async function loadModuleHealth() {
|
||||
try {
|
||||
const data = await api('/module-health/summary');
|
||||
if (data && data.health_percent !== undefined) {
|
||||
const pct = Math.round(data.health_percent);
|
||||
document.getElementById('module-health-pct').textContent = pct + '%';
|
||||
document.getElementById('module-health-bar').style.width = pct + '%';
|
||||
document.getElementById('module-health-ok').textContent = '🟢 ' + (data.healthy || 0);
|
||||
document.getElementById('module-health-warn').textContent = '🟡 ' + (data.degraded || 0);
|
||||
document.getElementById('module-health-down').textContent = '🔴 ' + (data.down || 0);
|
||||
}
|
||||
} catch (e) {
|
||||
// Fallback: compute from menu data
|
||||
if (menuData && menuData.categories) {
|
||||
const allItems = menuData.categories.flatMap(c => c.items);
|
||||
const active = allItems.filter(i => i.active).length;
|
||||
const total = allItems.length;
|
||||
const pct = total > 0 ? Math.round((active / total) * 100) : 0;
|
||||
document.getElementById('module-health-pct').textContent = pct + '%';
|
||||
document.getElementById('module-health-bar').style.width = pct + '%';
|
||||
document.getElementById('module-health-ok').textContent = '🟢 ' + active;
|
||||
document.getElementById('module-health-down').textContent = '🔴 ' + (total - active);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function loadVHostHealth() {
|
||||
try {
|
||||
const data = await api('/health-monitor/summary');
|
||||
if (data && data.health_percent !== undefined) {
|
||||
const pct = Math.round(data.health_percent);
|
||||
document.getElementById('vhost-health-pct').textContent = pct + '%';
|
||||
document.getElementById('vhost-health-bar').style.width = pct + '%';
|
||||
document.getElementById('vhost-health-ok').textContent = '🟢 ' + (data.ok || 0);
|
||||
document.getElementById('vhost-health-slow').textContent = '🟡 ' + (data.slow || 0);
|
||||
document.getElementById('vhost-health-down').textContent = '🔴 ' + (data.down || 0);
|
||||
document.getElementById('vhost-health-placeholder').textContent = '⬜ ' + (data.placeholder || 0);
|
||||
}
|
||||
} catch (e) {
|
||||
document.getElementById('vhost-health-pct').textContent = 'N/A';
|
||||
}
|
||||
}
|
||||
|
||||
async function loadAlerts() {
|
||||
const data = await api('/alerts');
|
||||
const alerts = data || [];
|
||||
|
|
@ -763,7 +839,7 @@
|
|||
|
||||
const list = document.getElementById('alerts-list');
|
||||
if (Array.isArray(alerts) && alerts.length > 0) {
|
||||
list.innerHTML = alerts.slice(0, 5).map(a => `
|
||||
list.innerHTML = alerts.slice(0, 3).map(a => `
|
||||
<div class="alert-item">
|
||||
<div class="time">${new Date().toLocaleTimeString()}</div>
|
||||
<div class="message">${a.type?.replace(/_/g, ' ') || 'Alert'}: ${a.module || ''}</div>
|
||||
|
|
@ -934,6 +1010,8 @@
|
|||
await loadMenu();
|
||||
loadDashboard();
|
||||
loadHealth();
|
||||
loadModuleHealth();
|
||||
loadVHostHealth();
|
||||
loadAlerts();
|
||||
loadNetwork();
|
||||
loadMemory();
|
||||
|
|
|
|||
|
|
@ -505,7 +505,7 @@
|
|||
const list = document.getElementById('services-list');
|
||||
const services = data.services || [];
|
||||
|
||||
list.innerHTML = services.slice(0, 15).map(s => `
|
||||
list.innerHTML = services.slice(0, 4).map(s => `
|
||||
<div class="service-item">
|
||||
<span><span class="status-dot ${s.active ? 'running' : 'stopped'}"></span>${s.name}</span>
|
||||
<button class="btn ${s.active ? 'danger' : 'success'}" onclick="toggleService('${s.name}', ${!s.active})">${s.active ? 'Stop' : 'Start'}</button>
|
||||
|
|
@ -518,7 +518,7 @@
|
|||
const info = document.getElementById('network-info');
|
||||
const interfaces = data.interfaces || [];
|
||||
|
||||
info.innerHTML = interfaces.map(i => `
|
||||
info.innerHTML = interfaces.slice(0, 4).map(i => `
|
||||
<div class="info-row">
|
||||
<span class="label">${i.name}</span>
|
||||
<span style="color: ${i.up ? 'var(--green)' : 'var(--red)'};">${i.addresses?.[0] || 'no IP'} ${i.up ? '▲' : '▼'}</span>
|
||||
|
|
|
|||
447
scripts/haproxy-workflow.sh
Executable file
447
scripts/haproxy-workflow.sh
Executable file
|
|
@ -0,0 +1,447 @@
|
|||
#!/bin/bash
|
||||
# haproxy-workflow.sh - HAProxy auto-rehealth, WAF sync, and certificate management
|
||||
# SecuBox-DEB :: Infrastructure Workflow Automation
|
||||
# CyberMind — Gérald Kerma
|
||||
set -euo pipefail
|
||||
|
||||
VERSION="1.0.0"
|
||||
SCRIPT_NAME=$(basename "$0")
|
||||
|
||||
# Paths
|
||||
HAPROXY_CFG="/etc/haproxy/haproxy.cfg"
|
||||
HAPROXY_TOML="/etc/secubox/haproxy.toml"
|
||||
MITMPROXY_TOML="/etc/secubox/mitmproxy.toml"
|
||||
ROUTES_JSON="/srv/mitmproxy-waf/data/routes.json"
|
||||
HEALTH_CACHE="/var/cache/secubox/health/status.json"
|
||||
CERTS_DIR="/etc/haproxy/certs"
|
||||
ACME_DIR="/etc/acme"
|
||||
VHOST_ROUTES="/var/lib/secubox/haproxy/vhost-routes.json"
|
||||
LOG_FILE="/var/log/secubox/haproxy-workflow.log"
|
||||
|
||||
# API endpoints (via Unix sockets)
|
||||
HAPROXY_SOCK="/run/secubox/haproxy.sock"
|
||||
MITMPROXY_SOCK="/run/secubox/mitmproxy.sock"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[0;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Logging
|
||||
log() { echo -e "${BLUE}[INFO]${NC} $*"; echo "[$(date '+%Y-%m-%d %H:%M:%S')] INFO: $*" >> "$LOG_FILE" 2>/dev/null || true; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; echo "[$(date '+%Y-%m-%d %H:%M:%S')] WARN: $*" >> "$LOG_FILE" 2>/dev/null || true; }
|
||||
error() { echo -e "${RED}[ERROR]${NC} $*" >&2; echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: $*" >> "$LOG_FILE" 2>/dev/null || true; }
|
||||
success() { echo -e "${GREEN}[OK]${NC} $*"; }
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# HELPER FUNCTIONS
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
# API call via curl with JWT
|
||||
api_call() {
|
||||
local socket="$1"
|
||||
local method="$2"
|
||||
local endpoint="$3"
|
||||
local data="${4:-}"
|
||||
|
||||
local jwt=""
|
||||
if [ -f /etc/secubox/jwt/token ]; then
|
||||
jwt=$(cat /etc/secubox/jwt/token)
|
||||
fi
|
||||
|
||||
local curl_args=(
|
||||
--silent
|
||||
--unix-socket "$socket"
|
||||
-X "$method"
|
||||
-H "Content-Type: application/json"
|
||||
)
|
||||
|
||||
[ -n "$jwt" ] && curl_args+=(-H "Authorization: Bearer $jwt")
|
||||
[ -n "$data" ] && curl_args+=(-d "$data")
|
||||
|
||||
curl "${curl_args[@]}" "http://localhost$endpoint" 2>/dev/null || echo '{"error": "API call failed"}'
|
||||
}
|
||||
|
||||
# Check if service is running
|
||||
service_running() {
|
||||
systemctl is-active --quiet "$1" 2>/dev/null
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# 1. AUTO REHEALTH HAPROXY
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
cmd_rehealth() {
|
||||
log "=== Auto Rehealth HAProxy ==="
|
||||
|
||||
# Step 1: Check HAProxy is running
|
||||
if service_running haproxy; then
|
||||
success "HAProxy service is active"
|
||||
else
|
||||
warn "HAProxy service not running, attempting start..."
|
||||
sudo systemctl start haproxy || { error "Failed to start HAProxy"; return 1; }
|
||||
sleep 2
|
||||
fi
|
||||
|
||||
# Step 2: Validate config
|
||||
log "Validating HAProxy configuration..."
|
||||
if haproxy -c -f "$HAPROXY_CFG" 2>/dev/null; then
|
||||
success "Configuration valid"
|
||||
else
|
||||
error "Configuration invalid, running detailed check..."
|
||||
haproxy -c -f "$HAPROXY_CFG" 2>&1 | head -20
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Step 3: Reload HAProxy to pick up any changes
|
||||
log "Reloading HAProxy..."
|
||||
sudo systemctl reload haproxy && success "HAProxy reloaded"
|
||||
|
||||
# Step 4: Clear health cache to force re-probe
|
||||
if [ -f "$HEALTH_CACHE" ]; then
|
||||
log "Invalidating health cache..."
|
||||
sudo rm -f "$HEALTH_CACHE"
|
||||
success "Health cache cleared"
|
||||
fi
|
||||
|
||||
# Step 5: Restart health prober if running
|
||||
if service_running secubox-health-prober; then
|
||||
log "Restarting health prober..."
|
||||
sudo systemctl restart secubox-health-prober
|
||||
success "Health prober restarted"
|
||||
fi
|
||||
|
||||
# Step 6: Trigger API cache refresh (if API running)
|
||||
if [ -S "$HAPROXY_SOCK" ]; then
|
||||
log "Triggering API cache refresh..."
|
||||
api_call "$HAPROXY_SOCK" GET "/api/v1/haproxy/status" >/dev/null
|
||||
success "API cache refreshed"
|
||||
fi
|
||||
|
||||
# Step 7: Show current status
|
||||
log "Current HAProxy status:"
|
||||
echo "show info" | socat stdio /run/haproxy/admin.sock 2>/dev/null | head -20 || \
|
||||
echo "Stats socket not available (native HAProxy check)"
|
||||
|
||||
# Quick backend check
|
||||
if [ -S /run/haproxy/admin.sock ]; then
|
||||
echo ""
|
||||
log "Backend status:"
|
||||
echo "show stat" | socat stdio /run/haproxy/admin.sock 2>/dev/null | \
|
||||
awk -F, '/BACKEND/ {print " " $1 ": " $18}' | head -10
|
||||
fi
|
||||
|
||||
success "HAProxy rehealth completed"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# 2. WORKFLOW BACKEND THROUGH WAF
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
cmd_waf_sync() {
|
||||
log "=== Workflow Backend Through WAF ==="
|
||||
|
||||
# Step 1: Check mitmproxy-waf container/service
|
||||
local waf_running=false
|
||||
if lxc-info -n mitmproxy-waf -s 2>/dev/null | grep -q "RUNNING"; then
|
||||
success "mitmproxy-waf LXC container is running"
|
||||
waf_running=true
|
||||
elif service_running mitmproxy; then
|
||||
success "mitmproxy service is running"
|
||||
waf_running=true
|
||||
else
|
||||
warn "WAF service not running, attempting start..."
|
||||
if [ -f /var/lib/lxc/mitmproxy-waf/config ]; then
|
||||
sudo lxc-start -n mitmproxy-waf || warn "LXC start failed"
|
||||
else
|
||||
sudo systemctl start mitmproxy 2>/dev/null || warn "mitmproxy service start failed"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Step 2: Parse HAProxy config to extract vhosts and backends (using awk for speed)
|
||||
log "Extracting vhost→backend mappings from HAProxy config..."
|
||||
|
||||
local routes="{}"
|
||||
local count=0
|
||||
local backend_count=0
|
||||
|
||||
if [ -f "$HAPROXY_CFG" ]; then
|
||||
# Use awk for fast parsing of large config
|
||||
routes=$(awk '
|
||||
BEGIN { first=1 }
|
||||
# Capture backend -> server mappings
|
||||
/^backend / { backend=$2 }
|
||||
/^[[:space:]]+server / {
|
||||
for (i=1; i<=NF; i++) {
|
||||
if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+$/) {
|
||||
split($i, addr, ":")
|
||||
if (backend) servers[backend] = addr[1] ":" addr[2]
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
# Capture ACL -> hostname mappings
|
||||
/^[[:space:]]+acl .* hdr\(host\) -i / {
|
||||
acl=$2
|
||||
hostname=$NF
|
||||
acls[acl] = hostname
|
||||
}
|
||||
# Capture use_backend -> ACL mappings
|
||||
/^[[:space:]]+use_backend .* if / {
|
||||
be=$2
|
||||
aclname=$4
|
||||
if (be !~ /waf_inspector|mitmproxy_inspector/ && acls[aclname] && servers[be]) {
|
||||
split(servers[be], a, ":")
|
||||
if (!first) printf ", "
|
||||
printf "\"%s\": [\"%s\", %s]", acls[aclname], a[1], a[2]
|
||||
first=0
|
||||
}
|
||||
}
|
||||
' "$HAPROXY_CFG")
|
||||
routes="{${routes}}"
|
||||
count=$(echo "$routes" | grep -o '":\[' | wc -l)
|
||||
backend_count=$(awk '/^backend / {c++} END {print c+0}' "$HAPROXY_CFG")
|
||||
fi
|
||||
|
||||
log "Found $count vhost→backend mappings"
|
||||
|
||||
# Step 3: Write routes to mitmproxy
|
||||
if [ $count -gt 0 ]; then
|
||||
log "Writing routes to WAF..."
|
||||
sudo mkdir -p "$(dirname $ROUTES_JSON)"
|
||||
echo "$routes" | jq '.' | sudo tee "$ROUTES_JSON" >/dev/null
|
||||
success "Routes written to $ROUTES_JSON"
|
||||
|
||||
# Also update the HAProxy-side routes file
|
||||
echo "$routes" | jq '.' | sudo tee "$VHOST_ROUTES" >/dev/null 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Step 4: Verify WAF backend exists in HAProxy config
|
||||
log "Checking WAF backend in HAProxy config..."
|
||||
if grep -q "backend mitmproxy_inspector\|backend waf_inspector" "$HAPROXY_CFG"; then
|
||||
success "WAF backend exists in HAProxy config"
|
||||
else
|
||||
warn "WAF backend not found, may need to regenerate config"
|
||||
echo " Run: haproxyctl generate"
|
||||
fi
|
||||
|
||||
# Step 5: Check WAF routing is active
|
||||
local waf_routes=$(grep -c "use_backend.*waf_inspector\|use_backend.*mitmproxy_inspector" "$HAPROXY_CFG" 2>/dev/null || echo "0")
|
||||
log "HAProxy routes through WAF: $waf_routes vhosts"
|
||||
|
||||
# Step 6: Reload mitmproxy to pick up routes
|
||||
if [ "$waf_running" = true ]; then
|
||||
log "Signaling mitmproxy to reload routes..."
|
||||
if lxc-info -n mitmproxy-waf -s 2>/dev/null | grep -q "RUNNING"; then
|
||||
sudo lxc-attach -n mitmproxy-waf -- pkill -HUP mitmproxy 2>/dev/null || true
|
||||
else
|
||||
sudo systemctl reload mitmproxy 2>/dev/null || true
|
||||
fi
|
||||
success "WAF routes synced"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
backend_count=${#backends[@]}
|
||||
echo ""
|
||||
log "WAF Sync Summary:"
|
||||
echo " Backends extracted: $backend_count"
|
||||
echo " Routes synced: $count"
|
||||
echo " Routes through WAF: $waf_routes"
|
||||
|
||||
success "WAF sync completed"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# 3. VHOST CERTIFICATES - FULL CERTS PUBLISHED
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
cmd_certs() {
|
||||
log "=== VHost Full Certificates Status ==="
|
||||
|
||||
local total=0
|
||||
local valid=0
|
||||
local expiring=0
|
||||
local missing=0
|
||||
local expired=0
|
||||
|
||||
# Collect all vhosts from HAProxy
|
||||
local vhosts=()
|
||||
if [ -f "$HAPROXY_CFG" ]; then
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ hdr\(host\)[[:space:]]+-i[[:space:]]+([^[:space:]\}]+) ]]; then
|
||||
vhosts+=("${BASH_REMATCH[1]}")
|
||||
fi
|
||||
done < "$HAPROXY_CFG"
|
||||
fi
|
||||
|
||||
total=${#vhosts[@]}
|
||||
log "Total vhosts configured: $total"
|
||||
|
||||
if [ $total -eq 0 ]; then
|
||||
warn "No vhosts found in HAProxy config"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo ""
|
||||
printf "%-40s %-12s %-10s %s\n" "DOMAIN" "STATUS" "EXPIRES" "PATH"
|
||||
printf "%s\n" "$(printf '%.0s-' {1..100})"
|
||||
|
||||
for domain in "${vhosts[@]}"; do
|
||||
local cert_path=""
|
||||
local status=""
|
||||
local expires=""
|
||||
|
||||
# Check HAProxy certs directory
|
||||
if [ -f "$CERTS_DIR/$domain.pem" ]; then
|
||||
cert_path="$CERTS_DIR/$domain.pem"
|
||||
elif [ -f "$CERTS_DIR/${domain%%.*}.pem" ]; then
|
||||
cert_path="$CERTS_DIR/${domain%%.*}.pem"
|
||||
# Check ACME directory
|
||||
elif [ -f "$ACME_DIR/$domain/fullchain.cer" ]; then
|
||||
cert_path="$ACME_DIR/$domain/fullchain.cer"
|
||||
elif [ -f "$ACME_DIR/$domain/${domain}.cer" ]; then
|
||||
cert_path="$ACME_DIR/$domain/${domain}.cer"
|
||||
fi
|
||||
|
||||
if [ -n "$cert_path" ] && [ -f "$cert_path" ]; then
|
||||
# Check expiry
|
||||
local end_date
|
||||
end_date=$(openssl x509 -in "$cert_path" -enddate -noout 2>/dev/null | cut -d= -f2)
|
||||
|
||||
if [ -n "$end_date" ]; then
|
||||
local end_epoch
|
||||
end_epoch=$(date -d "$end_date" +%s 2>/dev/null || echo 0)
|
||||
local now_epoch
|
||||
now_epoch=$(date +%s)
|
||||
local days_left=$(( (end_epoch - now_epoch) / 86400 ))
|
||||
|
||||
if [ $days_left -lt 0 ]; then
|
||||
status="${RED}EXPIRED${NC}"
|
||||
expires="$days_left days"
|
||||
((expired++))
|
||||
elif [ $days_left -lt 30 ]; then
|
||||
status="${YELLOW}EXPIRING${NC}"
|
||||
expires="${days_left}d"
|
||||
((expiring++))
|
||||
else
|
||||
status="${GREEN}VALID${NC}"
|
||||
expires="${days_left}d"
|
||||
((valid++))
|
||||
fi
|
||||
else
|
||||
status="${YELLOW}UNKNOWN${NC}"
|
||||
expires="?"
|
||||
((valid++)) # Assume valid if can't parse
|
||||
fi
|
||||
else
|
||||
status="${RED}MISSING${NC}"
|
||||
expires="-"
|
||||
cert_path="(not found)"
|
||||
((missing++))
|
||||
fi
|
||||
|
||||
printf "%-40s " "$domain"
|
||||
echo -e "${status} $(printf '%-10s' "$expires") ${cert_path:0:40}"
|
||||
done
|
||||
|
||||
echo ""
|
||||
log "Certificate Summary:"
|
||||
echo -e " ${GREEN}Valid${NC}: $valid"
|
||||
echo -e " ${YELLOW}Expiring${NC}: $expiring (< 30 days)"
|
||||
echo -e " ${RED}Expired${NC}: $expired"
|
||||
echo -e " ${RED}Missing${NC}: $missing"
|
||||
echo " ─────────────"
|
||||
echo " Total: $total"
|
||||
|
||||
# Recommendations
|
||||
if [ $missing -gt 0 ]; then
|
||||
echo ""
|
||||
warn "Missing certificates - run ACME for:"
|
||||
for domain in "${vhosts[@]}"; do
|
||||
local found=false
|
||||
[ -f "$CERTS_DIR/$domain.pem" ] && found=true
|
||||
[ -f "$ACME_DIR/$domain/fullchain.cer" ] && found=true
|
||||
[ "$found" = false ] && echo " acme.sh --issue -d $domain --webroot /var/www/acme"
|
||||
done
|
||||
fi
|
||||
|
||||
if [ $expiring -gt 0 ]; then
|
||||
echo ""
|
||||
warn "Certificates expiring soon - renew with:"
|
||||
echo " acme.sh --renew-all"
|
||||
fi
|
||||
|
||||
success "Certificate check completed"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# ALL: Run complete workflow
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
cmd_all() {
|
||||
log "=== Running Complete HAProxy Workflow ==="
|
||||
echo ""
|
||||
|
||||
cmd_rehealth
|
||||
echo ""
|
||||
|
||||
cmd_waf_sync
|
||||
echo ""
|
||||
|
||||
cmd_certs
|
||||
echo ""
|
||||
|
||||
success "Complete workflow finished"
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# USAGE
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
$SCRIPT_NAME v$VERSION - HAProxy Workflow Automation
|
||||
|
||||
Usage: $SCRIPT_NAME <command>
|
||||
|
||||
Commands:
|
||||
rehealth Auto rehealth HAProxy (reload, invalidate caches, restart probers)
|
||||
waf-sync Sync backend routes through WAF (mitmproxy)
|
||||
certs Check/list vhost certificates status
|
||||
all Run complete workflow (rehealth + waf-sync + certs)
|
||||
help Show this help
|
||||
|
||||
Examples:
|
||||
$SCRIPT_NAME rehealth # Refresh HAProxy health monitoring
|
||||
$SCRIPT_NAME waf-sync # Sync vhost→backend routes to WAF
|
||||
$SCRIPT_NAME certs # Check certificate status for all vhosts
|
||||
$SCRIPT_NAME all # Run everything
|
||||
|
||||
EOF
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
# MAIN
|
||||
# ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
main() {
|
||||
# Create log directory
|
||||
mkdir -p "$(dirname $LOG_FILE)" 2>/dev/null || true
|
||||
|
||||
case "${1:-}" in
|
||||
rehealth) cmd_rehealth ;;
|
||||
waf-sync) cmd_waf_sync ;;
|
||||
certs) cmd_certs ;;
|
||||
all) cmd_all ;;
|
||||
help|-h|--help) usage ;;
|
||||
*)
|
||||
[ -n "${1:-}" ] && error "Unknown command: $1"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Loading…
Reference in New Issue
Block a user