Merge pull request #907 from CyberMind-FR/fix/aggregator-watchdog-grace-and-wpad-port
Some checks failed
License Headers / check (push) Has been cancelled

fix: aggregator-watchdog startup grace + wpad-vhost :9080 (board-stability, backport gk2)
This commit is contained in:
CyberMind 2026-07-25 18:13:30 +02:00 committed by GitHub
commit f6bc3d5115
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 39 additions and 1 deletions

View File

@ -1,3 +1,13 @@
secubox-aggregator (0.2.4-1~bookworm1) bookworm; urgency=medium
* Watchdog: 480s startup grace before auto-heal restart. The aggregator's
in-process cold mount of ~110 modules takes minutes and RuntimeDirectoryPreserve
keeps the socket file, so the probe saw "present but unresponsive" and the
2-strike auto-heal restarted it before it could bind (self-inflicted loop →
board-wide 502). Skip the restart while MainPID uptime < SECUBOX_AGG_WD_GRACE.
-- Gerald KERMA <devel@cybermind.fr> Fri, 25 Jul 2026 12:00:00 +0200
secubox-aggregator (0.2.3-1~bookworm1) bookworm; urgency=medium secubox-aggregator (0.2.3-1~bookworm1) bookworm; urgency=medium
* #727 auto-heal watchdog: ship secubox-aggregator-watchdog.{sh,service,timer}. * #727 auto-heal watchdog: ship secubox-aggregator-watchdog.{sh,service,timer}.

View File

@ -40,6 +40,21 @@ n=$(( $(cat "$STATE" 2>/dev/null || echo 0) + 1 ))
echo "$n" > "$STATE" 2>/dev/null || true echo "$n" > "$STATE" 2>/dev/null || true
logger -t "$MODULE" "aggregator probe failed (code=$code, streak=$n/$FAIL_THRESHOLD)" logger -t "$MODULE" "aggregator probe failed (code=$code, streak=$n/$FAIL_THRESHOLD)"
# Startup grace: the aggregator mounts ~110 modules in-process on cold start
# (several minutes). RuntimeDirectoryPreserve keeps the socket FILE across
# restarts, so a still-starting aggregator looks "present but unresponsive" to
# the probe. Do NOT auto-heal while it is within the grace window, or we kill it
# before it can finish binding (self-inflicted restart loop).
GRACE="${SECUBOX_AGG_WD_GRACE:-480}"
_mainpid=$(systemctl show secubox-aggregator -p MainPID --value 2>/dev/null || echo 0)
if [ -n "$_mainpid" ] && [ "$_mainpid" != 0 ]; then
_up=$(ps -o etimes= -p "$_mainpid" 2>/dev/null | tr -d " ")
if [ -n "$_up" ] && [ "$_up" -lt "$GRACE" ]; then
logger -t "$MODULE" "aggregator up ${_up}s (<${GRACE}s grace) — still starting, not restarting"
exit 0
fi
fi
if [ "$n" -ge "$FAIL_THRESHOLD" ]; then if [ "$n" -ge "$FAIL_THRESHOLD" ]; then
logger -t "$MODULE" "restarting secubox-aggregator (auto-heal)" logger -t "$MODULE" "restarting secubox-aggregator (auto-heal)"
systemctl restart secubox-aggregator.service systemctl restart secubox-aggregator.service

View File

@ -1,3 +1,11 @@
secubox-proxypac (1.2.1-1~bookworm1) bookworm; urgency=medium
* wpad-vhost: listen on :9080 (internal) instead of :80. HAProxy fronts :80,
so `listen 80;` never bound and made every `nginx -s reload` abort on the
bind conflict — config changes silently stopped applying board-wide.
-- Gerald KERMA <devel@cybermind.fr> Fri, 25 Jul 2026 12:00:00 +0200
secubox-proxypac (1.2.0-1~bookworm1) bookworm; urgency=medium secubox-proxypac (1.2.0-1~bookworm1) bookworm; urgency=medium
* Package proxypac.toml (conffile) + proxypac-wpad ctl + scoped sudoers * Package proxypac.toml (conffile) + proxypac-wpad ctl + scoped sudoers

View File

@ -1,6 +1,11 @@
# secubox-proxypac — WPAD auto-discovery vhost (LAN + mesh only, never public) # secubox-proxypac — WPAD auto-discovery vhost (LAN + mesh only, never public)
server { server {
listen 80; # Listen on the internal SecuBox nginx port (HAProxy fronts :80/:443 and
# routes wpad.<domain> here by Host). A bare `listen 80;` collides with
# HAProxy's :80 and makes EVERY `nginx -s reload` abort on the bind, so
# config changes silently stop applying board-wide. See
# project_gk2_nginx_multimaster_blocker.
listen 9080;
server_name wpad.gk2.secubox.in; server_name wpad.gk2.secubox.in;
allow 192.168.0.0/16; allow 192.168.0.0/16;
allow 10.10.0.0/24; allow 10.10.0.0/24;