secubox-deb/tests
CyberMind 6aa5e38d7d
Mail Phase 2 — Rspamd migration (closes #153) (#160)
* test(mail): Phase 2 scaffolding — rspamd.sh + templates + routers + bats baseline (ref #153)

* feat(mail): rspamd.sh — install_rspamd + re-entry guard (ref #153)

* feat(mail): Rspamd config templates (9 .conf files) — DKIM/ARC/DMARC/greylist/ratelimit (ref #153)

* feat(mail): lib/mail/rspamd.sh — configure helpers + DKIM keygen + D9 purge gate (ref #153)

* feat(mail): Milestone C — rspamd_client + /rspamd/* router + legacy shims (ref #153)

- rspamd_client.py: httpx wrapper around the Rspamd HTTP controller
  (read /stat /history /graph; write /reload /learnspam /learnham).
- routers/rspamd.py: 11 new endpoints under /api/v1/mail/rspamd/*
  (status, history, scores, reload, learn-{spam,ham}, whitelist CRUD,
  dkim/{domain}/{status,keygen}).
- routers/legacy.py: 14 deprecation shims for the Phase 1 /dkim/*
  /spam/* /grey/* surface. Each emits X-Deprecated-Endpoint: rspamd
  and forwards to the Rspamd equivalent. Removed in v3.0.
- main.py: drops the 14 inline legacy handlers and mounts the two
  routers via app.include_router(). 62/62 Phase 1 endpoint pytest
  still passes.

* feat(mail): Milestone D — mailctl cmd_rspamd dispatcher (ref #153)

Adds rspamd subcommand to mailctl: install / start / stop / restart /
reload / status / dkim-keygen / dns-records / learn-spam / learn-ham /
purge-legacy. The install path calls lib/mail/rspamd.sh helpers in
order: install_rspamd → configure_rspamd_milter → configure_rspamd_controller
→ configure_rspamd_dkim → configure_rspamd_postfix_milter.

The purge-legacy verb requires Rspamd to respond healthy on :11334
before removing SA + OpenDKIM (D9 safety gate).

* feat(mail): Milestone E — install adds rspamd; mail.toml [mail.rspamd]; postinst secrets (ref #153)

- install_mail_packages: now installs rspamd + redis-server + enables
  postfix/dovecot/rspamd at LXC boot (Phase 1 follow-up). Drops Apache/
  Roundcube — they live in the roundcube LXC after the rev. 3 split.
- mail.toml: [mail.rspamd] section (greylist, bayes_autolearn,
  ratelimit_outbound=200/h/user, web_ui_host). horde_url added.
- debian/postinst: on upgrade <2.3, generate /etc/secubox/secrets/
  rspamd-controller.pw (32-byte random) and mkdir /data/volumes/mail/
  rspamd/{dkim,bayes,history,settings} chown 100110 (_rspamd in unpriv).

* feat(mail): Milestone F — rspamd-route-sync-patch deploy helper (ref #153)

Idempotent one-shot script run at deploy time:
1. Removes 10.100.0.{10,11,12} from sync-mitmproxy-routes.sh's
   DEAD_CONTAINER_IPS list so the periodic timer does not reroute
   the mail/horde/roundcube LXC routes to webui.
2. Adds rspamd.gk2.secubox.in → [10.100.0.10, 11334] to both the
   host's /srv/mitmproxy/haproxy-routes.json and the mitmproxy LXC
   copy, then restarts the mitmdump worker.

No source-side nginx vhost added: rspamd UI goes through the
existing HAProxy → mitmproxy_inspector → 10.100.0.10:11334 path.

* feat(mail): bump secubox-mail to 2.3.0 (Phase 2 Rspamd) (ref #153)

* test(mail): Phase 2 13-gate acceptance smoke (ref #153)

Run with: bash tests/scripts/test-mail-phase2-acceptance.sh root@admin.gk2.secubox.in

Gates 1-3 are source-side (parses, pytest, deb path coverage).
Gates 4-11 are board-side (Rspamd listening, Postfix milter wired,
DKIM key generated, modules loaded, web UI via WAF).
Gates 12-13 are post-cutover (legacy purged + Phase 1 regression
check — 5 secubox.in users intact, webmail still WAF-routed).

Every board call uses timeout — Phase 1 lesson, never raw pipes.

* fix(mail): debian/rules ships templates/rspamd/ (path-coverage fix, ref #153)

Phase 1 lesson redux: debian/rules silently misses files unless every
new source-tree subdir gets its explicit cp line. Adds the templates/
rspamd/ install line so the 9 local.d configs + the postfix milter
snippet land at /usr/lib/secubox/mail/templates/rspamd/.

* fix(mail): live-deploy fixes — _rspamd uid via lxc-attach + smoke gate 7/12 (ref #153)

Two issues surfaced during Phase 2 live deploy on admin.gk2.secubox.in:

1. configure_rspamd_controller hardcoded chown to host uid 100110, assuming
   _rspamd is uid 110 inside the LXC. On this Debian 12 image _rspamd is
   uid 107 — rspamd crash-looped with "Permission denied" on secrets.inc.
   Fix: chown via `lxc-attach -- chown _rspamd:_rspamd` so the kernel's
   idmap maps the right uid regardless of image package set. Mode 0640
   instead of 0600 so the worker (uid _rspamd) can read it even if launched
   from a slightly different group context.

2. Smoke gate 7 grepped for the literal string "rspamd" in the first 5
   lines of `rspamc stat`, but rspamc only prints "Results for command:
   stat" + counters. Fix: grep for an always-present stat marker
   ("Messages scanned" or "Pools allocated") with a wider head -10 window.

3. Smoke gate 12 ran `dpkg -l opendkim spamassassin` under `set -e` and
   silently aborted the smoke when the named packages were unknown (the
   success case after purge). Fix: append `|| true` so the grep below is
   the actual gate.

All 13 gates green on admin.gk2.secubox.in after these fixes + the manual
deploy steps (DKIM keygen via lxc-attach, bind-mount entries, HAProxy +
mitmproxy route map for rspamd.gk2.secubox.in).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(mail): rspamd lib + sbin write through live LXC (ref #153)

Phase 2 deploy on admin.gk2.secubox.in surfaced three brittleness points
that the live-deploy fixes patched ad-hoc:

1. configure_rspamd_milter wrote to ${LXC_BASE}/<container>/rootfs/etc/
   rspamd/local.d/ — but on this board the runtime LXC mounts a different
   rootfs (/data/lxc/<container>/rootfs/ per lxc.rootfs.path). Files written
   to the host-side guess landed in a stale shell the running LXC didn't
   see, and the first install's local.d was empty.

2. configure_rspamd_controller had the same path-guess issue plus
   hardcoded chown 100110:100110 (had to be patched in commit 637b2221
   because _rspamd is uid 107 on this image, not 110).

3. rspamd_keygen called rspamadm on the host PATH — rspamadm only exists
   inside the rspamd-installed LXC, so the function errored out on every
   install. The Phase 2 deploy worked around it manually.

This refactor makes all three write THROUGH the live LXC via lxc-attach:
- Each local.d template is streamed in with `lxc-attach -- tee` (kernel
  resolves the rootfs path; idmap applies to the resulting file uid).
- secrets.inc + worker-controller.inc likewise written via tee, then
  chown'd inside the LXC to _rspamd:_rspamd (kernel maps to the correct
  outside-LXC subuid regardless of image).
- rspamd_keygen takes a `container` arg, runs `rspamadm dkim_keygen`
  inside the LXC, writes the keypair to /etc/rspamd-keys/<domain>/ (the
  bind-mounted path; falls back to /var/lib/rspamd/dkim/ if the bind
  mount isn't active yet), and mirrors the DNS TXT to the host data dir
  for DNS-publishing tooling.
- mailctl's `dkim-keygen` subcommand now delegates to the lib function.
- rspamd-route-sync-patch.sh verifies each write (reads back + asserts
  the entry equals expected) and fails loudly on mismatch — the Phase 2
  deploy needed a manual second pass for the mitmproxy LXC copy.

Pre-conditions: configure_rspamd_milter and configure_rspamd_controller
now require the LXC to be RUNNING (added an explicit lxc-info guard).
This matches the cmd_rspamd install path which starts the LXC via
install_rspamd before configuring.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 08:21:23 +02:00
..
integration
scripts
test_license_headers.py
test_opad_schema.py
test_ssl_status.py