mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-28 21:17:36 +00:00
Health banner: live panel (visitor-origin + live-hosts + cert-status) (#98)
* docs(spec): Health banner live panel design (ref #92) Three public banner sections sharing one polling/CORS pipeline: - VisitorOrigin: nft set seen_src + GeoLite2-ASN.mmdb, threshold-gated rollup, raw IPs discarded before persistence - LiveHosts: HAProxy admin socket, 60 x 1-min ring buffer over req_tot deltas, hostname-heuristic frontend filter - CertStatus: scan /etc/letsencrypt/live + cryptography parse, classify valid / expiring_soon / expiring_critical / expired Each section fails independently; section hidden on enabled=false, empty entries, or fetch error. All three endpoints are unauthenticated, CORS-open, Cache-Control max-age=300. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(plan): Health banner live panel implementation plan (ref #92) 14-task TDD plan: tests scaffold -> config helpers -> three aggregators (visitor-origin / live-hosts / cert-status) -> FastAPI lifespan wiring -> nftables ruleset -> geoipupdate timer -> debian packaging -> banner v1.3.0 -> README + tracking docs -> full-suite verification + PR. Also reconciles spec with codebase conventions: service user is 'secubox' (not 'secubox-metrics'); config lives in /etc/secubox/secubox.conf, not a separate metrics.toml. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(metrics): Scaffold pytest layout for new aggregators (ref #92) Adds tests/__init__.py and conftest.py that wire packages/secubox-metrics/api and the repo-wide common/ onto sys.path so individual aggregator modules can be imported in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(core): Add visitor_origin / live_hosts / cert_status config helpers (ref #92) Three new section helpers in secubox_core.config that merge defaults with operator-supplied TOML overrides. Each section defaults to enabled=false so the live-panel aggregators stay quiet on systems that haven't opted in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(metrics): Use tmp_path fixture + drop unused import (ref #92) Switches the config-helper tests from a hard-coded /tmp path to pytest's tmp_path fixture, matching the pattern in packages/secubox-haproxy/tests/. Removes the now-unused 'from unittest.mock import patch' line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(metrics): VisitorOrigin aggregator (ref #92) Pure-Python aggregator that polls the nft seen_src set, resolves ASNs via GeoLite2 mmdb, and emits a threshold-gated top-N rollup. Private/loopback IPs are skipped at lookup time; raw IPs never leave the function scope; the threshold gate runs before persistence so the cache file never contains attributable counts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(metrics): VisitorOrigin code-review followups (ref #92) - mmdb auto-reopen on mtime change (Important): close + reopen when stat reports a different mtime; previously _mmdb_mtime was dead state. - _read_nft_set defensive parse (Important): guard against {elem: str} shapes to prevent a latent TypeError. - current() returns a defensive copy (Minor): no more by-reference leak of internal state. - Drop unused ip_address import and unused monkeypatch parameter (Minor). - Tighten tiebreak test to assert ASN order, not just count order (Minor). - Add test for mtime-based reopen. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(metrics): Pin VisitorOrigin error-path behaviour (ref #92) Regression tests for refresh_once: disabled config, missing mmdb, and nft subprocess failure must all yield a non-throwing degraded payload. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(metrics): LiveHosts aggregator with 60x1-min ring buffer (ref #92) Reads HAProxy admin socket via raw AF_UNIX, parses 'show stat' CSV, filters internal frontends (leading underscore or no dot), ring-buffers per-frontend deltas over 60 minutes, and emits a top-N hostname rollup. Counter-reset detection (cur < prev) yields a fresh-baseline bucket instead of a negative delta. current() returns a defensive copy mirroring the VisitorOrigin fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(metrics): Pin LiveHosts CSV parser + missing-socket paths (ref #92) Tests the show-stat CSV parser against the real HAProxy column order and asserts that an absent admin socket returns a degraded payload rather than raising. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(metrics): CertStatus aggregator (ref #92) Scans /etc/letsencrypt/live for cert.pem files, classifies each by days remaining (valid / expiring_soon / expiring_critical / expired) using the operator's warn_days/critical_days thresholds, and emits a summary + soonest next-renewal host. A single corrupt PEM never kills the scan. Days remaining computed with math.ceil so a cert expiring in 2.99d reports 3d, consistent with certbot/renewal tooling expectations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(metrics): correct expired boundary when using math.ceil (ref #92) math.ceil maps any cert that expired within the last 24 h to days=0, which the previous `days < 0` guard treated as expiring_critical instead of expired. Changing the guard to `days <= 0` closes the gap: with ceil, days=0 means actual remaining time is in (-86400, 0] — i.e. already past or exactly at expiry — so classifying it as expired is correct. All four existing tests continue to pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(metrics): Wire three live-panel aggregators into FastAPI lifespan (ref #92) Adds the three asyncio background tasks under a single lifespan and exposes their current() payloads on /api/v1/metrics/{visitor-origin,live-hosts,cert-status} with a 5-min Cache-Control. Endpoints stay unauthenticated by design — the aggregators only emit threshold-gated, hostname-only data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(metrics): Await cancelled lifespan tasks + tidy import order (ref #92) - Important: lifespan finally now awaits gather(*tasks, return_exceptions=True) after cancel(), so blocking subprocess/socket I/O in aggregator refreshes doesn't race uvicorn's shutdown timeout. - Minor: move 'from contextlib import asynccontextmanager' into the stdlib import group at the top of the file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(metrics): Ship nftables ingress tap for visitor-origin (ref #92) Private inet secubox_metrics table with a timeout'd src-IP set, hooked from prerouting at priority -300 so additions happen before secubox-firewall's filter chain decides whether to drop the packet. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(metrics): Weekly GeoLite2 ASN refresh timer (ref #92) Conditional on /etc/secubox/secrets/maxmind.conf existing, so the unit is a silent no-op on installs that haven't supplied a license key. RandomizedDelay spreads load when many boxes deploy together. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build(metrics): Package live-panel deps, nft ruleset, and geoipupdate timer (ref #92) - control: add python3-maxminddb, python3-cryptography, geoipupdate, nftables - rules: install nftables/ and systemd/ assets - postinst: secubox -> haproxy group, cache + secrets + GeoIP dirs, nftables reload, timer enable - service: ReadWritePaths gains /var/cache/secubox Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(metrics): Run geoipupdate as secubox + restart svc on upgrade (ref #92) - secubox-geoipupdate.service: User/Group=secubox so .mmdb files inherit the ownership the metrics service expects when reading them. Previously the unit ran as root and created root-owned files that secubox-metrics could not open. - postinst: switch 'systemctl start' to 'systemctl restart' so the secubox user's new haproxy-group membership is picked up by an already-running service after an upgrade. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(banner): v1.3.0 live panel — visitor origin, live hosts, cert status (ref #92) Three independent fetch loops on a shared 30s cadence, three DOM sections, per-section hide on enabled=false / empty / fetch error. Uses existing design tokens (gold/cyan/matrix-green) so no new CSS variables are added. A failing section never affects the others. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(banner): Correct banner element id lookup in live-panel (ref #92) The new live-panel sectionContainer() helper looked up getElementById('sbx-health-banner'), but the actual banner element is created with id='health-banner'. The mismatch made banner null, so the three live-panel sections were silently never appended to the DOM. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: Session 160 — Health Banner Live Panel (ref #92) README documents the three new endpoints + config blocks. HISTORY / WIP / MIGRATION-MAP entries describe the feature, the spec/plan paths, and the session's outcome. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(metrics): Address whole-branch review findings (ref #92) - Wrap _read_nft_set / _read_haproxy_stats in asyncio.to_thread so blocking I/O (subprocess up to 5s, AF_UNIX recv up to 2s) no longer stalls the event loop on every refresh tick. - Replace falsy current() guard with explicit _refreshed flag. Previously, a successful refresh that produced entries=[] would fall through to the on-disk cache, serving stale non-empty data during low-traffic periods. - Move geoipupdate from Depends to Recommends. It lives in bookworm/contrib, so a hard dependency breaks 'apt install secubox-metrics' on systems without contrib enabled. The aggregator already degrades gracefully when the mmdb is absent, making Recommends the correct strength. README documents the contrib note. - prerm stops + disables secubox-geoipupdate.timer/service so 'apt remove' doesn't leave an orphan timer firing weekly with a missing unit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: CyberMind-FR <gandalf@Gk2.net> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
CyberMind
GitHub
parent
8152484c42
commit
2744758b9e
|
|
@ -4,6 +4,28 @@
|
|||
---
|
||||
## 2026-05-12
|
||||
|
||||
### Session 160 — Health Banner Live Panel (Issue #92)
|
||||
|
||||
**Goal:** Add three public sections to the health banner — VisitorOrigin,
|
||||
LiveHosts, CertStatus — sharing one polling pipeline.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-05-12-visitor-origin-feed-design.md`
|
||||
**Plan:** `docs/superpowers/plans/2026-05-12-visitor-origin-feed.md`
|
||||
**Branch:** `feature/92-health-banner-visitor-origin-feed-anonym`
|
||||
**Issue:** [#92](https://github.com/CyberMind-FR/secubox-deb/issues/92)
|
||||
|
||||
**Highlights**
|
||||
- nftables `inet secubox_metrics` table (priority -300) — independent of
|
||||
`secubox-firewall`.
|
||||
- VisitorOrigin: kernel-dedupe via timeout'd set, mmdb resolution, threshold
|
||||
gate before persistence (raw IPs never leave the function).
|
||||
- LiveHosts: HAProxy admin socket + 60x1-min ring buffer; counter-reset
|
||||
detection.
|
||||
- CertStatus: cryptography parse of `/etc/letsencrypt/live/*/cert.pem`.
|
||||
- Banner v1.3.0: three fail-isolated fetch loops on a shared 30 s cadence.
|
||||
|
||||
---
|
||||
|
||||
### Session 160 — secubox apt + clone: validate against live repo (Issue #89)
|
||||
|
||||
**Goal:** Audit the 2026-05-11 plan vs the implemented Go CLI; close any genuine gap; end-to-end against `https://apt.secubox.in/` (commissioned in Session 152 / Issue #80).
|
||||
|
|
|
|||
|
|
@ -149,6 +149,12 @@ Légende : ✅ Terminé · 🔄 En cours · ⬜ À faire · ⏸ Bloqué
|
|||
|
||||
---
|
||||
|
||||
## Addenda
|
||||
|
||||
**2026-05-12 — Issue #92 extension:** `secubox-metrics` gains three public live-panel endpoints (`visitor-origin`, `live-hosts`, `cert-status`) consumed by the health banner.
|
||||
|
||||
---
|
||||
|
||||
## Phases du projet
|
||||
|
||||
### Phase 1 — Hardware ✅
|
||||
|
|
|
|||
|
|
@ -3,6 +3,24 @@
|
|||
|
||||
---
|
||||
|
||||
## ✅ Session 160: Health Banner Live Panel (Issue #92)
|
||||
|
||||
### Objective
|
||||
Add three public banner sections (VisitorOrigin / LiveHosts / CertStatus)
|
||||
sharing one polling and CORS pipeline in `secubox-metrics`.
|
||||
|
||||
### Completed
|
||||
- Spec + plan written and committed.
|
||||
- Three aggregators with unit + error-path tests (25 tests passing).
|
||||
- nftables ruleset, systemd timer, postinst plumbing.
|
||||
- Banner v1.3.0 with three fail-isolated fetch loops.
|
||||
- README updated.
|
||||
|
||||
### Status
|
||||
PR pending review against `master`.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Session 160: secubox apt + clone — validate implementation (Issue #89)
|
||||
|
||||
### Objective
|
||||
|
|
|
|||
|
|
@ -105,3 +105,53 @@ def get_board_info() -> dict:
|
|||
"mem_total_mb": mem.get("MemTotal", 0) // 1024,
|
||||
"mem_free_mb": mem.get("MemAvailable", 0) // 1024,
|
||||
}
|
||||
|
||||
|
||||
# ── Live-panel helpers (issue #92) ─────────────────────────────────
|
||||
|
||||
_VISITOR_ORIGIN_DEFAULTS = {
|
||||
"enabled": False,
|
||||
"window_minutes": 60,
|
||||
"min_count": 5,
|
||||
"top_n": 5,
|
||||
"asn_db_path": "/var/lib/GeoIP/GeoLite2-ASN.mmdb",
|
||||
"nft_table": "secubox_metrics",
|
||||
"nft_set": "seen_src",
|
||||
"nft_family": "inet",
|
||||
}
|
||||
|
||||
_LIVE_HOSTS_DEFAULTS = {
|
||||
"enabled": False,
|
||||
"window_minutes": 60,
|
||||
"top_n": 5,
|
||||
"haproxy_socket": "/run/haproxy/admin.sock",
|
||||
"frontend_filter": "*",
|
||||
}
|
||||
|
||||
_CERT_STATUS_DEFAULTS = {
|
||||
"enabled": False,
|
||||
"letsencrypt_live_dir": "/etc/letsencrypt/live",
|
||||
"warn_days": 30,
|
||||
"critical_days": 7,
|
||||
}
|
||||
|
||||
|
||||
def _merged(defaults: dict, section: str) -> dict:
|
||||
out = dict(defaults)
|
||||
out.update(get_config(section))
|
||||
return out
|
||||
|
||||
|
||||
def get_visitor_origin_config() -> dict:
|
||||
"""Return [visitor_origin] merged with defaults."""
|
||||
return _merged(_VISITOR_ORIGIN_DEFAULTS, "visitor_origin")
|
||||
|
||||
|
||||
def get_live_hosts_config() -> dict:
|
||||
"""Return [live_hosts] merged with defaults."""
|
||||
return _merged(_LIVE_HOSTS_DEFAULTS, "live_hosts")
|
||||
|
||||
|
||||
def get_cert_status_config() -> dict:
|
||||
"""Return [cert_status] merged with defaults."""
|
||||
return _merged(_CERT_STATUS_DEFAULTS, "cert_status")
|
||||
|
|
|
|||
1993
docs/superpowers/plans/2026-05-12-visitor-origin-feed.md
Normal file
1993
docs/superpowers/plans/2026-05-12-visitor-origin-feed.md
Normal file
File diff suppressed because it is too large
Load Diff
432
docs/superpowers/specs/2026-05-12-visitor-origin-feed-design.md
Normal file
432
docs/superpowers/specs/2026-05-12-visitor-origin-feed-design.md
Normal file
|
|
@ -0,0 +1,432 @@
|
|||
# Design — Health Banner Live Panel
|
||||
|
||||
*Issue: [#92](https://github.com/CyberMind-FR/secubox-deb/issues/92)*
|
||||
*Date: 2026-05-12*
|
||||
*Author: Gérald Kerma (via Claude, Session 160)*
|
||||
|
||||
> **Scope note.** Originally filed as "visitor-origin feed only". User expanded
|
||||
> the scope mid-brainstorm to also include live-hosts and cert-status sections.
|
||||
> All three are designed together because they share the banner module, polling
|
||||
> pattern, design tokens, and CORS/cache plumbing. Splitting them later is
|
||||
> always possible, but each section is small enough that combining them is
|
||||
> cheaper than three round-trips of design + PR + review.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Add a public "live panel" to the SecuBox health banner — the floating widget
|
||||
injected on every SecuBox vhost — with three independent sections:
|
||||
|
||||
1. **VisitorOrigin** — top source ASNs hitting the box (anonymized, last 60 min).
|
||||
2. **LiveHosts** — top vhosts being served (by request rate, last 60 min).
|
||||
3. **CertStatus** — Let's Encrypt / ACME state rollup across vhosts.
|
||||
|
||||
All three sections are visible to anonymous viewers. Hostnames and cert
|
||||
metadata are already discoverable via DNS and the TLS handshake; exposing them
|
||||
in the banner adds no recon surface beyond what is already public. Raw client
|
||||
IPs are never exposed.
|
||||
|
||||
## 2. Non-goals
|
||||
|
||||
- Per-vhost rollup of visitor ASNs (single global rollup for v1).
|
||||
- Historical charts or sparklines (current 60 min snapshot only).
|
||||
- Country/city geolocation (ASN only).
|
||||
- Bot/scraper classification or threat scoring.
|
||||
- IPv6 source coverage for VisitorOrigin (follow-up).
|
||||
- Admin-gated views (everything in v1 is public; nothing exposes IPs).
|
||||
- Manual cert renewal trigger UI (CertStatus is read-only).
|
||||
|
||||
## 3. User-facing surface
|
||||
|
||||
The health banner gains one new panel containing three stacked sections, placed
|
||||
after the existing SSL/health indicators:
|
||||
|
||||
```text
|
||||
┌─ VisitorOrigin ──────────────────────────────────┐
|
||||
│ AS13335 Cloudflare ████████ 142 │
|
||||
│ AS16276 OVH SAS ████ 78 │
|
||||
│ AS3320 Deutsche Telekom ███ 54 │
|
||||
│ AS3215 Orange S.A. ██ 31 │
|
||||
│ AS5089 Virgin Media █ 12 │
|
||||
└─ last 60 min · top 5 ────────────────────────────┘
|
||||
|
||||
┌─ LiveHosts ──────────────────────────────────────┐
|
||||
│ secubox.in ████████ 412 req │
|
||||
│ apt.secubox.in █████ 214 req │
|
||||
│ hub.secubox.in ███ 138 req │
|
||||
│ auth.secubox.in ██ 67 req │
|
||||
│ p2p.secubox.in █ 22 req │
|
||||
└─ last 60 min · top 5 ────────────────────────────┘
|
||||
|
||||
┌─ CertStatus ─────────────────────────────────────┐
|
||||
│ ✅ 11 valid ⚠ 2 expiring <30d ✗ 0 failed │
|
||||
│ next renewal: auth.secubox.in · 14d │
|
||||
└──────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Each section is hidden independently when:
|
||||
|
||||
- Its data source is unavailable (e.g. `mmdb` missing for VisitorOrigin,
|
||||
HAProxy socket missing for LiveHosts, no certs found for CertStatus).
|
||||
- Its `entries` list is empty after filtering.
|
||||
- Its fetch fails for any reason.
|
||||
|
||||
The whole panel collapses gracefully — a single failing section never breaks
|
||||
the others.
|
||||
|
||||
## 4. Architecture overview
|
||||
|
||||
All three sections follow the same shape: a 60 s background asyncio task in
|
||||
`secubox-metrics` reads raw data, computes a sanitized rollup, persists it to
|
||||
`/var/cache/secubox/metrics/<section>.json`, and exposes it via a public,
|
||||
CORS-open, 5-minute-cached endpoint. The banner polls each endpoint on its
|
||||
own 30 s cycle.
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ secubox-metrics │
|
||||
│ │
|
||||
│ VisitorOriginAggregator ──▶ visitor-origin.json ──▶ /visitor-origin│
|
||||
│ LiveHostsAggregator ──▶ live-hosts.json ──▶ /live-hosts │
|
||||
│ CertStatusAggregator ──▶ cert-status.json ──▶ /cert-status │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
nft set seen_src HAProxy stats socket /etc/letsencrypt/live
|
||||
(kernel TTL'd) show stat -1 6 -1 + cryptography parse
|
||||
|
||||
Banner polls each endpoint independently every 30 s.
|
||||
```
|
||||
|
||||
## 5. Components in detail
|
||||
|
||||
### 5.1 nftables ruleset (`/etc/nftables.d/secubox-metrics.nft`)
|
||||
|
||||
Idempotent include shipped by the `secubox-metrics` package. Lives in its own
|
||||
table to stay clear of `secubox-firewall`'s ruleset.
|
||||
|
||||
```nft
|
||||
table inet secubox_metrics {
|
||||
set seen_src {
|
||||
type ipv4_addr
|
||||
flags timeout
|
||||
timeout 1h
|
||||
size 65536
|
||||
}
|
||||
|
||||
chain ingress_tap {
|
||||
type filter hook prerouting priority -300; policy accept;
|
||||
tcp dport { 80, 443 } add @seen_src { ip saddr }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Separate table → no entanglement with `secubox-firewall`.
|
||||
- `priority -300` runs before standard filter; set additions happen even if a
|
||||
later rule drops the packet.
|
||||
- `size 65536` caps memory; on overflow the kernel evicts oldest entries.
|
||||
- Per-packet cost on Armada 3720/7040: ~hundreds of nanoseconds. Negligible.
|
||||
|
||||
### 5.2 Config blocks (`/etc/secubox/secubox.conf`)
|
||||
|
||||
The codebase uses a single TOML config file `/etc/secubox/secubox.conf` read by
|
||||
`secubox_core.config.get_config(section)`. Three new sections are added:
|
||||
|
||||
```toml
|
||||
[visitor_origin]
|
||||
enabled = true
|
||||
window_minutes = 60
|
||||
min_count = 5
|
||||
top_n = 5
|
||||
asn_db_path = "/var/lib/GeoIP/GeoLite2-ASN.mmdb"
|
||||
nft_table = "secubox_metrics"
|
||||
nft_set = "seen_src"
|
||||
nft_family = "inet"
|
||||
|
||||
[live_hosts]
|
||||
enabled = true
|
||||
window_minutes = 60
|
||||
top_n = 5
|
||||
haproxy_socket = "/run/haproxy/admin.sock"
|
||||
# Frontends to count; "*" means all that show vhost-style names.
|
||||
frontend_filter = "*"
|
||||
|
||||
[cert_status]
|
||||
enabled = true
|
||||
letsencrypt_live_dir = "/etc/letsencrypt/live"
|
||||
warn_days = 30 # "expiring soon" threshold
|
||||
critical_days = 7 # surfaced as ⚠ vs ✗ tightening
|
||||
```
|
||||
|
||||
Parsed by `common/secubox_core/config.py` with sensible defaults when blocks
|
||||
are absent (each section defaults to `enabled=False`).
|
||||
|
||||
### 5.3 VisitorOrigin aggregator (`packages/secubox-metrics/api/visitor_origin.py`)
|
||||
|
||||
```python
|
||||
class VisitorOriginAggregator:
|
||||
def __init__(self, cfg: VisitorOriginConfig): ...
|
||||
async def run_forever(self) -> None: ...
|
||||
async def refresh_once(self) -> dict: ...
|
||||
def _read_nft_set(self) -> list[IPv4Address]: ... # subprocess nft -j
|
||||
def _lookup_asn(self, ip) -> tuple[int, str] | None: ... # maxminddb
|
||||
def _aggregate(self, ips) -> list[dict]: ... # threshold + sort
|
||||
def _persist(self, payload) -> None: ... # atomic write
|
||||
def current(self) -> dict: ... # sync read
|
||||
```
|
||||
|
||||
**Guarantees.** Raw IPs never leave the aggregator's local scope. The threshold
|
||||
gate (`count >= min_count`) runs *before* cache write, so the on-disk file
|
||||
cannot contain an ASN attributable to fewer than `min_count` distinct sources.
|
||||
|
||||
### 5.4 LiveHosts aggregator (`packages/secubox-metrics/api/live_hosts.py`)
|
||||
|
||||
Data source: HAProxy admin socket. The socket exposes per-frontend request
|
||||
counters (`req_tot`) that are monotonic since HAProxy start, so we sample at
|
||||
fixed intervals and ring-buffer 1 min deltas over 60 slots.
|
||||
|
||||
```python
|
||||
class LiveHostsAggregator:
|
||||
def __init__(self, cfg: LiveHostsConfig):
|
||||
self._buckets = collections.deque(maxlen=60) # 60 × 1 min
|
||||
self._prev_totals: dict[str, int] = {}
|
||||
async def run_forever(self) -> None: ...
|
||||
async def refresh_once(self) -> dict: ...
|
||||
def _read_haproxy_stats(self) -> dict[str, int]: ... # frontend → req_tot
|
||||
def _delta_and_buffer(self, totals: dict[str, int]) -> dict[str, int]: ...
|
||||
def _aggregate(self) -> list[dict]: ... # sum buckets, top_n
|
||||
def _persist(self, payload) -> None: ...
|
||||
def current(self) -> dict: ...
|
||||
```
|
||||
|
||||
**Why HAProxy socket, not log parsing?** The socket is structured CSV that
|
||||
HAProxy guarantees stable per major version; log parsing has to track format
|
||||
changes, sampling, rotation, and is heavier on the box. The socket is also
|
||||
already used by `haproxyctl`.
|
||||
|
||||
**Frontend filtering.** Only frontends whose name looks like a hostname (or
|
||||
matches `frontend_filter` when not `"*"`) are counted. Internal frontends like
|
||||
`stats-https` are filtered out to avoid noise. Heuristic: contains a `.`,
|
||||
doesn't start with `_`, isn't in a blocklist.
|
||||
|
||||
**Schema.**
|
||||
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"window_minutes": 60,
|
||||
"generated_at": "2026-05-12T14:23:00Z",
|
||||
"entries": [
|
||||
{"host": "secubox.in", "count": 412},
|
||||
{"host": "apt.secubox.in", "count": 214}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 5.5 CertStatus aggregator (`packages/secubox-metrics/api/cert_status.py`)
|
||||
|
||||
Reads every `*/cert.pem` under `/etc/letsencrypt/live/`, parses with
|
||||
`cryptography.x509.load_pem_x509_certificate`, extracts `not_valid_after`.
|
||||
|
||||
```python
|
||||
class CertStatusAggregator:
|
||||
def __init__(self, cfg: CertStatusConfig): ...
|
||||
async def run_forever(self) -> None: ...
|
||||
async def refresh_once(self) -> dict: ...
|
||||
def _scan_certs(self) -> list[CertInfo]: ...
|
||||
def _classify(self, cert: CertInfo, now: datetime) -> str: ...
|
||||
# → "valid" | "expiring_soon" | "expiring_critical" | "expired"
|
||||
def _summarize(self, infos) -> dict: ...
|
||||
def _persist(self, payload) -> None: ...
|
||||
def current(self) -> dict: ...
|
||||
```
|
||||
|
||||
**Schema.**
|
||||
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"generated_at": "2026-05-12T14:23:00Z",
|
||||
"summary": {
|
||||
"total": 13,
|
||||
"valid": 11,
|
||||
"expiring_soon": 2,
|
||||
"expiring_critical": 0,
|
||||
"expired": 0,
|
||||
"failed_renewal": 0
|
||||
},
|
||||
"next_renewal": {"host": "auth.secubox.in", "days": 14},
|
||||
"warnings": [
|
||||
{"host": "auth.secubox.in", "days": 14, "state": "expiring_soon"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Failed-renewal detection.** Optional v1 enhancement: tail
|
||||
`/var/log/letsencrypt/letsencrypt.log` for the last 7 days, count failures by
|
||||
host. If the log is unreadable, `failed_renewal=0` (degrade gracefully). This
|
||||
can ship as a follow-up if it complicates v1.
|
||||
|
||||
**Permissions.** `cert.pem` is world-readable by default on Debian; the
|
||||
aggregator runs as `secubox-metrics` and only reads. Private keys are never
|
||||
touched.
|
||||
|
||||
### 5.6 FastAPI endpoints
|
||||
|
||||
Three GET endpoints, all unauthenticated, all CORS-open, all `Cache-Control: public, max-age=300`:
|
||||
|
||||
```http
|
||||
GET /api/v1/metrics/visitor-origin
|
||||
GET /api/v1/metrics/live-hosts
|
||||
GET /api/v1/metrics/cert-status
|
||||
```
|
||||
|
||||
Each endpoint returns its aggregator's `current()` payload or
|
||||
`{"enabled": false, ...}` when the aggregator is disabled or has no data.
|
||||
|
||||
### 5.7 GeoIP database lifecycle (VisitorOrigin only)
|
||||
|
||||
- `apt install geoipupdate python3-maxminddb` (added to `debian/control`
|
||||
Depends).
|
||||
- License key in `/etc/secubox/secrets/maxmind.conf` (mode `0600`, owner
|
||||
`secubox`). Operator-supplied; absent ⇒ updater no-op (logged once).
|
||||
- `secubox-geoipupdate.timer` runs weekly, calling
|
||||
`secubox-geoipupdate.service` which exec's `geoipupdate -f /etc/secubox/secrets/maxmind.conf`.
|
||||
- Aggregator opens the mmdb read-only via `maxminddb.open_database(path, MODE_MMAP)`
|
||||
and reopens on mtime change.
|
||||
|
||||
### 5.8 Health-banner integration (`packages/secubox-hub/www/shared/health-banner.js`)
|
||||
|
||||
Bump `VERSION` to `'1.3.0'`. Add three independent fetch loops mirroring the
|
||||
existing `HealthCache` pattern. Each section:
|
||||
|
||||
- Has its own 30 s timer.
|
||||
- Has its own cache + last-error tracker.
|
||||
- Renders only when `enabled === true` and `entries.length > 0`
|
||||
(or, for cert-status, `summary.total > 0`).
|
||||
- A section's failure never affects the others.
|
||||
|
||||
```js
|
||||
const VISITOR_ORIGIN_API = window.SECUBOX_VISITOR_ORIGIN_API
|
||||
|| '/api/v1/metrics/visitor-origin';
|
||||
const LIVE_HOSTS_API = window.SECUBOX_LIVE_HOSTS_API
|
||||
|| '/api/v1/metrics/live-hosts';
|
||||
const CERT_STATUS_API = window.SECUBOX_CERT_STATUS_API
|
||||
|| '/api/v1/metrics/cert-status';
|
||||
|
||||
const LIVE_REFRESH_INTERVAL = 30000; // 30s, shared across the three
|
||||
```
|
||||
|
||||
Section rendering uses the existing design tokens (`design-tokens.css`)
|
||||
— no new colours, no new fonts. Bars use the `--gold-hermetic` accent for
|
||||
VisitorOrigin, `--cyber-cyan` for LiveHosts, `--matrix-green`/`--cinnabar`
|
||||
for CertStatus pass/fail icons.
|
||||
|
||||
## 6. Failure modes (catalogue)
|
||||
|
||||
| Trigger | Affected | Effect | UI |
|
||||
|------------------------------------------|---------------|----------------------------------------------|--------------------------|
|
||||
| `mmdb` missing | VisitorOrigin | `enabled=false` | Section hidden |
|
||||
| `mmdb` lookup raises | VisitorOrigin | IP skipped, others continue | Logged |
|
||||
| `nft` set missing | VisitorOrigin | `enabled=false`, logged once | Section hidden |
|
||||
| `nft -j` returns malformed JSON | VisitorOrigin | Refresh skipped, prior payload kept | Logged warning |
|
||||
| All ASNs below `min_count` | VisitorOrigin | `entries=[]` | Section hidden |
|
||||
| HAProxy socket missing/unreadable | LiveHosts | `enabled=false`, logged once | Section hidden |
|
||||
| HAProxy `show stat` parse fail | LiveHosts | Refresh skipped, prior payload kept | Logged warning |
|
||||
| All frontends below 1 req in 60 min | LiveHosts | `entries=[]` | Section hidden |
|
||||
| `/etc/letsencrypt/live` missing or empty | CertStatus | `enabled=false` | Section hidden |
|
||||
| Cert parse error on one host | CertStatus | That host skipped, others continue | Logged |
|
||||
| Endpoint 5xx | Any | Banner caches prior payload 5 min then hides | Section hidden |
|
||||
| Aggregator task dies | Any | Watchdog logs, asyncio restart | Stale data until restart |
|
||||
|
||||
## 7. Testing strategy
|
||||
|
||||
### 7.1 Unit tests
|
||||
|
||||
`packages/secubox-metrics/tests/test_visitor_origin.py`
|
||||
|
||||
- `_aggregate`: synthetic IPs → expected counts.
|
||||
- `_aggregate`: threshold edge — `min_count` kept, `min_count - 1` dropped.
|
||||
- `_aggregate`: top-N sort, ASN-numeric tiebreak.
|
||||
- `_lookup_asn`: private addresses → `None`; mmdb mock for known IPs.
|
||||
- `refresh_once`: missing mmdb / empty set / subprocess raise.
|
||||
- `_persist`: atomic write, `0644`, parent dir created.
|
||||
|
||||
`packages/secubox-metrics/tests/test_live_hosts.py`
|
||||
|
||||
- Ring-buffer accumulates 60 buckets, oldest evicted.
|
||||
- Delta computation handles HAProxy restart (counter reset) without negatives.
|
||||
- Frontend filter excludes internal names.
|
||||
- Top-N + tiebreak.
|
||||
- Missing socket / malformed CSV → `enabled=false` or prior payload.
|
||||
|
||||
`packages/secubox-metrics/tests/test_cert_status.py`
|
||||
|
||||
- Classifier: `now < not_valid_after - warn_days` → valid.
|
||||
- Classifier: critical/expired/expiring_soon boundaries.
|
||||
- Summary counts match fixtures.
|
||||
- Missing live dir → `enabled=false`.
|
||||
- Parse failure on one cert doesn't kill the scan.
|
||||
|
||||
### 7.2 Integration tests
|
||||
|
||||
- VisitorOrigin: populate `inet secubox_metrics seen_src` with `nft add element`
|
||||
for known public IPs, run `refresh_once()` against a fixture mmdb, assert
|
||||
endpoint schema.
|
||||
- LiveHosts: spawn a `socat` listener mimicking the HAProxy socket reply,
|
||||
drive two refreshes, assert deltas + top-N.
|
||||
- CertStatus: fixture directory with hand-crafted PEMs (valid / 5d / -1d),
|
||||
assert summary + `next_renewal`.
|
||||
|
||||
### 7.3 Manual verification
|
||||
|
||||
- Curl each endpoint on a live box, observe sane payloads.
|
||||
- Open a public vhost in a browser, confirm all three sections render with
|
||||
current data.
|
||||
- Stop nft / HAProxy / move the live dir — confirm each section disappears
|
||||
independently within 60 s while the others stay up.
|
||||
|
||||
## 8. Open questions resolved before code
|
||||
|
||||
1. **MaxMind license key**: operator-supplied at install. Postinst creates
|
||||
`/etc/secubox/secrets/` with right perms and adds a `README.md` note
|
||||
pointing at the free GeoLite2 signup. Feature ships disabled if absent —
|
||||
no install breakage.
|
||||
2. **nftables conflict with `secubox-firewall`**: avoided by living in a
|
||||
separate table `inet secubox_metrics`, separate chain, priority `-300`.
|
||||
3. **HAProxy socket reachability from `secubox-metrics`**: the `secubox`
|
||||
system user (already used by the service) is added to the `haproxy` group in
|
||||
postinst, matching the socket's group ownership (mode `0660`). No new
|
||||
sudoers exceptions.
|
||||
|
||||
## 9. Acceptance criteria
|
||||
|
||||
- [ ] `apt install secubox-metrics` configures the nft table, deploys the
|
||||
timer, joins the haproxy group, and starts the service without manual
|
||||
steps.
|
||||
- [ ] All three endpoints (`/visitor-origin`, `/live-hosts`, `/cert-status`)
|
||||
respond 200 with documented schemas, in both enabled and disabled states.
|
||||
- [ ] No raw IP appears in any cache file, the systemd journal, or any HTTP
|
||||
response.
|
||||
- [ ] An ASN with fewer than `min_count` distinct sources is absent from all
|
||||
observable surfaces.
|
||||
- [ ] Each banner section appears within 90 s of first applicable data and
|
||||
disappears within 60 s of its data source being removed.
|
||||
- [ ] Unit test suite passes at ≥ 80 % coverage for the three aggregator
|
||||
modules.
|
||||
- [ ] Integration tests pass against fixtures (mmdb, mock HAProxy socket,
|
||||
fixture cert dir).
|
||||
- [ ] No regression on the existing `/api/v1/metrics/health/summary` payload
|
||||
or the banner's SSL section.
|
||||
|
||||
## 10. Out of scope (candidate follow-ups)
|
||||
|
||||
- IPv6 source coverage in VisitorOrigin (`type ipv6_addr` set + mmdb v6).
|
||||
- Per-vhost rollup of visitor ASNs (admin-only).
|
||||
- 24 h trend sparklines for any section.
|
||||
- Operator-configurable port list for the ingress_tap chain.
|
||||
- ASN reputation hint (flag known scraper ASNs).
|
||||
- Manual cert renewal trigger UI.
|
||||
- CertStatus failed-renewal log scanner (if it complicates v1).
|
||||
|
|
@ -15,7 +15,14 @@
|
|||
(function() {
|
||||
'use strict';
|
||||
|
||||
const VERSION = '1.2.1';
|
||||
const VERSION = '1.3.0';
|
||||
const VISITOR_ORIGIN_API = window.SECUBOX_VISITOR_ORIGIN_API
|
||||
|| '/api/v1/metrics/visitor-origin';
|
||||
const LIVE_HOSTS_API = window.SECUBOX_LIVE_HOSTS_API
|
||||
|| '/api/v1/metrics/live-hosts';
|
||||
const CERT_STATUS_API = window.SECUBOX_CERT_STATUS_API
|
||||
|| '/api/v1/metrics/cert-status';
|
||||
const LIVE_REFRESH_INTERVAL = 30000; // 30 s
|
||||
// Use global config if injected by CDN/WAF, otherwise use relative path
|
||||
const HEALTH_API = window.SECUBOX_HEALTH_API || '/api/v1/metrics/health/summary';
|
||||
const REFRESH_INTERVAL = 30000; // 30s
|
||||
|
|
@ -287,6 +294,93 @@
|
|||
return { banner, trigger };
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// LIVE PANEL — VisitorOrigin / LiveHosts / CertStatus (issue #92)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
function sectionContainer(id) {
|
||||
let el = document.getElementById(id);
|
||||
if (!el) {
|
||||
el = document.createElement('div');
|
||||
el.id = id;
|
||||
el.className = 'sbx-live-section';
|
||||
const banner = document.getElementById('health-banner');
|
||||
if (banner) banner.appendChild(el);
|
||||
}
|
||||
return el;
|
||||
}
|
||||
|
||||
function hideSection(id) {
|
||||
const el = document.getElementById(id);
|
||||
if (el) el.style.display = 'none';
|
||||
}
|
||||
|
||||
function showSection(id) {
|
||||
const el = document.getElementById(id);
|
||||
if (el) el.style.display = '';
|
||||
}
|
||||
|
||||
function renderVisitorOrigin(data) {
|
||||
if (!data || !data.enabled || !data.entries || !data.entries.length) {
|
||||
hideSection('sbx-visitor-origin');
|
||||
return;
|
||||
}
|
||||
const el = sectionContainer('sbx-visitor-origin');
|
||||
const rows = data.entries.map(e =>
|
||||
`<div class="sbx-row"><span class="sbx-asn">AS${e.asn}</span> <span class="sbx-org">${e.org}</span><span class="sbx-count">${e.count}</span></div>`
|
||||
).join('');
|
||||
el.innerHTML = `<div class="sbx-section-title">VisitorOrigin · ${data.window_minutes}min · top ${data.entries.length}</div>${rows}`;
|
||||
showSection('sbx-visitor-origin');
|
||||
}
|
||||
|
||||
function renderLiveHosts(data) {
|
||||
if (!data || !data.enabled || !data.entries || !data.entries.length) {
|
||||
hideSection('sbx-live-hosts');
|
||||
return;
|
||||
}
|
||||
const el = sectionContainer('sbx-live-hosts');
|
||||
const rows = data.entries.map(e =>
|
||||
`<div class="sbx-row"><span class="sbx-host">${e.host}</span><span class="sbx-count">${e.count}</span></div>`
|
||||
).join('');
|
||||
el.innerHTML = `<div class="sbx-section-title">LiveHosts · ${data.window_minutes}min · top ${data.entries.length}</div>${rows}`;
|
||||
showSection('sbx-live-hosts');
|
||||
}
|
||||
|
||||
function renderCertStatus(data) {
|
||||
if (!data || !data.enabled || !data.summary || !data.summary.total) {
|
||||
hideSection('sbx-cert-status');
|
||||
return;
|
||||
}
|
||||
const el = sectionContainer('sbx-cert-status');
|
||||
const s = data.summary;
|
||||
const next = data.next_renewal
|
||||
? `<div class="sbx-row">next renewal: ${data.next_renewal.host} · ${data.next_renewal.days}d</div>`
|
||||
: '';
|
||||
el.innerHTML =
|
||||
`<div class="sbx-section-title">CertStatus · ${s.total} total</div>` +
|
||||
`<div class="sbx-row">✓ ${s.valid} valid · ⚠ ${s.expiring_soon} soon · ✗ ${s.expiring_critical + s.expired} critical</div>` +
|
||||
next;
|
||||
showSection('sbx-cert-status');
|
||||
}
|
||||
|
||||
async function pollLivePanel() {
|
||||
const fetchSafe = async (url) => {
|
||||
try {
|
||||
const r = await fetch(url, { credentials: 'omit' });
|
||||
if (!r.ok) return null;
|
||||
return await r.json();
|
||||
} catch (e) { return null; }
|
||||
};
|
||||
const [vo, lh, cs] = await Promise.all([
|
||||
fetchSafe(VISITOR_ORIGIN_API),
|
||||
fetchSafe(LIVE_HOSTS_API),
|
||||
fetchSafe(CERT_STATUS_API),
|
||||
]);
|
||||
if (vo) renderVisitorOrigin(vo); else hideSection('sbx-visitor-origin');
|
||||
if (lh) renderLiveHosts(lh); else hideSection('sbx-live-hosts');
|
||||
if (cs) renderCertStatus(cs); else hideSection('sbx-cert-status');
|
||||
}
|
||||
|
||||
function injectBannerStyles() {
|
||||
if (document.getElementById('health-banner-styles')) return;
|
||||
|
||||
|
|
@ -653,6 +747,11 @@
|
|||
margin-bottom: 60vh;
|
||||
}
|
||||
}
|
||||
.sbx-live-section { padding: 6px 10px; border-top: 1px solid var(--text-muted, #6b6b7a); font-size: 11px; line-height: 1.4; }
|
||||
.sbx-section-title { font-weight: 600; color: var(--gold-hermetic, #c9a84c); margin-bottom: 2px; }
|
||||
.sbx-row { display: flex; justify-content: space-between; gap: 8px; }
|
||||
.sbx-row .sbx-count { color: var(--cyber-cyan, #00d4ff); font-variant-numeric: tabular-nums; }
|
||||
.sbx-row .sbx-asn { color: var(--matrix-green, #00ff41); }
|
||||
`;
|
||||
document.head.appendChild(style);
|
||||
}
|
||||
|
|
@ -820,6 +919,8 @@
|
|||
function init() {
|
||||
// Inject styles
|
||||
injectBannerStyles();
|
||||
pollLivePanel();
|
||||
setInterval(pollLivePanel, LIVE_REFRESH_INTERVAL);
|
||||
|
||||
// Create banner and trigger
|
||||
const { banner, trigger } = createBannerElement();
|
||||
|
|
|
|||
|
|
@ -34,6 +34,41 @@ Configuration file: `/etc/secubox/metrics.toml`
|
|||
- `GET /api/v1/metrics/status` - Module status
|
||||
- `GET /api/v1/metrics/health` - Health check
|
||||
|
||||
## Endpoints — live panel (issue #92)
|
||||
|
||||
Three public endpoints feed the health-banner live panel. All are
|
||||
unauthenticated, CORS-open, and `Cache-Control: public, max-age=300`.
|
||||
|
||||
| Method | Path | Schema (high-level) |
|
||||
|--------|-----------------------------------|------------------------------------------------------|
|
||||
| GET | `/api/v1/metrics/visitor-origin` | `{enabled, window_minutes, entries:[{asn,org,count}]}`|
|
||||
| GET | `/api/v1/metrics/live-hosts` | `{enabled, window_minutes, entries:[{host,count}]}` |
|
||||
| GET | `/api/v1/metrics/cert-status` | `{enabled, summary, next_renewal, warnings}` |
|
||||
|
||||
Config blocks live in `/etc/secubox/secubox.conf`:
|
||||
|
||||
```toml
|
||||
[visitor_origin]
|
||||
enabled = true
|
||||
min_count = 5
|
||||
|
||||
[live_hosts]
|
||||
enabled = true
|
||||
|
||||
[cert_status]
|
||||
enabled = true
|
||||
warn_days = 30
|
||||
```
|
||||
|
||||
MaxMind GeoLite2-ASN refresh: install `geoipupdate` (available in Debian
|
||||
bookworm's `contrib` repository; `secubox-metrics` lists it as a
|
||||
`Recommends`, not `Depends`, so `apt install secubox-metrics` succeeds
|
||||
without `contrib` enabled). Drop a license file at
|
||||
`/etc/secubox/secrets/maxmind.conf` (mode 0600, owner `secubox`).
|
||||
The `secubox-geoipupdate.timer` runs weekly; if either the binary or the
|
||||
key is absent, the unit is a silent no-op and the VisitorOrigin banner
|
||||
section stays hidden.
|
||||
|
||||
## License
|
||||
|
||||
MIT License - CyberMind © 2024-2026
|
||||
|
|
|
|||
138
packages/secubox-metrics/api/cert_status.py
Normal file
138
packages/secubox-metrics/api/cert_status.py
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""
|
||||
SecuBox-Deb :: CertStatus aggregator
|
||||
Scans /etc/letsencrypt/live/*/cert.pem, parses each cert via cryptography,
|
||||
and emits a rollup of {valid, expiring_soon, expiring_critical, expired}
|
||||
plus the soonest-renewing non-expired host.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from cryptography import x509
|
||||
|
||||
|
||||
log = logging.getLogger("secubox.cert_status")
|
||||
|
||||
CACHE_PATH = Path("/var/cache/secubox/metrics/cert-status.json")
|
||||
|
||||
|
||||
class CertStatusAggregator:
|
||||
def __init__(self, cfg: dict):
|
||||
self.cfg = cfg
|
||||
self._payload: dict = {"enabled": False, "summary": {}, "warnings": []}
|
||||
|
||||
# -- public ---------------------------------------------
|
||||
|
||||
def current(self) -> dict:
|
||||
if self._payload.get("summary"):
|
||||
return dict(self._payload)
|
||||
if CACHE_PATH.exists():
|
||||
try:
|
||||
return json.loads(CACHE_PATH.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return {"enabled": False, "summary": {}, "warnings": []}
|
||||
|
||||
async def run_forever(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
self._payload = await self.refresh_once()
|
||||
except Exception as e:
|
||||
log.warning("refresh_once raised: %s", e)
|
||||
await asyncio.sleep(60)
|
||||
|
||||
async def refresh_once(self) -> dict:
|
||||
if not self.cfg.get("enabled"):
|
||||
return self._disabled_payload()
|
||||
live = Path(self.cfg["letsencrypt_live_dir"])
|
||||
if not live.is_dir():
|
||||
return self._disabled_payload()
|
||||
infos = self._scan_certs(live)
|
||||
if not infos:
|
||||
return self._disabled_payload()
|
||||
payload = self._summarize(infos)
|
||||
self._persist(payload)
|
||||
return payload
|
||||
|
||||
# -- helpers --------------------------------------------
|
||||
|
||||
def _scan_certs(self, live: Path) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
now = datetime.now(timezone.utc)
|
||||
for host_dir in sorted(live.iterdir()):
|
||||
if not host_dir.is_dir():
|
||||
continue
|
||||
cert_path = host_dir / "cert.pem"
|
||||
if not cert_path.exists():
|
||||
continue
|
||||
try:
|
||||
cert = x509.load_pem_x509_certificate(cert_path.read_bytes())
|
||||
not_after = cert.not_valid_after_utc
|
||||
except Exception as e:
|
||||
log.warning("parse failed for %s: %s", host_dir.name, e)
|
||||
continue
|
||||
days = math.ceil((not_after - now).total_seconds() / 86400)
|
||||
out.append({"host": host_dir.name, "days": days, "state": self._classify(days)})
|
||||
return out
|
||||
|
||||
def _classify(self, days: int) -> str:
|
||||
if days <= 0:
|
||||
return "expired"
|
||||
if days <= self.cfg["critical_days"]:
|
||||
return "expiring_critical"
|
||||
if days <= self.cfg["warn_days"]:
|
||||
return "expiring_soon"
|
||||
return "valid"
|
||||
|
||||
def _summarize(self, infos: list[dict]) -> dict:
|
||||
buckets = {"valid": 0, "expiring_soon": 0, "expiring_critical": 0, "expired": 0}
|
||||
for i in infos:
|
||||
buckets[i["state"]] += 1
|
||||
non_expired = sorted(
|
||||
(i for i in infos if i["state"] != "expired"),
|
||||
key=lambda x: x["days"],
|
||||
)
|
||||
next_renewal = (
|
||||
{"host": non_expired[0]["host"], "days": non_expired[0]["days"]}
|
||||
if non_expired
|
||||
else None
|
||||
)
|
||||
warnings = sorted(
|
||||
(i for i in infos if i["state"] in ("expiring_soon", "expiring_critical", "expired")),
|
||||
key=lambda x: x["days"],
|
||||
)
|
||||
return {
|
||||
"enabled": True,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"summary": {"total": len(infos), **buckets, "failed_renewal": 0},
|
||||
"next_renewal": next_renewal,
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
def _persist(self, payload: dict) -> None:
|
||||
try:
|
||||
CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE_PATH.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(payload))
|
||||
tmp.replace(CACHE_PATH)
|
||||
except Exception as e:
|
||||
log.warning("persist failed: %s", e)
|
||||
|
||||
def _disabled_payload(self) -> dict:
|
||||
return {
|
||||
"enabled": False,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"summary": {},
|
||||
"warnings": [],
|
||||
}
|
||||
181
packages/secubox-metrics/api/live_hosts.py
Normal file
181
packages/secubox-metrics/api/live_hosts.py
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""
|
||||
SecuBox-Deb :: LiveHosts aggregator
|
||||
Polls the HAProxy admin socket once per minute, ring-buffers per-frontend
|
||||
request deltas over 60 minutes, and emits a sanitized top-N rollup of the
|
||||
hostnames being served.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import collections
|
||||
import json
|
||||
import logging
|
||||
import socket
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
log = logging.getLogger("secubox.live_hosts")
|
||||
|
||||
CACHE_PATH = Path("/var/cache/secubox/metrics/live-hosts.json")
|
||||
|
||||
|
||||
class LiveHostsAggregator:
|
||||
def __init__(self, cfg: dict):
|
||||
self.cfg = cfg
|
||||
self._buckets: collections.deque[dict[str, int]] = collections.deque(maxlen=60)
|
||||
self._prev_totals: dict[str, int] = {}
|
||||
self._payload: dict = {"enabled": False, "entries": []}
|
||||
self._refreshed = False
|
||||
|
||||
# -- public ---------------------------------------------
|
||||
|
||||
def current(self) -> dict:
|
||||
if self._refreshed:
|
||||
return dict(self._payload)
|
||||
if CACHE_PATH.exists():
|
||||
try:
|
||||
return json.loads(CACHE_PATH.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return {"enabled": False, "window_minutes": self.cfg["window_minutes"], "entries": []}
|
||||
|
||||
async def run_forever(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
self._payload = await self.refresh_once()
|
||||
except Exception as e:
|
||||
log.warning("refresh_once raised: %s", e)
|
||||
await asyncio.sleep(60)
|
||||
|
||||
async def refresh_once(self) -> dict:
|
||||
if not self.cfg.get("enabled"):
|
||||
self._refreshed = True
|
||||
return self._disabled_payload()
|
||||
totals = await asyncio.to_thread(self._read_haproxy_stats)
|
||||
if totals is None:
|
||||
self._refreshed = True
|
||||
return self._disabled_payload()
|
||||
kept = self._filter_frontends(totals)
|
||||
self._delta_and_buffer(kept)
|
||||
entries = self._aggregate()
|
||||
payload = {
|
||||
"enabled": True,
|
||||
"window_minutes": self.cfg["window_minutes"],
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"entries": entries,
|
||||
}
|
||||
self._persist(payload)
|
||||
self._refreshed = True
|
||||
return payload
|
||||
|
||||
# -- helpers --------------------------------------------
|
||||
|
||||
def _filter_frontends(self, totals: dict[str, int]) -> dict[str, int]:
|
||||
flt = self.cfg.get("frontend_filter", "*")
|
||||
out: dict[str, int] = {}
|
||||
for name, n in totals.items():
|
||||
if name.startswith("_"):
|
||||
continue
|
||||
if "." not in name:
|
||||
continue
|
||||
if flt != "*" and flt not in name:
|
||||
continue
|
||||
out[name] = n
|
||||
return out
|
||||
|
||||
def _delta_and_buffer(self, totals: dict[str, int]) -> None:
|
||||
if not self._prev_totals:
|
||||
self._buckets.append({k: 0 for k in totals})
|
||||
self._prev_totals = dict(totals)
|
||||
return
|
||||
bucket: dict[str, int] = {}
|
||||
for host, cur in totals.items():
|
||||
prev = self._prev_totals.get(host)
|
||||
if prev is None or cur < prev:
|
||||
bucket[host] = 0
|
||||
else:
|
||||
bucket[host] = cur - prev
|
||||
self._buckets.append(bucket)
|
||||
self._prev_totals = dict(totals)
|
||||
|
||||
def _aggregate(self) -> list[dict]:
|
||||
totals: dict[str, int] = collections.Counter()
|
||||
for bucket in self._buckets:
|
||||
for host, n in bucket.items():
|
||||
totals[host] += n
|
||||
entries = [{"host": h, "count": c} for h, c in totals.items() if c > 0]
|
||||
entries.sort(key=lambda e: (-e["count"], e["host"]))
|
||||
return entries[: self.cfg["top_n"]]
|
||||
|
||||
def _read_haproxy_stats(self) -> Optional[dict[str, int]]:
|
||||
sock_path = self.cfg["haproxy_socket"]
|
||||
if not Path(sock_path).exists():
|
||||
return None
|
||||
try:
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
|
||||
s.settimeout(2.0)
|
||||
s.connect(sock_path)
|
||||
s.sendall(b"show stat\n")
|
||||
chunks = []
|
||||
while True:
|
||||
data = s.recv(8192)
|
||||
if not data:
|
||||
break
|
||||
chunks.append(data)
|
||||
blob = b"".join(chunks).decode("utf-8", errors="replace")
|
||||
except Exception as e:
|
||||
log.warning("haproxy socket read failed: %s", e)
|
||||
return None
|
||||
return self._parse_show_stat(blob)
|
||||
|
||||
@staticmethod
|
||||
def _parse_show_stat(blob: str) -> dict[str, int]:
|
||||
"""Extract {frontend_name: req_tot} from `show stat` CSV output."""
|
||||
out: dict[str, int] = {}
|
||||
lines = blob.splitlines()
|
||||
if not lines:
|
||||
return out
|
||||
header = lines[0].lstrip("# ").split(",")
|
||||
try:
|
||||
pxname_i = header.index("pxname")
|
||||
svname_i = header.index("svname")
|
||||
req_tot_i = header.index("req_tot")
|
||||
except ValueError:
|
||||
return out
|
||||
for line in lines[1:]:
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
cols = line.split(",")
|
||||
if len(cols) <= max(pxname_i, svname_i, req_tot_i):
|
||||
continue
|
||||
if cols[svname_i] != "FRONTEND":
|
||||
continue
|
||||
try:
|
||||
out[cols[pxname_i]] = int(cols[req_tot_i] or "0")
|
||||
except ValueError:
|
||||
continue
|
||||
return out
|
||||
|
||||
def _persist(self, payload: dict) -> None:
|
||||
try:
|
||||
CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE_PATH.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(payload))
|
||||
tmp.replace(CACHE_PATH)
|
||||
except Exception as e:
|
||||
log.warning("persist failed: %s", e)
|
||||
|
||||
def _disabled_payload(self) -> dict:
|
||||
return {
|
||||
"enabled": False,
|
||||
"window_minutes": self.cfg["window_minutes"],
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"entries": [],
|
||||
}
|
||||
|
|
@ -10,6 +10,7 @@ import json
|
|||
import os
|
||||
import subprocess
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
|
@ -29,10 +30,46 @@ except ImportError:
|
|||
async def require_jwt():
|
||||
pass
|
||||
|
||||
from visitor_origin import VisitorOriginAggregator
|
||||
from live_hosts import LiveHostsAggregator
|
||||
from cert_status import CertStatusAggregator
|
||||
|
||||
try:
|
||||
from secubox_core.config import (
|
||||
get_visitor_origin_config,
|
||||
get_live_hosts_config,
|
||||
get_cert_status_config,
|
||||
)
|
||||
except ImportError: # dev fallback
|
||||
def get_visitor_origin_config(): return {"enabled": False, "window_minutes": 60, "min_count": 5, "top_n": 5, "asn_db_path": "/var/lib/GeoIP/GeoLite2-ASN.mmdb", "nft_table": "secubox_metrics", "nft_set": "seen_src", "nft_family": "inet"}
|
||||
def get_live_hosts_config(): return {"enabled": False, "window_minutes": 60, "top_n": 5, "haproxy_socket": "/run/haproxy/admin.sock", "frontend_filter": "*"}
|
||||
def get_cert_status_config(): return {"enabled": False, "letsencrypt_live_dir": "/etc/letsencrypt/live", "warn_days": 30, "critical_days": 7}
|
||||
|
||||
visitor_origin_agg = VisitorOriginAggregator(get_visitor_origin_config())
|
||||
live_hosts_agg = LiveHostsAggregator(get_live_hosts_config())
|
||||
cert_status_agg = CertStatusAggregator(get_cert_status_config())
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app):
|
||||
tasks = [
|
||||
asyncio.create_task(visitor_origin_agg.run_forever()),
|
||||
asyncio.create_task(live_hosts_agg.run_forever()),
|
||||
asyncio.create_task(cert_status_agg.run_forever()),
|
||||
]
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="SecuBox Metrics Dashboard",
|
||||
description="Real-time system metrics with caching",
|
||||
version="1.0.0"
|
||||
version="1.0.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
# CORS middleware for cross-origin health banner requests
|
||||
|
|
@ -705,6 +742,30 @@ async def get_health_summary(request: Request, domain: Optional[str] = None):
|
|||
|
||||
return summary
|
||||
|
||||
@app.get("/api/v1/metrics/visitor-origin")
|
||||
async def visitor_origin_endpoint():
|
||||
return JSONResponse(
|
||||
content=visitor_origin_agg.current(),
|
||||
headers={"Cache-Control": "public, max-age=300"},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/v1/metrics/live-hosts")
|
||||
async def live_hosts_endpoint():
|
||||
return JSONResponse(
|
||||
content=live_hosts_agg.current(),
|
||||
headers={"Cache-Control": "public, max-age=300"},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/v1/metrics/cert-status")
|
||||
async def cert_status_endpoint():
|
||||
return JSONResponse(
|
||||
content=cert_status_agg.current(),
|
||||
headers={"Cache-Control": "public, max-age=300"},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="127.0.0.1", port=8000)
|
||||
|
|
|
|||
182
packages/secubox-metrics/api/visitor_origin.py
Normal file
182
packages/secubox-metrics/api/visitor_origin.py
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""
|
||||
SecuBox-Deb :: VisitorOrigin aggregator
|
||||
Polls the secubox_metrics seen_src nft set, resolves ASNs via MaxMind GeoLite2,
|
||||
threshold-gates the rollup, and exposes a sanitized payload. Raw IPs never
|
||||
leave the local scope.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from ipaddress import IPv4Address
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
try:
|
||||
import maxminddb
|
||||
except ImportError:
|
||||
maxminddb = None # type: ignore[assignment]
|
||||
|
||||
|
||||
log = logging.getLogger("secubox.visitor_origin")
|
||||
|
||||
CACHE_PATH = Path("/var/cache/secubox/metrics/visitor-origin.json")
|
||||
|
||||
|
||||
class VisitorOriginAggregator:
|
||||
def __init__(self, cfg: dict):
|
||||
self.cfg = cfg
|
||||
self._payload: dict = {"enabled": False, "entries": []}
|
||||
self._refreshed = False
|
||||
self._mmdb = None
|
||||
self._mmdb_mtime: float = 0.0
|
||||
|
||||
# -- public ---------------------------------------------
|
||||
|
||||
def current(self) -> dict:
|
||||
if self._refreshed:
|
||||
return dict(self._payload)
|
||||
if CACHE_PATH.exists():
|
||||
try:
|
||||
return json.loads(CACHE_PATH.read_text())
|
||||
except Exception:
|
||||
pass
|
||||
return {"enabled": False, "window_minutes": self.cfg["window_minutes"], "entries": []}
|
||||
|
||||
async def run_forever(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
self._payload = await self.refresh_once()
|
||||
except Exception as e:
|
||||
log.warning("refresh_once raised: %s", e)
|
||||
await asyncio.sleep(60)
|
||||
|
||||
async def refresh_once(self) -> dict:
|
||||
if not self.cfg.get("enabled"):
|
||||
self._refreshed = True
|
||||
return self._disabled_payload()
|
||||
if not Path(self.cfg["asn_db_path"]).exists() or maxminddb is None:
|
||||
self._refreshed = True
|
||||
return self._disabled_payload()
|
||||
ips = await asyncio.to_thread(self._read_nft_set)
|
||||
entries = self._aggregate(ips)
|
||||
payload = {
|
||||
"enabled": True,
|
||||
"window_minutes": self.cfg["window_minutes"],
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"entries": entries,
|
||||
}
|
||||
self._persist(payload)
|
||||
self._refreshed = True
|
||||
return payload
|
||||
|
||||
# -- pure helpers ---------------------------------------
|
||||
|
||||
def _aggregate(self, ips: list[IPv4Address]) -> list[dict]:
|
||||
counter: Counter[tuple[int, str]] = Counter()
|
||||
seen: set[IPv4Address] = set()
|
||||
for ip in ips:
|
||||
if ip in seen:
|
||||
continue
|
||||
seen.add(ip)
|
||||
asn = self._lookup_asn(ip)
|
||||
if asn is None:
|
||||
continue
|
||||
counter[asn] += 1
|
||||
groups = [
|
||||
{"asn": asn, "org": org, "count": cnt}
|
||||
for (asn, org), cnt in counter.items()
|
||||
if cnt >= self.cfg["min_count"]
|
||||
]
|
||||
groups.sort(key=lambda e: (-e["count"], e["asn"]))
|
||||
return groups[: self.cfg["top_n"]]
|
||||
|
||||
def _lookup_asn(self, ip: IPv4Address) -> Optional[tuple[int, str]]:
|
||||
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast:
|
||||
return None
|
||||
try:
|
||||
current_mtime = Path(self.cfg["asn_db_path"]).stat().st_mtime
|
||||
except OSError:
|
||||
current_mtime = 0.0
|
||||
if self._mmdb is not None and current_mtime != self._mmdb_mtime:
|
||||
try:
|
||||
self._mmdb.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._mmdb = None
|
||||
if self._mmdb is None:
|
||||
try:
|
||||
self._mmdb = maxminddb.open_database(self.cfg["asn_db_path"])
|
||||
self._mmdb_mtime = current_mtime
|
||||
except Exception as e:
|
||||
log.warning("mmdb open failed: %s", e)
|
||||
return None
|
||||
try:
|
||||
rec = self._mmdb.get(str(ip))
|
||||
except Exception:
|
||||
return None
|
||||
if not rec:
|
||||
return None
|
||||
asn = rec.get("autonomous_system_number")
|
||||
org = rec.get("autonomous_system_organization") or ""
|
||||
if asn is None:
|
||||
return None
|
||||
return (int(asn), str(org))
|
||||
|
||||
def _read_nft_set(self) -> list[IPv4Address]:
|
||||
cmd = [
|
||||
"nft", "-j", "list", "set",
|
||||
self.cfg["nft_family"], self.cfg["nft_table"], self.cfg["nft_set"],
|
||||
]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, timeout=5)
|
||||
if res.returncode != 0:
|
||||
return []
|
||||
doc = json.loads(res.stdout)
|
||||
except Exception:
|
||||
return []
|
||||
ips: list[IPv4Address] = []
|
||||
for obj in doc.get("nftables", []):
|
||||
elem = obj.get("set", {}).get("elem")
|
||||
if not elem:
|
||||
continue
|
||||
for e in elem:
|
||||
# nftables JSON can wrap addresses in dicts when flags are set
|
||||
raw = (
|
||||
e["elem"]["val"]
|
||||
if isinstance(e, dict) and isinstance(e.get("elem"), dict)
|
||||
else e
|
||||
)
|
||||
if isinstance(raw, dict):
|
||||
raw = raw.get("val", "")
|
||||
try:
|
||||
ips.append(IPv4Address(str(raw)))
|
||||
except Exception:
|
||||
continue
|
||||
return ips
|
||||
|
||||
def _persist(self, payload: dict) -> None:
|
||||
try:
|
||||
CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = CACHE_PATH.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(payload))
|
||||
tmp.replace(CACHE_PATH)
|
||||
except Exception as e:
|
||||
log.warning("persist failed: %s", e)
|
||||
|
||||
def _disabled_payload(self) -> dict:
|
||||
return {
|
||||
"enabled": False,
|
||||
"window_minutes": self.cfg["window_minutes"],
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"entries": [],
|
||||
}
|
||||
|
|
@ -12,7 +12,11 @@ Depends: ${misc:Depends},
|
|||
secubox-core,
|
||||
python3,
|
||||
python3-fastapi | python3-pip,
|
||||
python3-uvicorn | python3-pip
|
||||
python3-uvicorn | python3-pip,
|
||||
python3-maxminddb | python3-pip,
|
||||
python3-cryptography,
|
||||
nftables
|
||||
Recommends: geoipupdate
|
||||
Description: SecuBox Metrics Dashboard
|
||||
Real-time system metrics dashboard with caching.
|
||||
Provides overview stats, WAF metrics, connection counts,
|
||||
|
|
|
|||
|
|
@ -16,10 +16,36 @@ case "$1" in
|
|||
mkdir -p /tmp/secubox
|
||||
chown secubox:secubox /tmp/secubox
|
||||
|
||||
# Add secubox to haproxy group for admin-socket access (issue #92)
|
||||
if getent group haproxy >/dev/null; then
|
||||
usermod -aG haproxy secubox || true
|
||||
fi
|
||||
|
||||
# Persistent cache dir for live-panel rollups
|
||||
mkdir -p /var/cache/secubox/metrics
|
||||
chown -R secubox:secubox /var/cache/secubox
|
||||
|
||||
# Secrets dir for MaxMind license key
|
||||
mkdir -p /etc/secubox/secrets
|
||||
chmod 0700 /etc/secubox/secrets
|
||||
chown secubox:secubox /etc/secubox/secrets
|
||||
|
||||
# GeoIP db dir
|
||||
mkdir -p /var/lib/GeoIP
|
||||
chown secubox:secubox /var/lib/GeoIP
|
||||
|
||||
# Reload nftables to pick up /etc/nftables.d/secubox-metrics.nft
|
||||
if systemctl is-active --quiet nftables; then
|
||||
systemctl reload nftables || true
|
||||
fi
|
||||
|
||||
# Enable the geoipupdate timer (no-op if user hasn't supplied a key)
|
||||
systemctl enable --now secubox-geoipupdate.timer || true
|
||||
|
||||
# Enable and start service
|
||||
systemctl daemon-reload
|
||||
systemctl enable secubox-metrics.service
|
||||
systemctl start secubox-metrics.service || true
|
||||
systemctl restart secubox-metrics.service || true
|
||||
|
||||
# Reload nginx if running
|
||||
if systemctl is-active --quiet nginx; then
|
||||
|
|
|
|||
|
|
@ -3,6 +3,9 @@ set -e
|
|||
|
||||
case "$1" in
|
||||
remove|upgrade|deconfigure)
|
||||
systemctl stop secubox-geoipupdate.timer 2>/dev/null || true
|
||||
systemctl stop secubox-geoipupdate.service 2>/dev/null || true
|
||||
systemctl disable secubox-geoipupdate.timer 2>/dev/null || true
|
||||
systemctl stop secubox-metrics.service || true
|
||||
systemctl disable secubox-metrics.service || true
|
||||
;;
|
||||
|
|
|
|||
|
|
@ -30,3 +30,12 @@ override_dh_auto_install:
|
|||
echo ' proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for;' >> $(CURDIR)/debian/secubox-metrics/etc/nginx/secubox.d/metrics.conf
|
||||
echo ' proxy_set_header X-Forwarded-Proto $$scheme;' >> $(CURDIR)/debian/secubox-metrics/etc/nginx/secubox.d/metrics.conf
|
||||
echo '}' >> $(CURDIR)/debian/secubox-metrics/etc/nginx/secubox.d/metrics.conf
|
||||
|
||||
override_dh_install:
|
||||
dh_install
|
||||
install -D -m 0644 nftables/secubox-metrics.nft \
|
||||
debian/secubox-metrics/etc/nftables.d/secubox-metrics.nft
|
||||
install -D -m 0644 systemd/secubox-geoipupdate.service \
|
||||
debian/secubox-metrics/lib/systemd/system/secubox-geoipupdate.service
|
||||
install -D -m 0644 systemd/secubox-geoipupdate.timer \
|
||||
debian/secubox-metrics/lib/systemd/system/secubox-geoipupdate.timer
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ RestartSec=5
|
|||
# PrivateTmp=true
|
||||
# ProtectSystem=full
|
||||
NoNewPrivileges=true
|
||||
ReadWritePaths=/run/secubox /tmp/secubox
|
||||
ReadWritePaths=/run/secubox /tmp/secubox /var/cache/secubox
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
|
|||
17
packages/secubox-metrics/nftables/secubox-metrics.nft
Normal file
17
packages/secubox-metrics/nftables/secubox-metrics.nft
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# SecuBox-Deb :: secubox-metrics ingress tap (issue #92)
|
||||
# Owns a private table; never touches secubox-firewall's chains.
|
||||
|
||||
table inet secubox_metrics {
|
||||
set seen_src {
|
||||
type ipv4_addr
|
||||
flags timeout
|
||||
timeout 1h
|
||||
size 65536
|
||||
}
|
||||
|
||||
chain ingress_tap {
|
||||
type filter hook prerouting priority -300; policy accept;
|
||||
tcp dport { 80, 443 } add @seen_src { ip saddr }
|
||||
}
|
||||
}
|
||||
14
packages/secubox-metrics/systemd/secubox-geoipupdate.service
Normal file
14
packages/secubox-metrics/systemd/secubox-geoipupdate.service
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
[Unit]
|
||||
Description=SecuBox — refresh GeoLite2 ASN database
|
||||
ConditionPathExists=/etc/secubox/secrets/maxmind.conf
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=secubox
|
||||
Group=secubox
|
||||
ExecStart=/usr/bin/geoipupdate -f /etc/secubox/secrets/maxmind.conf -d /var/lib/GeoIP
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/var/lib/GeoIP
|
||||
10
packages/secubox-metrics/systemd/secubox-geoipupdate.timer
Normal file
10
packages/secubox-metrics/systemd/secubox-geoipupdate.timer
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[Unit]
|
||||
Description=SecuBox — weekly GeoLite2 ASN refresh
|
||||
|
||||
[Timer]
|
||||
OnCalendar=weekly
|
||||
RandomizedDelaySec=4h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
0
packages/secubox-metrics/tests/__init__.py
Normal file
0
packages/secubox-metrics/tests/__init__.py
Normal file
11
packages/secubox-metrics/tests/conftest.py
Normal file
11
packages/secubox-metrics/tests/conftest.py
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
"""Add the package's api/ and the repo-wide common/ to sys.path for tests."""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# packages/secubox-metrics/
|
||||
_pkg_root = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(_pkg_root / "api"))
|
||||
|
||||
# repo root → common/
|
||||
_repo_root = Path(__file__).resolve().parents[3]
|
||||
sys.path.insert(0, str(_repo_root / "common"))
|
||||
88
packages/secubox-metrics/tests/test_cert_status.py
Normal file
88
packages/secubox-metrics/tests/test_cert_status.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
"""Unit tests for CertStatusAggregator."""
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
from cryptography.x509.oid import NameOID
|
||||
|
||||
from cert_status import CertStatusAggregator
|
||||
|
||||
|
||||
CFG_BASE = {
|
||||
"enabled": True,
|
||||
"warn_days": 30,
|
||||
"critical_days": 7,
|
||||
}
|
||||
|
||||
|
||||
def _make_cert(host: str, days_left: int, out_dir: Path) -> Path:
|
||||
key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
subject = issuer = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, host)])
|
||||
now = datetime.now(timezone.utc)
|
||||
# For already-expired certs (days_left < 0), not_valid_before must be
|
||||
# earlier than not_valid_after, so push it back far enough.
|
||||
not_before_offset = max(1, abs(days_left) + 1) if days_left < 0 else 1
|
||||
cert = (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(subject).issuer_name(issuer)
|
||||
.public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number())
|
||||
.not_valid_before(now - timedelta(days=not_before_offset))
|
||||
.not_valid_after(now + timedelta(days=days_left))
|
||||
.sign(key, hashes.SHA256())
|
||||
)
|
||||
host_dir = out_dir / host
|
||||
host_dir.mkdir(parents=True, exist_ok=True)
|
||||
pem_path = host_dir / "cert.pem"
|
||||
pem_path.write_bytes(cert.public_bytes(serialization.Encoding.PEM))
|
||||
return pem_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def live_dir(tmp_path):
|
||||
_make_cert("good.example.com", 60, tmp_path)
|
||||
_make_cert("soon.example.com", 14, tmp_path)
|
||||
_make_cert("crit.example.com", 3, tmp_path)
|
||||
_make_cert("dead.example.com", -2, tmp_path)
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_summary_counts_by_state(live_dir):
|
||||
agg = CertStatusAggregator(dict(CFG_BASE, letsencrypt_live_dir=str(live_dir)))
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
s = out["summary"]
|
||||
assert s["total"] == 4
|
||||
assert s["valid"] == 1
|
||||
assert s["expiring_soon"] == 1
|
||||
assert s["expiring_critical"] == 1
|
||||
assert s["expired"] == 1
|
||||
|
||||
|
||||
def test_next_renewal_is_soonest_non_expired(live_dir):
|
||||
agg = CertStatusAggregator(dict(CFG_BASE, letsencrypt_live_dir=str(live_dir)))
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
# Soonest non-expired is crit.example.com at 3d
|
||||
assert out["next_renewal"]["host"] == "crit.example.com"
|
||||
assert out["next_renewal"]["days"] == 3
|
||||
|
||||
|
||||
def test_missing_live_dir_disabled(tmp_path):
|
||||
agg = CertStatusAggregator(
|
||||
dict(CFG_BASE, letsencrypt_live_dir=str(tmp_path / "missing"))
|
||||
)
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
assert out["enabled"] is False
|
||||
|
||||
|
||||
def test_corrupt_cert_doesnt_kill_scan(live_dir):
|
||||
bad_dir = live_dir / "bad.example.com"
|
||||
bad_dir.mkdir()
|
||||
(bad_dir / "cert.pem").write_bytes(b"not a real cert")
|
||||
agg = CertStatusAggregator(dict(CFG_BASE, letsencrypt_live_dir=str(live_dir)))
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
# Still surfaces the other four
|
||||
assert out["summary"]["total"] == 4
|
||||
56
packages/secubox-metrics/tests/test_config_helpers.py
Normal file
56
packages/secubox-metrics/tests/test_config_helpers.py
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
"""Tests for the three new config helpers in secubox_core.config."""
|
||||
from pathlib import Path
|
||||
|
||||
import secubox_core.config as cfg
|
||||
|
||||
|
||||
def _reload_with(monkeypatch, tmp_path: Path, toml_text: str):
|
||||
monkeypatch.setattr(cfg, "_CONFIG", None)
|
||||
fake = tmp_path / "_secubox_test_conf.toml"
|
||||
fake.write_text(toml_text)
|
||||
monkeypatch.setattr(cfg, "_CONF_PATHS", [fake])
|
||||
|
||||
|
||||
def test_visitor_origin_defaults_when_absent(monkeypatch, tmp_path):
|
||||
_reload_with(monkeypatch, tmp_path, "[global]\nhostname='test'\n")
|
||||
out = cfg.get_visitor_origin_config()
|
||||
assert out["enabled"] is False
|
||||
assert out["window_minutes"] == 60
|
||||
assert out["min_count"] == 5
|
||||
assert out["top_n"] == 5
|
||||
assert out["asn_db_path"] == "/var/lib/GeoIP/GeoLite2-ASN.mmdb"
|
||||
assert out["nft_table"] == "secubox_metrics"
|
||||
assert out["nft_set"] == "seen_src"
|
||||
assert out["nft_family"] == "inet"
|
||||
|
||||
|
||||
def test_visitor_origin_overrides(monkeypatch, tmp_path):
|
||||
_reload_with(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
"[visitor_origin]\nenabled=true\nmin_count=2\ntop_n=3\n",
|
||||
)
|
||||
out = cfg.get_visitor_origin_config()
|
||||
assert out["enabled"] is True
|
||||
assert out["min_count"] == 2
|
||||
assert out["top_n"] == 3
|
||||
assert out["window_minutes"] == 60
|
||||
|
||||
|
||||
def test_live_hosts_defaults(monkeypatch, tmp_path):
|
||||
_reload_with(monkeypatch, tmp_path, "[global]\nhostname='test'\n")
|
||||
out = cfg.get_live_hosts_config()
|
||||
assert out["enabled"] is False
|
||||
assert out["window_minutes"] == 60
|
||||
assert out["top_n"] == 5
|
||||
assert out["haproxy_socket"] == "/run/haproxy/admin.sock"
|
||||
assert out["frontend_filter"] == "*"
|
||||
|
||||
|
||||
def test_cert_status_defaults(monkeypatch, tmp_path):
|
||||
_reload_with(monkeypatch, tmp_path, "[global]\nhostname='test'\n")
|
||||
out = cfg.get_cert_status_config()
|
||||
assert out["enabled"] is False
|
||||
assert out["letsencrypt_live_dir"] == "/etc/letsencrypt/live"
|
||||
assert out["warn_days"] == 30
|
||||
assert out["critical_days"] == 7
|
||||
110
packages/secubox-metrics/tests/test_live_hosts.py
Normal file
110
packages/secubox-metrics/tests/test_live_hosts.py
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
"""Unit tests for LiveHosts ring buffer + parsing + filter + aggregation."""
|
||||
import asyncio
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
from live_hosts import LiveHostsAggregator
|
||||
|
||||
|
||||
CFG = {
|
||||
"enabled": True,
|
||||
"window_minutes": 60,
|
||||
"top_n": 5,
|
||||
"haproxy_socket": "/nonexistent.sock",
|
||||
"frontend_filter": "*",
|
||||
}
|
||||
|
||||
|
||||
def _drive(agg, totals_sequence):
|
||||
"""Feed a sequence of {frontend: req_tot} dicts to _delta_and_buffer."""
|
||||
for totals in totals_sequence:
|
||||
agg._delta_and_buffer(totals)
|
||||
|
||||
|
||||
def test_delta_buffers_first_sample_as_zeros():
|
||||
agg = LiveHostsAggregator(CFG)
|
||||
_drive(agg, [{"foo.com": 100, "bar.com": 50}])
|
||||
assert len(agg._buckets) == 1
|
||||
assert agg._buckets[0] == {"foo.com": 0, "bar.com": 0}
|
||||
|
||||
|
||||
def test_delta_computes_increment():
|
||||
agg = LiveHostsAggregator(CFG)
|
||||
_drive(agg, [
|
||||
{"foo.com": 100}, # init
|
||||
{"foo.com": 142}, # +42
|
||||
{"foo.com": 150}, # +8
|
||||
])
|
||||
assert agg._buckets[-1] == {"foo.com": 8}
|
||||
assert agg._buckets[-2] == {"foo.com": 42}
|
||||
|
||||
|
||||
def test_delta_handles_haproxy_restart_no_negatives():
|
||||
agg = LiveHostsAggregator(CFG)
|
||||
_drive(agg, [
|
||||
{"foo.com": 1000},
|
||||
{"foo.com": 1050}, # +50
|
||||
{"foo.com": 12}, # counter reset -> treat as fresh
|
||||
])
|
||||
# After reset the bucket value should be 0 (fresh baseline), never -1038.
|
||||
assert agg._buckets[-1].get("foo.com", 0) == 0
|
||||
|
||||
|
||||
def test_ring_buffer_caps_at_60():
|
||||
agg = LiveHostsAggregator(CFG)
|
||||
for i in range(65):
|
||||
agg._delta_and_buffer({"foo.com": i * 10})
|
||||
assert len(agg._buckets) == 60
|
||||
|
||||
|
||||
def test_aggregate_sums_buckets_top_n():
|
||||
agg = LiveHostsAggregator(dict(CFG, top_n=2))
|
||||
_drive(agg, [
|
||||
{"a.com": 0, "b.com": 0, "c.com": 0}, # baseline
|
||||
{"a.com": 30, "b.com": 10, "c.com": 5}, # deltas 30/10/5
|
||||
{"a.com": 40, "b.com": 25, "c.com": 6}, # deltas 10/15/1
|
||||
])
|
||||
entries = agg._aggregate()
|
||||
hosts = [e["host"] for e in entries]
|
||||
assert hosts == ["a.com", "b.com"]
|
||||
counts = {e["host"]: e["count"] for e in entries}
|
||||
assert counts["a.com"] == 40
|
||||
assert counts["b.com"] == 25
|
||||
|
||||
|
||||
def test_frontend_filter_strips_internal_names():
|
||||
agg = LiveHostsAggregator(CFG)
|
||||
raw = {
|
||||
"foo.com": 10,
|
||||
"_stats": 999, # leading underscore -> drop
|
||||
"stats-https": 5, # no dot -> drop
|
||||
"secubox.in": 50,
|
||||
}
|
||||
kept = agg._filter_frontends(raw)
|
||||
assert "foo.com" in kept
|
||||
assert "secubox.in" in kept
|
||||
assert "_stats" not in kept
|
||||
assert "stats-https" not in kept
|
||||
|
||||
|
||||
def test_parse_show_stat_csv_extracts_frontends():
|
||||
csv = (
|
||||
"# pxname,svname,qcur,scur,smax,slim,stot,req_tot,extra\n"
|
||||
"stats-https,FRONTEND,0,0,0,0,0,5,\n"
|
||||
"secubox.in,FRONTEND,0,1,1,0,42,142,\n"
|
||||
"secubox.in,backend-a,0,0,0,0,0,0,\n"
|
||||
"apt.secubox.in,FRONTEND,0,0,0,0,0,9,\n"
|
||||
)
|
||||
out = LiveHostsAggregator._parse_show_stat(csv)
|
||||
assert out == {"stats-https": 5, "secubox.in": 142, "apt.secubox.in": 9}
|
||||
|
||||
|
||||
def test_parse_show_stat_empty_or_malformed_returns_empty():
|
||||
assert LiveHostsAggregator._parse_show_stat("") == {}
|
||||
assert LiveHostsAggregator._parse_show_stat("garbage\n") == {}
|
||||
|
||||
|
||||
def test_refresh_missing_socket_returns_disabled(tmp_path):
|
||||
agg = LiveHostsAggregator(dict(CFG, haproxy_socket=str(tmp_path / "missing.sock")))
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
assert out["enabled"] is False
|
||||
assert out["entries"] == []
|
||||
158
packages/secubox-metrics/tests/test_visitor_origin.py
Normal file
158
packages/secubox-metrics/tests/test_visitor_origin.py
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
"""Unit tests for the VisitorOrigin aggregator pure functions."""
|
||||
from ipaddress import IPv4Address
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from visitor_origin import VisitorOriginAggregator
|
||||
|
||||
|
||||
CFG = {
|
||||
"enabled": True,
|
||||
"window_minutes": 60,
|
||||
"min_count": 5,
|
||||
"top_n": 5,
|
||||
"asn_db_path": "/nonexistent/asn.mmdb",
|
||||
"nft_table": "secubox_metrics",
|
||||
"nft_set": "seen_src",
|
||||
"nft_family": "inet",
|
||||
}
|
||||
|
||||
|
||||
def _agg_with_mock_asn(mapping: dict[str, tuple[int, str]]):
|
||||
agg = VisitorOriginAggregator(CFG)
|
||||
|
||||
def fake_lookup(ip):
|
||||
return mapping.get(str(ip))
|
||||
|
||||
agg._lookup_asn = fake_lookup # type: ignore[assignment]
|
||||
return agg
|
||||
|
||||
|
||||
def test_aggregate_counts_unique_ips_per_asn():
|
||||
agg = _agg_with_mock_asn({
|
||||
"1.1.1.1": (13335, "Cloudflare, Inc."),
|
||||
"1.0.0.1": (13335, "Cloudflare, Inc."),
|
||||
"8.8.8.8": (15169, "Google LLC"),
|
||||
"9.9.9.9": (19281, "Quad9"),
|
||||
})
|
||||
# Force min_count=1 for this test so all groups survive
|
||||
agg.cfg = dict(CFG, min_count=1)
|
||||
entries = agg._aggregate([IPv4Address("1.1.1.1"),
|
||||
IPv4Address("1.0.0.1"),
|
||||
IPv4Address("8.8.8.8"),
|
||||
IPv4Address("9.9.9.9")])
|
||||
counts = {e["asn"]: e["count"] for e in entries}
|
||||
assert counts == {13335: 2, 15169: 1, 19281: 1}
|
||||
|
||||
|
||||
def test_aggregate_threshold_keeps_equal_drops_below():
|
||||
agg = _agg_with_mock_asn({
|
||||
"1.1.1.1": (13335, "Cloudflare"),
|
||||
"1.0.0.1": (13335, "Cloudflare"),
|
||||
"1.0.0.2": (13335, "Cloudflare"),
|
||||
"1.0.0.3": (13335, "Cloudflare"),
|
||||
"1.0.0.4": (13335, "Cloudflare"), # 5 distinct -> kept
|
||||
"8.8.8.8": (15169, "Google"),
|
||||
"8.8.4.4": (15169, "Google"),
|
||||
"8.8.0.1": (15169, "Google"),
|
||||
"8.8.0.2": (15169, "Google"), # 4 distinct -> dropped
|
||||
})
|
||||
entries = agg._aggregate([IPv4Address(s) for s in [
|
||||
"1.1.1.1", "1.0.0.1", "1.0.0.2", "1.0.0.3", "1.0.0.4",
|
||||
"8.8.8.8", "8.8.4.4", "8.8.0.1", "8.8.0.2",
|
||||
]])
|
||||
asns = {e["asn"] for e in entries}
|
||||
assert 13335 in asns
|
||||
assert 15169 not in asns
|
||||
|
||||
|
||||
def test_aggregate_top_n_and_tiebreak_by_asn():
|
||||
agg = _agg_with_mock_asn({
|
||||
f"10.0.0.{i}": (100 + (i % 3), f"org{i % 3}") for i in range(1, 16)
|
||||
})
|
||||
agg.cfg = dict(CFG, top_n=2, min_count=1)
|
||||
entries = agg._aggregate([IPv4Address(f"10.0.0.{i}") for i in range(1, 16)])
|
||||
assert len(entries) == 2
|
||||
# All three ASNs (100, 101, 102) have count=5; tie must resolve to smallest ASN first.
|
||||
assert entries[0]["asn"] == 100
|
||||
assert entries[1]["asn"] == 101
|
||||
assert entries[0]["count"] == 5
|
||||
assert entries[1]["count"] == 5
|
||||
|
||||
|
||||
def test_lookup_asn_returns_none_for_private():
|
||||
agg = VisitorOriginAggregator(CFG)
|
||||
assert agg._lookup_asn(IPv4Address("10.0.0.1")) is None
|
||||
assert agg._lookup_asn(IPv4Address("192.168.1.1")) is None
|
||||
assert agg._lookup_asn(IPv4Address("127.0.0.1")) is None
|
||||
|
||||
|
||||
def test_lookup_asn_reopens_on_mtime_change(monkeypatch, tmp_path):
|
||||
"""If the mmdb file's mtime changes between calls, the handle is reopened."""
|
||||
db_path = tmp_path / "asn.mmdb"
|
||||
db_path.write_bytes(b"v1")
|
||||
cfg = dict(CFG, asn_db_path=str(db_path))
|
||||
agg = VisitorOriginAggregator(cfg)
|
||||
|
||||
open_calls = []
|
||||
|
||||
class FakeMmdb:
|
||||
def __init__(self, n): self.n = n
|
||||
def get(self, _): return None
|
||||
def close(self): pass
|
||||
|
||||
def fake_open(path):
|
||||
open_calls.append(path)
|
||||
return FakeMmdb(len(open_calls))
|
||||
|
||||
monkeypatch.setattr("visitor_origin.maxminddb",
|
||||
type("M", (), {"open_database": staticmethod(fake_open)}))
|
||||
|
||||
# First lookup opens v1
|
||||
agg._lookup_asn(IPv4Address("1.1.1.1"))
|
||||
assert len(open_calls) == 1
|
||||
|
||||
# No file change → no reopen
|
||||
agg._lookup_asn(IPv4Address("1.1.1.2"))
|
||||
assert len(open_calls) == 1
|
||||
|
||||
# Bump mtime → reopen on next call
|
||||
import os
|
||||
new_mtime = db_path.stat().st_mtime + 100
|
||||
os.utime(db_path, (new_mtime, new_mtime))
|
||||
agg._lookup_asn(IPv4Address("1.1.1.3"))
|
||||
assert len(open_calls) == 2
|
||||
|
||||
|
||||
def test_refresh_disabled_returns_disabled():
|
||||
agg = VisitorOriginAggregator(dict(CFG, enabled=False))
|
||||
import asyncio
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
assert out["enabled"] is False
|
||||
assert out["entries"] == []
|
||||
|
||||
|
||||
def test_refresh_missing_mmdb_returns_disabled(tmp_path):
|
||||
agg = VisitorOriginAggregator(dict(CFG, asn_db_path=str(tmp_path / "missing.mmdb")))
|
||||
import asyncio
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
assert out["enabled"] is False
|
||||
|
||||
|
||||
def test_refresh_handles_nft_failure(monkeypatch, tmp_path):
|
||||
# Pretend mmdb exists so we reach _read_nft_set
|
||||
fake_db = tmp_path / "asn.mmdb"
|
||||
fake_db.write_bytes(b"\x00") # truthy existence
|
||||
agg = VisitorOriginAggregator(dict(CFG, asn_db_path=str(fake_db)))
|
||||
# Force maxminddb import path to succeed without a real db by mocking _lookup_asn
|
||||
agg._lookup_asn = lambda ip: None # type: ignore[assignment]
|
||||
# Ensure module-level maxminddb guard passes even if package is not installed
|
||||
monkeypatch.setattr("visitor_origin.maxminddb", object())
|
||||
monkeypatch.setattr(
|
||||
"visitor_origin.subprocess.run",
|
||||
lambda *a, **kw: MagicMock(returncode=1, stdout=""),
|
||||
)
|
||||
import asyncio
|
||||
out = asyncio.run(agg.refresh_once())
|
||||
assert out["enabled"] is True
|
||||
assert out["entries"] == []
|
||||
Loading…
Reference in New Issue
Block a user