mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-29 09:14:33 +00:00
Some checks are pending
License Headers / check (push) Waiting to run
* feat(p2p): DHT, Federation, MasterLink evolutions for P2P-EVO-2026-07-001 Implements three major features for secubox-p2p: 1. DHT (Distributed Hash Table) based on Kademlia algorithm - Peer discovery across different subnets - Resilience to temporary node failures - Scalability for large number of nodes - REST API endpoints for DHT management 2. Service Federation - Service registration and discovery - Multi-version service management - Automatic health checks - DHT integration for service propagation 3. Master-Link Hierarchical Topology - Master/satellite/leaf hierarchical structure - Automatic master election and failover - Heartbeat monitoring - Routing policy management - OPAD security integration New files: - api/dht.py: Kademlia DHT implementation - api/federation.py: Service federation system - api/masterlink.py: Hierarchical topology manager - api/main_evolutions.py: Integrated FastAPI server - tests/test_dht.py: Unit tests for DHT Issue: P2P-EVO-2026-07-001 Worktree: secubox-p2p-evolutions Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * revert(p2p): remove non-integrating DHT/federation/masterlink code, keep requirements The DHT/federation/masterlink modules + main_evolutions.py + test_dht.py did not integrate with the module: tests imported 'secubox.p2p.api.*' (the package uses 'from api import ...' via conftest sys.path) so nothing even collected, and the modules reinvented standalone aiohttp servers instead of building on the existing WireGuard mesh (mesh.py) + annuaire-backed registry (registry.py/annuaire_client.py). Removing them to restart on the real base. The requirements doc (.github/ISSUES/2026-07-P2P-EVOLUTIONS.md) is kept as the spec source. * docs(p2p): point the local evolutions note to real issue #774 The '.github/ISSUES/2026-07-P2P-EVOLUTIONS.md' was a local file with a fabricated id, not a tracked issue. Replace its contents with a pointer to the real GitHub issue #774 (Kademlia DHT + hierarchical master-link + federation health-checks), framed against the already-done federation (#766) and directory (#768) work and the master-link/auto-enrollment issue (#762). * docs(spec): p2p Kademlia DHT + master-link + federation health-checks (#774) Clean redesign on the real base (mesh.py + registry/annuaire_client) after reverting the non-integrating attempt. Custom minimal Kademlia (pure asyncio UDP, JSON wire, signed reachability records), federation health-checks layered over the annuaire federation (#766) publishing status via the DHT, and a deterministic-election master-link (term-based failover, aligns #762). All feature-flagged off by default (OPAD opt-in), from api import convention, TDD per module. Refs #774. * docs(plan): p2p DHT/federation/master-link TDD implementation plan (#774) 17 bite-sized TDD tasks across 3 phases (DHT 1-9, federation health-checks 10-13, master-link 14-16, finalization 17). Each task = failing test -> minimal impl -> pass -> commit, on the real base (from api import ...). Crypto (Ed25519 sign/verify) and UDP transport are module-level seams injected in unit tests so the Kademlia primitives, health debounce, and election/term logic are tested without real sockets. Refs #774. * feat(p2p): DHT scaffold — node id + xor distance (#774) * feat(p2p): DHT k-bucket with LRU (#774) Add DHTNode contact + DHTBucket (k-bucket with LRU ordering). - DHTNode: dataclass with node_id, did, endpoint, last_seen - DHTBucket: Kademlia k-bucket with OrderedDict LRU management - add(node) → bool (True if stored/refreshed, False if full) - refresh moves node to tail (most-recent) - oldest() returns head node - remove(node_id) and nodes property for list access All Task 1 tests + new Task 2 tests passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(p2p): DHT routing table + closest-N (#774) * feat(p2p): DHT signed reachability records + verify (#774) Implement canonical_record, sign_record, verify_record functions with crypto SEAMS (_did_from_pubkey, _verify_sig, _sign_sig) for testing. Tests use monkeypatching; real crypto wired in Task 8. Canonical records are deterministic sorted JSON. verify_record checks for sig field, DID validity via wg_pubkey, and cryptographic signature. * feat(p2p): DHT JSON UDP codec + hardening (#774) * feat(p2p): DHTNetwork store+dispatch (transport-injected) (#774) * feat(p2p): DHT iterative find_node/find_value + announce (#774) Add asyncio-based iterative Kademlia lookup on top of the injected DHTNetwork transport from Task 6: - pending: dict[rpc_id, Future] on DHTNetwork, resolved by handle_message when a reply (pong/nodes/value/ok) arrives with a matching rpc_id. - _rpc(node, msg): send + await the matching future with RPC_TIMEOUT, returns None on timeout. - iterative_find(target_id, mode): alpha-parallel iterative lookup, dedups queried nodes, merges discovered contacts into the shortlist (dedup by node_id, skip self), terminates on no-improvement-in-a-round or exhausted candidates; mode="value" short-circuits on first verified record. - find_peer(did) / announce(): DID resolution and self-record publication (sign, cache locally, push to the k closest nodes found). Tests (tests/test_dht.py, +2 async, pytest-asyncio strict mode): - in-process UDP router delivering via call_soon (never inline) so awaited RPC futures resolve on a later loop turn; - find_peer resolves a record announced by a node reachable only through a bootstrap intermediary; - dedup: a node introduced twice into the shortlist (via two different repliers) is queried exactly once. All 16 tests green (14 prior + 2 new). * fix(p2p): harden DHT iterative lookup — skip malformed contacts, cap shortlist, tolerate rpc exceptions (#774) * feat(p2p): DHT record schema (id+wg keys) + real Ed25519 sign/verify (#774) * feat(p2p): DHT UDP transport + routing persistence + bootstrap (#774) * feat(p2p): mount DHT — endpoints + startup guard + [dht] config (#774) Task 9: wire the finished Kademlia DHT (api/dht.py) into the FastAPI app as a feature-flagged background service, fully backward compatible. - api/mesh.py: load_p2p_config() now also returns a `dht` sub-dict (enabled/port/bootstrap/announce/announce_interval/rps) built from an optional [dht] toml section, with defaults when absent. - api/main.py: @app.on_event("startup") _dht_startup creates+binds a DHTNetwork only when [dht].enabled is true; any failure (missing identity, bind error, ...) is caught and leaves app.state.dht = None so the app always starts. _dht_shutdown persists the routing table and closes the UDP transport. - New endpoints (bare paths, nginx adds /api/v1/p2p prefix): GET /dht/peers (routing snapshot), POST /dht/announce (JWT-protected, audit-logged to /var/log/secubox/p2p-audit.log), GET /dht/find/{did}. Tests: tests/test_dht_wiring.py (config defaults/overrides + disabled TestClient smoke). All 22 existing DHT tests + 3 new tests green (74 total in the package). * feat(p2p): federation HealthStore + debounce (#774) * feat(p2p): federation HealthChecker sweep (#774) * feat(p2p): federation real probe + registry services (#774) * fix(p2p): bound TCP-probe close, declare aiohttp dep, guard non-dict service (#774) * feat(p2p): federation health publish-via-DHT + endpoints + [federation] config (#774) * fix(p2p): wire federation probe_timeout + harden sweep/health-TTL tests (#774) * feat(p2p): master-link elect() + term store (#774) * feat(p2p): master-link state machine + failover (#774) * fix(p2p): master-link equal-term tie-break (no zero-master window) + defensive peers (#774) * feat(p2p): master-link signed heartbeats + UDP transport + tick loop (#774) * feat(p2p): mount master-link — endpoints + startup guard + [masterlink] config (#774) - masterlink.MasterLink.request_promotion(): on-demand promotion attempt (bump term, run election, become MASTER + heartbeat on win, else CANDIDATE) - mesh.load_p2p_config(): add [masterlink] section defaults/overrides (enabled, role_preference, priority, heartbeat_interval, election_timeout, port, peer_addrs) alongside existing [dht]/[federation] - main.py: _masterlink_startup/_masterlink_shutdown mirror the DHT/federation defensive startup pattern (feature-flagged, never breaks app startup); GET /masterlink/topology (public) + POST /masterlink/promote (JWT, audited to p2p-audit.log) * fix(p2p): master-link request_promotion records winner on loss + strengthen test (#774) * docs(p2p): document DHT/federation/master-link evolutions (#774) Add an "Evolutions (#774)" section to packages/secubox-p2p/README.md covering the DHT (api/dht.py), federation health-checks (api/federation.py) and hierarchical master-link (api/masterlink.py) subsystems: what each does, their API endpoints, the real p2p.toml config defaults from api/mesh.py, and the audit log. All three remain OFF by default / OPAD opt-in. Docs only, no behavior change. Append a matching dated entry to .claude/HISTORY.md. * fix(p2p): final-review fixes — audit-log path, master-link peers_fn, wg_pubkey, handle_message hardening (#774) * fix(p2p): implement mesh visualization tab — load on activation + master-link roles (#774) The Mesh tab canvas was never rendered (the tab-switch handler didn't call initMesh, and drawing into a hidden tab gave clientWidth 0 -> blank canvas). Now: initMesh fires when the Mesh tab is activated, retries if the canvas has no layout yet, and enriches the star graph with the live master-link state (role/term/master + DHT peer count from /masterlink/topology + /dht/peers) — center node coloured gold=MASTER / cyan=SATELLITE, peers by online status. * docs(p2p): poster GPT + roadmap des évolutions DHT/federation/master-link; maj WIP/HISTORY/TODO (ref #774) --------- Co-authored-by: CyberMind-FR <gandalf@Gk2.net> Co-authored-by: Mistral Vibe <vibe@mistral.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
158 lines
5.3 KiB
Python
158 lines
5.3 KiB
Python
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
|
"""Issue #774 Task 16b: wire the MasterLink into api/main.py +
|
|
[masterlink] config section."""
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
from api import masterlink, mesh
|
|
|
|
|
|
def test_load_p2p_config_masterlink_defaults(tmp_path):
|
|
p = tmp_path / "p2p.toml"
|
|
p.write_text('[wireguard]\nrole = "satellite"\n')
|
|
cfg = mesh.load_p2p_config(p)
|
|
assert cfg["masterlink"] == {
|
|
"enabled": False,
|
|
"role_preference": "auto",
|
|
"priority": 100,
|
|
"heartbeat_interval": 5,
|
|
"election_timeout": 15,
|
|
"port": 51824,
|
|
"peer_addrs": [],
|
|
}
|
|
|
|
|
|
def test_load_p2p_config_masterlink_section_overrides(tmp_path):
|
|
p = tmp_path / "p2p.toml"
|
|
p.write_text(
|
|
"[wireguard]\n"
|
|
'role = "master"\n'
|
|
"[masterlink]\n"
|
|
"enabled = true\n"
|
|
"priority = 10\n"
|
|
'peer_addrs = ["10.10.0.2:51824"]\n'
|
|
)
|
|
cfg = mesh.load_p2p_config(p)
|
|
assert cfg["masterlink"]["enabled"] is True
|
|
assert cfg["masterlink"]["priority"] == 10
|
|
assert cfg["masterlink"]["peer_addrs"] == ["10.10.0.2:51824"]
|
|
# Untouched defaults are preserved.
|
|
assert cfg["masterlink"]["heartbeat_interval"] == 5
|
|
assert cfg["masterlink"]["election_timeout"] == 15
|
|
assert cfg["masterlink"]["port"] == 51824
|
|
|
|
|
|
def test_masterlink_topology_endpoint_disabled():
|
|
from api.main import app
|
|
|
|
with TestClient(app) as client:
|
|
# masterlink is not enabled in the test environment (no p2p.toml
|
|
# with [masterlink].enabled = true), so startup must set
|
|
# app.state.masterlink = None and never break app startup.
|
|
assert app.state.masterlink is None
|
|
r = client.get("/masterlink/topology")
|
|
assert r.status_code == 200
|
|
assert r.json() == {"enabled": False}
|
|
|
|
|
|
def test_masterlink_promote_endpoint_disabled():
|
|
from api.main import app, require_jwt
|
|
|
|
async def _override_jwt():
|
|
return {"sub": "admin"}
|
|
|
|
app.dependency_overrides[require_jwt] = _override_jwt
|
|
try:
|
|
with TestClient(app) as client:
|
|
assert app.state.masterlink is None
|
|
r = client.post("/masterlink/promote")
|
|
assert r.status_code == 400
|
|
finally:
|
|
app.dependency_overrides.clear()
|
|
|
|
|
|
# -- Final-review fix (Issue #774): peers_fn must reflect the real mesh --
|
|
#
|
|
# _masterlink_startup used to wire peers_fn=lambda: [], so every node's
|
|
# _candidates() only ever contained itself and it trivially self-elected
|
|
# regardless of any real peers. peers_fn must be built from the live wg-mesh
|
|
# state (wg_mesh.json) via masterlink.peers_from_mesh(), the adapter that
|
|
# already existed but was never called.
|
|
|
|
def test_masterlink_peers_fn_uses_real_mesh_state(monkeypatch):
|
|
import api.main as main_mod
|
|
|
|
fake_mesh = {
|
|
"peers": [
|
|
{"public_key": "peerpubkey1", "endpoint": "10.10.0.2:51822",
|
|
"allowed_ips": "10.10.0.2/32"},
|
|
]
|
|
}
|
|
monkeypatch.setattr(main_mod, "get_wg_mesh_config", lambda: fake_mesh)
|
|
|
|
peers = main_mod._masterlink_peers_fn()
|
|
|
|
assert peers == masterlink.peers_from_mesh(fake_mesh)
|
|
assert peers != []
|
|
assert peers[0]["priority"] == 100
|
|
assert "node_id_hex" in peers[0]
|
|
|
|
|
|
def test_masterlink_peers_fn_never_raises_on_bad_mesh_state(monkeypatch):
|
|
"""peers_fn must degrade to an empty candidate list (self-election),
|
|
never raise, when the mesh-state accessor is unavailable/broken."""
|
|
import api.main as main_mod
|
|
|
|
def _boom():
|
|
raise RuntimeError("wg_mesh.json unreadable")
|
|
|
|
monkeypatch.setattr(main_mod, "get_wg_mesh_config", _boom)
|
|
|
|
assert main_mod._masterlink_peers_fn() == []
|
|
|
|
|
|
@pytest.mark.integration
|
|
@pytest.mark.asyncio
|
|
async def test_masterlink_startup_wires_real_peers_fn(tmp_path, monkeypatch):
|
|
"""End-to-end: _masterlink_startup must wire the MasterLink's peers_fn to
|
|
_masterlink_peers_fn (real mesh state), not a `lambda: []` closure."""
|
|
from cryptography.hazmat.primitives.asymmetric import ed25519
|
|
from cryptography.hazmat.primitives.serialization import (
|
|
Encoding, NoEncryption, PrivateFormat,
|
|
)
|
|
|
|
import api.annuaire_client as annuaire_client
|
|
import api.main as main_mod
|
|
|
|
priv_key = ed25519.Ed25519PrivateKey.generate()
|
|
priv_hex = priv_key.private_bytes(Encoding.Raw, PrivateFormat.Raw, NoEncryption()).hex()
|
|
monkeypatch.setattr(annuaire_client, "node_identity", lambda *a, **kw: ("did:x", priv_hex))
|
|
|
|
toml = tmp_path / "p2p.toml"
|
|
toml.write_text(
|
|
'[wireguard]\nrole = "satellite"\n'
|
|
"[masterlink]\nenabled = true\nport = 0\n"
|
|
)
|
|
monkeypatch.setattr(main_mod, "CONFIG_FILE", toml)
|
|
|
|
fake_mesh = {
|
|
"peers": [
|
|
{"public_key": "peerpubkey1", "endpoint": "10.10.0.2:51822",
|
|
"allowed_ips": "10.10.0.2/32"},
|
|
]
|
|
}
|
|
monkeypatch.setattr(main_mod, "get_wg_mesh_config", lambda: fake_mesh)
|
|
|
|
await main_mod._masterlink_startup()
|
|
ml = main_mod.app.state.masterlink
|
|
try:
|
|
assert ml is not None
|
|
assert ml._peers_fn is main_mod._masterlink_peers_fn
|
|
peers = ml._peers_fn()
|
|
assert peers == masterlink.peers_from_mesh(fake_mesh)
|
|
assert peers != []
|
|
finally:
|
|
await ml.stop()
|
|
main_mod.app.state.masterlink = None
|