mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-29 22:07:24 +00:00
feat: Media Buffer Phase 2 — HLS reassembly via read-time URL join (#815)
Some checks failed
License Headers / check (push) Has been cancelled
Some checks failed
License Headers / check (push) Has been cancelled
* feat(sbxmitm): capture HLS segments as kind=segment (Phase 2, ref #812) Add segmentKind(path, ctype) to classify HLS/DASH segment chunks (.ts, .m4s, .m4v paths or video/mp2t, video/iso.segment content types), and give it precedence over mediaKind in Capture so a .ts served as video/mp2t is tagged kind="segment" instead of "video". Whole-file .mp4/video/* stays "video" and .m3u8/.mpd manifests stay "manifest". IsMedia now also matches segmentKind so ambiguous-ctype segments (e.g. .m4s over application/octet-stream) are still captured. * feat(core): HLS media-playlist parser + segment-URI rewriter (Phase 2, ref #812) * feat(dpi): HLS manifest replay — rewrite segment URIs to captured segments (Phase 2, ref #812) media_replay() branches on record kind: a captured HLS manifest is parsed with secubox_core.hls and its segment URIs are rewritten to the matching captured kind=="segment" records' replay URLs (read-time join by absolute URL, scoped to the same mac_hash+host). Master/ABR and AES-128 encrypted playlists are detected and returned raw with X-SecuBox-Media: unsupported-variant instead of a broken rewrite. Every other kind keeps the unchanged Phase-1 FileResponse path; any manifest parse/read error falls back to the raw FileResponse (never a 500). * fix(dpi): log HLS manifest/segment/rewrite truncation + pin manifest replay fail-safe (Phase 2 review, ref #812) * feat(dpi): Media tab — hide HLS segments, manifest playback/download + unsupported-variant note (Phase 2, ref #812) * test(#812): HLS reassembly end-to-end + changelog 0.1.28 (Phase 2) Cross-layer integration test seeding a temp media-buffer exactly as the Go tee + janitor produce on disk (metatag JSONL + per-session object files), then driving the real DPI media_replay handler: manifest rewrite joins each segment URI to its captured segment by resolved URL, every rewritten URL's id replays to byte-identical captured bytes, and a janitor-style append-only eviction flip leaves the evicted segment unrewritten (410 on direct replay) while its siblings keep replaying. This is the whole-branch-review-flagged integration coverage Phase 1 was missing. Bumps secubox-toolbox-ng changelog to 0.1.28-1~bookworm1 with the Phase 2 HLS reassembly summary (segment capture, manifest rewrite, unsupported-variant detection for master/ABR + AES + DASH). * chore(dpi): changelog 1.2.0 — media buffer API + HLS reassembly (#812 Phase 1+2) --------- Co-authored-by: CyberMind-FR <gandalf@Gk2.net>
This commit is contained in:
CyberMind
GitHub
parent
418126c541
commit
fecd233d88
235
common/secubox_core/hls.py
Normal file
235
common/secubox_core/hls.py
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""
|
||||
SecuBox-Deb :: HLS media-playlist parser + segment-URI rewriter
|
||||
|
||||
Phase 2 (media-buffer reassembly) uses a **read-time join**: sbxmitm never
|
||||
tracks HLS session state, it just captures the manifest and each segment
|
||||
independently (tagged `kind="manifest"`/`kind="segment"`). At replay time
|
||||
the DPI backend parses the captured manifest bytes with this module,
|
||||
resolves every segment URI against the manifest's own URL, and looks the
|
||||
resulting absolute URL up against the captured segment records to rewrite
|
||||
the playlist to point at `/api/v1/dpi/media/replay/{seg_id}`.
|
||||
|
||||
Scope — **HLS media (single-variant) playlists only**:
|
||||
- Master/multivariant (ABR) playlists (`#EXT-X-STREAM-INF`) and encrypted
|
||||
media (`#EXT-X-KEY` with METHOD != NONE) are detected via
|
||||
`is_master_playlist`/`is_encrypted` and marked unsupported by the caller —
|
||||
this module does not attempt to parse variants or decrypt segments.
|
||||
- DASH `.mpd` manifests are out of scope entirely (not HLS `#EXTM3U`).
|
||||
|
||||
Canonicalization (must match the Go capture side — see mediabuffer.go /
|
||||
main.go where a captured segment's `url` metatag field is always built as
|
||||
`"https://" + host + req.URL.RequestURI()`, i.e. scheme forced to `https`,
|
||||
host + path + query, exactly as observed on the wire):
|
||||
|
||||
`resolve(base_url, uri)` = `urllib.parse.urljoin(base_url, uri)`, and if
|
||||
the result is an absolute `http://` URL, the scheme is rewritten to
|
||||
`https://` before returning. Path and query are left exactly as urljoin
|
||||
produces them (query is never stripped). This is the ONE canonical form
|
||||
used on both sides of the join; `rewrite()` builds its lookup key the
|
||||
same way, so a `mapping` built from segment records' stored `url`
|
||||
values (already `https://...`) matches directly.
|
||||
|
||||
Pure stdlib only (`re`, `urllib.parse`) — no FastAPI, no I/O. Fail-safe:
|
||||
malformed/empty/non-string input never raises; functions degrade to their
|
||||
empty/unsupported-shaped return value instead.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from urllib.parse import urljoin, urlsplit, urlunsplit
|
||||
|
||||
_STREAM_INF_RE = re.compile(r"^#EXT-X-STREAM-INF\b", re.IGNORECASE)
|
||||
_KEY_RE = re.compile(r"^#EXT-X-KEY:(.*)$", re.IGNORECASE)
|
||||
_METHOD_RE = re.compile(r"METHOD=([^,\s]+)", re.IGNORECASE)
|
||||
_MAP_RE = re.compile(r'^#EXT-X-MAP:(.*)$', re.IGNORECASE)
|
||||
_MAP_URI_RE = re.compile(r'URI="([^"]*)"', re.IGNORECASE)
|
||||
|
||||
DEFAULT_MAX_SEGMENTS = 5000
|
||||
|
||||
|
||||
def _lines(text: str) -> list[str]:
|
||||
"""Split playlist text into lines, fail-safe on non-string input."""
|
||||
if not isinstance(text, str) or not text:
|
||||
return []
|
||||
return text.splitlines()
|
||||
|
||||
|
||||
def is_master_playlist(text: str) -> bool:
|
||||
"""True if the playlist is a multivariant/ABR master playlist.
|
||||
|
||||
Detected by the presence of an `#EXT-X-STREAM-INF` tag. Fail-safe:
|
||||
never raises, returns False on malformed/empty/non-string input.
|
||||
"""
|
||||
try:
|
||||
for line in _lines(text):
|
||||
if _STREAM_INF_RE.match(line.strip()):
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def is_encrypted(text: str) -> bool:
|
||||
"""True if any `#EXT-X-KEY` tag has a METHOD other than NONE.
|
||||
|
||||
Fail-safe: never raises, returns False on malformed/empty/non-string
|
||||
input or when no `#EXT-X-KEY` tag is present.
|
||||
"""
|
||||
try:
|
||||
for line in _lines(text):
|
||||
m = _KEY_RE.match(line.strip())
|
||||
if not m:
|
||||
continue
|
||||
method_m = _METHOD_RE.search(m.group(1))
|
||||
if method_m and method_m.group(1).strip().upper() != "NONE":
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def segment_uris(text: str) -> list[str]:
|
||||
"""Ordered list of media segment URIs in a media playlist.
|
||||
|
||||
Includes the `#EXT-X-MAP:URI="..."` init-segment URI first (if present),
|
||||
followed by every non-blank, non-`#`-prefixed line (the `#EXTINF`
|
||||
segment URIs) in order. All other `#`-tag lines are ignored.
|
||||
|
||||
Fail-safe: never raises; returns `[]` on malformed/empty/non-string
|
||||
input.
|
||||
"""
|
||||
out: list[str] = []
|
||||
try:
|
||||
lines = _lines(text)
|
||||
# Init segment (if any) is logically "first" regardless of where
|
||||
# the #EXT-X-MAP tag sits relative to #EXTINF lines.
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
m = _MAP_RE.match(stripped)
|
||||
if m:
|
||||
uri_m = _MAP_URI_RE.search(m.group(1))
|
||||
if uri_m:
|
||||
out.append(uri_m.group(1))
|
||||
break # only one init segment is meaningful
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if stripped.startswith("#"):
|
||||
continue
|
||||
out.append(stripped)
|
||||
except Exception:
|
||||
return []
|
||||
return out
|
||||
|
||||
|
||||
def resolve(base_url: str, uri: str) -> str:
|
||||
"""Resolve a segment URI to an absolute URL against the manifest's URL.
|
||||
|
||||
Uses `urllib.parse.urljoin`, then normalizes an absolute `http://`
|
||||
result to `https://` — the canonical form the Go capture side always
|
||||
stores (`"https://" + host + req.URL.RequestURI()`). Path and query are
|
||||
kept exactly as urljoin produces them.
|
||||
|
||||
Fail-safe: never raises; returns `""` on malformed/non-string input.
|
||||
"""
|
||||
try:
|
||||
if not isinstance(base_url, str) or not isinstance(uri, str):
|
||||
return ""
|
||||
absolute = urljoin(base_url, uri)
|
||||
parts = urlsplit(absolute)
|
||||
if parts.scheme == "http":
|
||||
parts = parts._replace(scheme="https")
|
||||
absolute = urlunsplit(parts)
|
||||
return absolute
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def rewrite(text: str, mapping: dict[str, str], base_url: str,
|
||||
max_segments: int = DEFAULT_MAX_SEGMENTS) -> tuple[str, int, int]:
|
||||
"""Rewrite segment URIs in a media playlist per `mapping`.
|
||||
|
||||
For each segment URI (the `#EXT-X-MAP` init URI and every `#EXTINF`
|
||||
segment line, in order, up to `max_segments`), resolves it against
|
||||
`base_url` and replaces it with `mapping[abs_url]` when present. Only
|
||||
the URI token is replaced — the quoted value for `#EXT-X-MAP`, the
|
||||
whole line for a bare segment URI. Unmatched URIs are left unchanged.
|
||||
|
||||
Returns `(rewritten_text, matched, total)` where `total` is the number
|
||||
of segment URIs seen (capped at `max_segments`) and `matched` is how
|
||||
many were found in `mapping`.
|
||||
|
||||
Fail-safe: never raises; returns `(text or "", 0, 0)` on malformed
|
||||
input.
|
||||
"""
|
||||
try:
|
||||
if not isinstance(text, str) or not text:
|
||||
return (text if isinstance(text, str) else "", 0, 0)
|
||||
if not isinstance(mapping, dict):
|
||||
mapping = {}
|
||||
lines = text.splitlines(keepends=True)
|
||||
out_lines: list[str] = []
|
||||
matched = 0
|
||||
total = 0
|
||||
map_seen = False
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
|
||||
map_m = _MAP_RE.match(stripped) if not map_seen else None
|
||||
if map_m:
|
||||
map_seen = True
|
||||
if total >= max_segments:
|
||||
out_lines.append(line)
|
||||
continue
|
||||
uri_m = _MAP_URI_RE.search(map_m.group(1))
|
||||
if uri_m:
|
||||
total += 1
|
||||
uri = uri_m.group(1)
|
||||
abs_url = resolve(base_url, uri)
|
||||
if abs_url in mapping:
|
||||
matched += 1
|
||||
# Replace only the quoted value inside the line
|
||||
# (locate the exact quoted token, not regex offsets
|
||||
# relative to the group(1) substring).
|
||||
quoted = f'"{uri}"'
|
||||
idx = line.find(quoted)
|
||||
if idx >= 0:
|
||||
value_start = idx + 1
|
||||
value_end = idx + len(quoted) - 1
|
||||
new_line = line[:value_start] + mapping[abs_url] + line[value_end:]
|
||||
out_lines.append(new_line)
|
||||
continue
|
||||
out_lines.append(line)
|
||||
continue
|
||||
|
||||
if stripped and not stripped.startswith("#"):
|
||||
if total >= max_segments:
|
||||
out_lines.append(line)
|
||||
continue
|
||||
total += 1
|
||||
abs_url = resolve(base_url, stripped)
|
||||
if abs_url in mapping:
|
||||
matched += 1
|
||||
# Preserve original line ending / surrounding whitespace
|
||||
# by replacing just the URI token within the line.
|
||||
idx = line.find(stripped)
|
||||
if idx >= 0:
|
||||
new_line = line[:idx] + mapping[abs_url] + line[idx + len(stripped):]
|
||||
else:
|
||||
new_line = mapping[abs_url] + "\n"
|
||||
out_lines.append(new_line)
|
||||
continue
|
||||
out_lines.append(line)
|
||||
continue
|
||||
|
||||
out_lines.append(line)
|
||||
|
||||
return ("".join(out_lines), matched, total)
|
||||
except Exception:
|
||||
return (text if isinstance(text, str) else "", 0, 0)
|
||||
164
common/secubox_core/tests/test_hls.py
Normal file
164
common/secubox_core/tests/test_hls.py
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""Tests for secubox_core.hls — HLS media-playlist parser/rewriter."""
|
||||
from __future__ import annotations
|
||||
|
||||
from secubox_core import hls
|
||||
|
||||
MEDIA_PLAYLIST = """#EXTM3U
|
||||
#EXT-X-VERSION:6
|
||||
#EXT-X-TARGETDURATION:6
|
||||
#EXT-X-PLAYLIST-TYPE:VOD
|
||||
#EXT-X-MAP:URI="init.mp4"
|
||||
#EXTINF:6.006,
|
||||
seg0.ts
|
||||
#EXTINF:6.006,
|
||||
seg1.ts
|
||||
#EXTINF:6.006,
|
||||
seg2.ts
|
||||
#EXT-X-ENDLIST
|
||||
"""
|
||||
|
||||
MASTER_PLAYLIST = """#EXTM3U
|
||||
#EXT-X-STREAM-INF:BANDWIDTH=1280000,RESOLUTION=640x360
|
||||
low/index.m3u8
|
||||
#EXT-X-STREAM-INF:BANDWIDTH=2560000,RESOLUTION=1280x720
|
||||
high/index.m3u8
|
||||
"""
|
||||
|
||||
ENCRYPTED_PLAYLIST = """#EXTM3U
|
||||
#EXT-X-VERSION:6
|
||||
#EXT-X-KEY:METHOD=AES-128,URI="key.bin",IV=0x0123456789abcdef0123456789abcdef
|
||||
#EXTINF:6.006,
|
||||
seg0.ts
|
||||
#EXT-X-ENDLIST
|
||||
"""
|
||||
|
||||
UNENCRYPTED_KEY_PLAYLIST = """#EXTM3U
|
||||
#EXT-X-KEY:METHOD=NONE
|
||||
#EXTINF:6.006,
|
||||
seg0.ts
|
||||
#EXT-X-ENDLIST
|
||||
"""
|
||||
|
||||
BASE_URL = "https://h/hls/index.m3u8"
|
||||
|
||||
|
||||
def test_segment_uris_ordered_with_map_first():
|
||||
assert hls.segment_uris(MEDIA_PLAYLIST) == [
|
||||
"init.mp4", "seg0.ts", "seg1.ts", "seg2.ts",
|
||||
]
|
||||
|
||||
|
||||
def test_segment_uris_no_map():
|
||||
text = "#EXTM3U\n#EXTINF:6,\nseg0.ts\n#EXT-X-ENDLIST\n"
|
||||
assert hls.segment_uris(text) == ["seg0.ts"]
|
||||
|
||||
|
||||
def test_resolve_relative():
|
||||
assert hls.resolve(BASE_URL, "seg0.ts") == "https://h/hls/seg0.ts"
|
||||
|
||||
|
||||
def test_resolve_keeps_query():
|
||||
abs_url = hls.resolve(BASE_URL, "https://h/hls/seg0.ts?x=1")
|
||||
assert abs_url == "https://h/hls/seg0.ts?x=1"
|
||||
|
||||
|
||||
def test_resolve_normalizes_http_to_https():
|
||||
abs_url = hls.resolve(BASE_URL, "http://h/hls/seg0.ts")
|
||||
assert abs_url == "https://h/hls/seg0.ts"
|
||||
|
||||
|
||||
def test_rewrite_partial_mapping():
|
||||
mapping = {
|
||||
"https://h/hls/init.mp4": "/api/v1/dpi/media/replay/aaa",
|
||||
"https://h/hls/seg0.ts": "/api/v1/dpi/media/replay/bbb",
|
||||
}
|
||||
rewritten, matched, total = hls.rewrite(MEDIA_PLAYLIST, mapping, BASE_URL)
|
||||
assert matched == 2
|
||||
assert total == 4
|
||||
assert '#EXT-X-MAP:URI="/api/v1/dpi/media/replay/aaa"' in rewritten
|
||||
assert "/api/v1/dpi/media/replay/bbb" in rewritten
|
||||
assert "seg1.ts" in rewritten # unmapped, unchanged
|
||||
assert "seg2.ts" in rewritten # unmapped, unchanged
|
||||
assert "seg0.ts" not in rewritten # mapped, replaced
|
||||
|
||||
|
||||
def test_rewrite_no_matches_leaves_text_shape():
|
||||
rewritten, matched, total = hls.rewrite(MEDIA_PLAYLIST, {}, BASE_URL)
|
||||
assert matched == 0
|
||||
assert total == 4
|
||||
assert hls.segment_uris(rewritten) == hls.segment_uris(MEDIA_PLAYLIST)
|
||||
|
||||
|
||||
def test_rewrite_caps_at_max_segments():
|
||||
lines = ["#EXTM3U"]
|
||||
for i in range(10):
|
||||
lines.append("#EXTINF:6,")
|
||||
lines.append(f"seg{i}.ts")
|
||||
lines.append("#EXT-X-ENDLIST")
|
||||
text = "\n".join(lines) + "\n"
|
||||
mapping = {f"https://h/hls/seg{i}.ts": f"/r/{i}" for i in range(10)}
|
||||
rewritten, matched, total = hls.rewrite(text, mapping, BASE_URL,
|
||||
max_segments=3)
|
||||
assert total == 3
|
||||
assert matched == 3
|
||||
# Segments beyond the cap are left untouched.
|
||||
assert "seg9.ts" in rewritten
|
||||
|
||||
|
||||
def test_is_master_playlist_true():
|
||||
assert hls.is_master_playlist(MASTER_PLAYLIST) is True
|
||||
|
||||
|
||||
def test_is_master_playlist_false_on_media_playlist():
|
||||
assert hls.is_master_playlist(MEDIA_PLAYLIST) is False
|
||||
|
||||
|
||||
def test_is_encrypted_true():
|
||||
assert hls.is_encrypted(ENCRYPTED_PLAYLIST) is True
|
||||
|
||||
|
||||
def test_is_encrypted_false_on_method_none():
|
||||
assert hls.is_encrypted(UNENCRYPTED_KEY_PLAYLIST) is False
|
||||
|
||||
|
||||
def test_is_encrypted_false_on_no_key():
|
||||
assert hls.is_encrypted(MEDIA_PLAYLIST) is False
|
||||
|
||||
|
||||
def test_fail_safe_empty_string():
|
||||
assert hls.is_master_playlist("") is False
|
||||
assert hls.is_encrypted("") is False
|
||||
assert hls.segment_uris("") == []
|
||||
rewritten, matched, total = hls.rewrite("", {}, BASE_URL)
|
||||
assert rewritten == ""
|
||||
assert matched == 0
|
||||
assert total == 0
|
||||
|
||||
|
||||
def test_fail_safe_garbage_input():
|
||||
# All non-blank lines are tag lines ("#"-prefixed) — no bare segment
|
||||
# URIs — so parsing degrades cleanly to empty results, never raises.
|
||||
garbage = '#EXT-X-??? \x00\xff garbled\n#EXT-X-KEY:GARBAGE\n\n'
|
||||
assert hls.is_master_playlist(garbage) is False
|
||||
assert hls.is_encrypted(garbage) is False
|
||||
assert hls.segment_uris(garbage) == []
|
||||
rewritten, matched, total = hls.rewrite(garbage, {"x": "y"}, BASE_URL)
|
||||
assert rewritten == garbage
|
||||
assert matched == 0
|
||||
assert total == 0
|
||||
|
||||
|
||||
def test_fail_safe_non_string_types_never_raise():
|
||||
assert hls.is_master_playlist(None) is False # type: ignore[arg-type]
|
||||
assert hls.is_encrypted(None) is False # type: ignore[arg-type]
|
||||
assert hls.segment_uris(None) == [] # type: ignore[arg-type]
|
||||
rewritten, matched, total = hls.rewrite(None, {}, BASE_URL) # type: ignore[arg-type]
|
||||
assert rewritten == ""
|
||||
assert matched == 0
|
||||
assert total == 0
|
||||
assert hls.resolve(BASE_URL, None) == "" # type: ignore[arg-type]
|
||||
|
|
@ -165,9 +165,11 @@ import re # noqa: E402
|
|||
import glob # noqa: E402
|
||||
from datetime import timezone # noqa: E402
|
||||
from fastapi import Request # noqa: E402
|
||||
from fastapi import Response # noqa: E402
|
||||
from fastapi.responses import FileResponse # noqa: E402
|
||||
from secubox_core import media_buffer # noqa: E402
|
||||
from secubox_core import user_store # noqa: E402
|
||||
from secubox_core import hls # noqa: E402
|
||||
|
||||
MEDIA_BUFFER_ROOT = "/data/secubox/media-buffer"
|
||||
AUDIT_LOG = "/var/log/secubox/audit.log"
|
||||
|
|
@ -258,6 +260,111 @@ def _audit_replay(sub: str, rec_id: str, host: str, ip: str) -> None:
|
|||
pass
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Phase 2 (#812) — HLS manifest reassembly.
|
||||
#
|
||||
# Segments are ordinary Phase-1 buffer objects (kind="segment") served
|
||||
# unchanged by GET /media/replay/{id}. When the requested record is a
|
||||
# manifest, media_replay() parses the stored playlist bytes with
|
||||
# secubox_core.hls and rewrites each segment URI to point at the matching
|
||||
# captured segment's replay URL — a pure read-time join by absolute URL,
|
||||
# never live cross-request session state (see the Phase 2 plan's Global
|
||||
# Constraints).
|
||||
# ============================================================================
|
||||
MAX_MANIFEST_BYTES = 8 * 1024 * 1024
|
||||
MAX_SEGMENT_INDEX = 5000
|
||||
MAX_MANIFEST_SEGMENTS = 5000
|
||||
|
||||
|
||||
def _segment_index(mac_hash: Optional[str], host: Optional[str]) -> Dict[str, str]:
|
||||
"""Map a captured segment's absolute `url` -> its record `id`.
|
||||
|
||||
Scoped to the SAME mac_hash + host as the manifest being replayed (never
|
||||
joins across personas/hosts); only live (non-expired) `kind=="segment"`
|
||||
records are eligible. Bounded to MAX_SEGMENT_INDEX entries so a
|
||||
pathological session can't blow up the join. Fail-empty: any read error
|
||||
yields {}.
|
||||
"""
|
||||
try:
|
||||
records = media_buffer.read_records(mac_hash=mac_hash, path=_media_log_path())
|
||||
except Exception:
|
||||
return {}
|
||||
out: Dict[str, str] = {}
|
||||
try:
|
||||
for rec in records:
|
||||
if not isinstance(rec, dict):
|
||||
continue
|
||||
if rec.get("kind") != "segment":
|
||||
continue
|
||||
if rec.get("expired"):
|
||||
continue
|
||||
if rec.get("host") != host:
|
||||
continue
|
||||
url = rec.get("url")
|
||||
seg_id = rec.get("id")
|
||||
if not url or not seg_id:
|
||||
continue
|
||||
out[url] = seg_id
|
||||
if len(out) >= MAX_SEGMENT_INDEX:
|
||||
log.warning(
|
||||
"dpi media: segment index for %s capped at %d",
|
||||
host, MAX_SEGMENT_INDEX,
|
||||
)
|
||||
break
|
||||
except Exception:
|
||||
return out
|
||||
return out
|
||||
|
||||
|
||||
def _replay_manifest(rec: dict, path: str) -> Optional[Response]:
|
||||
"""Manifest replay branch: parse the captured playlist and rewrite
|
||||
segment URIs to the matching captured segments' replay URLs.
|
||||
|
||||
Master/multivariant (ABR) and encrypted (#EXT-X-KEY) playlists are out of
|
||||
scope for Phase 2 — the raw manifest is returned unchanged with an
|
||||
`X-SecuBox-Media: unsupported-variant` header rather than attempting a
|
||||
broken rewrite.
|
||||
|
||||
Fail-safe: any parse/read error returns None so the caller falls back to
|
||||
the Phase-1 raw FileResponse — this branch must NEVER 500.
|
||||
"""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read(MAX_MANIFEST_BYTES + 1)
|
||||
if len(raw) > MAX_MANIFEST_BYTES:
|
||||
log.warning(
|
||||
"dpi media: manifest %s truncated at %d bytes",
|
||||
rec.get("id") or rec.get("url") or "?",
|
||||
MAX_MANIFEST_BYTES,
|
||||
)
|
||||
raw = raw[:MAX_MANIFEST_BYTES]
|
||||
text = raw.decode("utf-8", errors="replace")
|
||||
|
||||
if hls.is_master_playlist(text) or hls.is_encrypted(text):
|
||||
return Response(content=text, media_type="application/vnd.apple.mpegurl",
|
||||
headers={"X-SecuBox-Media": "unsupported-variant"})
|
||||
|
||||
mapping = {
|
||||
seg_url: f"/api/v1/dpi/media/replay/{seg_id}"
|
||||
for seg_url, seg_id in _segment_index(rec.get("mac_hash"), rec.get("host")).items()
|
||||
}
|
||||
rewritten, matched, total = hls.rewrite(
|
||||
text, mapping, rec.get("url") or "", max_segments=MAX_MANIFEST_SEGMENTS
|
||||
)
|
||||
if total >= MAX_MANIFEST_SEGMENTS:
|
||||
log.warning(
|
||||
"dpi media: manifest rewrite hit segment cap %d (total=%d matched=%d)",
|
||||
MAX_MANIFEST_SEGMENTS, total, matched,
|
||||
)
|
||||
return Response(
|
||||
content=rewritten,
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
headers={"X-SecuBox-Media": f"hls-reassembled; matched={matched}; total={total}"},
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
class QuotaType(str, Enum):
|
||||
DAILY = "daily"
|
||||
WEEKLY = "weekly"
|
||||
|
|
@ -553,6 +660,13 @@ def media_replay(rec_id: str, request: Request = None,
|
|||
410 once the janitor has evicted the bytes (metatag-only). The object path
|
||||
is resolved from the RECORD's session_id under MEDIA_BUFFER_ROOT — never
|
||||
from `rec_id`, which is additionally validated against a strict hex regex.
|
||||
|
||||
Phase 2 (#812): when the record is a captured HLS manifest
|
||||
(kind=="manifest"), the response is a REWRITTEN playlist whose segment
|
||||
URIs point at their matching captured `kind=="segment"` records' replay
|
||||
URLs (see `_replay_manifest`) instead of the raw manifest bytes. Every
|
||||
other kind (video/audio/file/segment) keeps the unchanged Phase-1
|
||||
FileResponse path.
|
||||
"""
|
||||
if not _REC_ID_RE.match(rec_id or ""):
|
||||
raise HTTPException(status_code=400, detail="invalid record id")
|
||||
|
|
@ -569,6 +683,15 @@ def media_replay(rec_id: str, request: Request = None,
|
|||
ip = request.client.host or ""
|
||||
except Exception:
|
||||
ip = ""
|
||||
|
||||
if rec.get("kind") == "manifest":
|
||||
manifest_resp = _replay_manifest(rec, path)
|
||||
if manifest_resp is not None:
|
||||
_audit_replay(sub, rec_id, rec.get("host") or "", ip)
|
||||
return manifest_resp
|
||||
# Fail-safe: parse/read error in the manifest branch falls through to
|
||||
# the raw Phase-1 FileResponse below — never a 500.
|
||||
|
||||
_audit_replay(sub, rec_id, rec.get("host") or "", ip)
|
||||
return FileResponse(path, media_type=rec.get("ctype") or "application/octet-stream")
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,15 @@
|
|||
secubox-dpi (1.2.0-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* #812 Media Buffer — DPI side. Phase 1: media-buffer list/replay/thumb API
|
||||
(admin/owner-gated, audited, 410-on-evict, hex-id + realpath traversal-safe,
|
||||
all plain def) + a Media gallery tab in the dashboard (escaped fields, sbx_token,
|
||||
expired greying). Phase 2 (HLS): manifest replay rewrites captured segment URIs to
|
||||
their replay URLs (read-time URL join); master/ABR + AES-encrypted playlists detected
|
||||
and served raw as unsupported-variant; segments hidden from the gallery. Truncation
|
||||
of oversized manifests/segment-indexes is logged.
|
||||
|
||||
-- Gerald KERMA <devel@cybermind.fr> Sat, 04 Jul 2026 16:30:00 +0000
|
||||
|
||||
secubox-dpi (1.1.4-1~bookworm1) bookworm; urgency=low
|
||||
|
||||
* #720 Phase 3 — per-device DAILY history buckets (collector history.json, 14d)
|
||||
|
|
|
|||
203
packages/secubox-dpi/tests/test_hls_e2e.py
Normal file
203
packages/secubox-dpi/tests/test_hls_e2e.py
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||
# See LICENCE-CMSD-1.0.md for terms.
|
||||
|
||||
"""End-to-end HLS reassembly test (Phase 2, ref #812).
|
||||
|
||||
`test_media_buffer_api.py` already unit-tests the manifest-replay branch in
|
||||
isolation. This module is the CROSS-LAYER integration check the Phase-1
|
||||
whole-branch review flagged as missing: it seeds a temp media-buffer that
|
||||
mimics exactly what the Go tee + janitor write to disk (a metatag JSONL log
|
||||
+ per-session `object-0.*` files), then drives the REAL `media_replay`
|
||||
handler through the full chain:
|
||||
|
||||
capture (metatag + object bytes)
|
||||
-> join (manifest url + segment urls, same mac_hash/host)
|
||||
-> rewrite (secubox_core.hls)
|
||||
-> per-segment replay (FileResponse, byte-identical to the capture)
|
||||
-> janitor eviction flip (append-only, last-line-wins)
|
||||
|
||||
Handlers are plain `def` (aggregator-mounted; ref #808) so we call them
|
||||
directly, exactly like `test_media_buffer_api.py`.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # make api importable
|
||||
from fastapi import HTTPException # noqa: E402
|
||||
from fastapi.responses import FileResponse # noqa: E402
|
||||
from api import main as m # noqa: E402
|
||||
|
||||
ADMIN = {"role": "admin", "sub": "root"}
|
||||
|
||||
# Hex-conformant ids/session ids (^[0-9a-f]{8,32}$) — session_id doubles as
|
||||
# the on-disk directory name and is itself validated against the same regex
|
||||
# by `_resolve_object_path`, so both must stay pure hex.
|
||||
MANIFEST_ID = "e5e5111122223333"
|
||||
MANIFEST_SESSION = "e5e5111122223333aaaa"
|
||||
SEG0_ID = "e5e5aaaa11112222"
|
||||
SEG0_SESSION = "e5e5aaaa11112222bbbb"
|
||||
SEG1_ID = "e5e5bbbb11112222"
|
||||
SEG1_SESSION = "e5e5bbbb11112222cccc"
|
||||
SEG2_ID = "e5e5cccc11112222"
|
||||
SEG2_SESSION = "e5e5cccc11112222dddd"
|
||||
|
||||
HOST = "h"
|
||||
MAC_HASH = "M"
|
||||
|
||||
# A realistic 3-segment VOD media playlist — the exact shape sbxmitm's tee
|
||||
# would have captured as the manifest's buffer object.
|
||||
MEDIA_PLAYLIST = (
|
||||
"#EXTM3U\n"
|
||||
"#EXT-X-VERSION:3\n"
|
||||
"#EXT-X-TARGETDURATION:6\n"
|
||||
"#EXT-X-PLAYLIST-TYPE:VOD\n"
|
||||
"#EXTINF:6.000,\n"
|
||||
"seg0.ts\n"
|
||||
"#EXTINF:6.000,\n"
|
||||
"seg1.ts\n"
|
||||
"#EXTINF:6.000,\n"
|
||||
"seg2.ts\n"
|
||||
"#EXT-X-ENDLIST\n"
|
||||
)
|
||||
|
||||
SEG0_BYTES = b"segment-zero-payload-0123456789"
|
||||
SEG1_BYTES = b"segment-one-payload-abcdefghij"
|
||||
SEG2_BYTES = b"segment-two-payload-ZZZZZZZZZZ"
|
||||
|
||||
_REPLAY_URL_RE = re.compile(r"/api/v1/dpi/media/replay/([0-9a-f]{8,32})")
|
||||
|
||||
|
||||
def _write_object(tmp_path, session_id, filename, content):
|
||||
"""Write <tmp_path>/<session_id>/<filename>, mimicking the buffer-object
|
||||
layout the Go tee writes under MEDIA_BUFFER_ROOT."""
|
||||
obj_dir = tmp_path / session_id
|
||||
obj_dir.mkdir(exist_ok=True)
|
||||
if isinstance(content, str):
|
||||
content = content.encode("utf-8")
|
||||
(obj_dir / filename).write_bytes(content)
|
||||
|
||||
|
||||
def _rec(rec_id, session_id, url, kind, ctype, expired=False, buffer_ref=None):
|
||||
"""Build one metatag-log record, matching the shape sbxmitm/the janitor
|
||||
append to media-buffer.jsonl (see secubox_core.media_buffer)."""
|
||||
return {
|
||||
"id": rec_id, "session_id": session_id, "first_ts": 10, "last_ts": 10,
|
||||
"mac_hash": MAC_HASH, "host": HOST, "url": url, "direction": "down",
|
||||
"kind": kind, "ctype": ctype, "bytes": len(url), "segments": 0,
|
||||
"truncated": False,
|
||||
"buffer_ref": buffer_ref if buffer_ref is not None else (None if expired else session_id),
|
||||
"expired": expired,
|
||||
}
|
||||
|
||||
|
||||
def _seed_full_capture(tmp_path):
|
||||
"""Seed a temp MEDIA_BUFFER_ROOT with a manifest + 3 live matching
|
||||
segments — the state right after capture, before any eviction."""
|
||||
log = tmp_path / "media-buffer.jsonl"
|
||||
manifest = _rec(MANIFEST_ID, MANIFEST_SESSION, "https://h/hls/index.m3u8",
|
||||
kind="manifest", ctype="application/vnd.apple.mpegurl")
|
||||
seg0 = _rec(SEG0_ID, SEG0_SESSION, "https://h/hls/seg0.ts",
|
||||
kind="segment", ctype="video/mp2t")
|
||||
seg1 = _rec(SEG1_ID, SEG1_SESSION, "https://h/hls/seg1.ts",
|
||||
kind="segment", ctype="video/mp2t")
|
||||
seg2 = _rec(SEG2_ID, SEG2_SESSION, "https://h/hls/seg2.ts",
|
||||
kind="segment", ctype="video/mp2t")
|
||||
log.write_text("\n".join(json.dumps(r) for r in (manifest, seg0, seg1, seg2)) + "\n")
|
||||
|
||||
_write_object(tmp_path, MANIFEST_SESSION, "object-0.m3u8", MEDIA_PLAYLIST)
|
||||
_write_object(tmp_path, SEG0_SESSION, "object-0.ts", SEG0_BYTES)
|
||||
_write_object(tmp_path, SEG1_SESSION, "object-0.ts", SEG1_BYTES)
|
||||
_write_object(tmp_path, SEG2_SESSION, "object-0.ts", SEG2_BYTES)
|
||||
return log
|
||||
|
||||
|
||||
def _append_janitor_eviction_flip(log_path, rec_id, session_id, url):
|
||||
"""Append-only eviction flip, exactly as the janitor does: a fresh line
|
||||
for the SAME id with expired=true/buffer_ref=null — never rewrites
|
||||
history. `_deduped_records` in secubox_core.media_buffer is last-line-wins
|
||||
by id, so this is what actually flips the record for every reader."""
|
||||
flip = _rec(rec_id, session_id, url, kind="segment", ctype="video/mp2t",
|
||||
expired=True, buffer_ref=None)
|
||||
with open(log_path, "a") as f:
|
||||
f.write(json.dumps(flip) + "\n")
|
||||
|
||||
|
||||
def test_hls_e2e_full_chain_capture_join_rewrite_replay(tmp_path, monkeypatch):
|
||||
"""Capture -> join -> rewrite -> per-segment replay, all live."""
|
||||
_seed_full_capture(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
resp = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert not isinstance(resp, FileResponse)
|
||||
assert resp.media_type == "application/vnd.apple.mpegurl"
|
||||
body = resp.body.decode("utf-8")
|
||||
|
||||
header = resp.headers.get("x-secubox-media") or resp.headers.get("X-SecuBox-Media")
|
||||
assert header is not None
|
||||
assert "hls-reassembled" in header
|
||||
assert "matched=3" in header
|
||||
assert "total=3" in header
|
||||
|
||||
# Every original segment URI was rewritten — none of the raw names survive.
|
||||
assert "seg0.ts" not in body
|
||||
assert "seg1.ts" not in body
|
||||
assert "seg2.ts" not in body
|
||||
|
||||
rewritten_ids = _REPLAY_URL_RE.findall(body)
|
||||
assert rewritten_ids == [SEG0_ID, SEG1_ID, SEG2_ID]
|
||||
|
||||
expected_bytes = {SEG0_ID: SEG0_BYTES, SEG1_ID: SEG1_BYTES, SEG2_ID: SEG2_BYTES}
|
||||
|
||||
# Drive the REAL per-segment replay for every rewritten URL: this proves
|
||||
# the whole capture -> metatag -> join -> rewrite -> segment-replay chain,
|
||||
# not just that the URL string looks right.
|
||||
for seg_id in rewritten_ids:
|
||||
seg_resp = m.media_replay(seg_id, request=None, user=ADMIN)
|
||||
assert isinstance(seg_resp, FileResponse)
|
||||
assert Path(seg_resp.path).read_bytes() == expected_bytes[seg_id]
|
||||
|
||||
|
||||
def test_hls_e2e_eviction_leaves_segment_unrewritten_and_410s(tmp_path, monkeypatch):
|
||||
"""Janitor evicts seg2 (append-only flip) after capture: re-replaying the
|
||||
manifest must leave seg2.ts untouched (no live index entry) while seg0/
|
||||
seg1 still rewrite and replay correctly; replaying seg2's id now 410s."""
|
||||
log = _seed_full_capture(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
# Sanity: before the flip, all 3 are live (mirrors the previous test).
|
||||
resp_before = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert "seg2.ts" not in resp_before.body.decode("utf-8")
|
||||
|
||||
_append_janitor_eviction_flip(log, SEG2_ID, SEG2_SESSION, "https://h/hls/seg2.ts")
|
||||
|
||||
resp_after = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
body_after = resp_after.body.decode("utf-8")
|
||||
header_after = (resp_after.headers.get("x-secubox-media")
|
||||
or resp_after.headers.get("X-SecuBox-Media"))
|
||||
assert "matched=2" in header_after
|
||||
assert "total=3" in header_after
|
||||
|
||||
# seg0/seg1 still rewritten and still replay to their captured bytes.
|
||||
assert f"/api/v1/dpi/media/replay/{SEG0_ID}" in body_after
|
||||
assert f"/api/v1/dpi/media/replay/{SEG1_ID}" in body_after
|
||||
seg0_resp = m.media_replay(SEG0_ID, request=None, user=ADMIN)
|
||||
assert Path(seg0_resp.path).read_bytes() == SEG0_BYTES
|
||||
seg1_resp = m.media_replay(SEG1_ID, request=None, user=ADMIN)
|
||||
assert Path(seg1_resp.path).read_bytes() == SEG1_BYTES
|
||||
|
||||
# seg2 dropped out of the live index -> left UNCHANGED (not rewritten,
|
||||
# not in the rewritten playlist's live index) — partial playback, not a
|
||||
# broken rewrite.
|
||||
assert "seg2.ts" in body_after
|
||||
assert f"/api/v1/dpi/media/replay/{SEG2_ID}" not in body_after
|
||||
|
||||
# And replaying seg2's own id directly now 410s (metatag-only, evicted).
|
||||
with pytest.raises(HTTPException) as e:
|
||||
m.media_replay(SEG2_ID, request=None, user=ADMIN)
|
||||
assert e.value.status_code == 410
|
||||
|
|
@ -135,3 +135,203 @@ def test_thumb_missing_returns_404(tmp_path, monkeypatch):
|
|||
with pytest.raises(HTTPException) as e2:
|
||||
m.media_thumb("../x", user=ADMIN)
|
||||
assert e2.value.status_code == 400
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Phase 2 (#812) — HLS manifest reassembly branch.
|
||||
# ============================================================================
|
||||
|
||||
MANIFEST_ID = "aaaa11112222"
|
||||
MANIFEST_SESSION = "aaaa1111222233334444"
|
||||
SEG0_ID = "bbbb11112222"
|
||||
SEG0_SESSION = "bbbb1111222233334444"
|
||||
SEG1_ID = "cccc11112222"
|
||||
SEG1_SESSION = "cccc1111222233334444"
|
||||
SEG2_EXP_ID = "dddd11112222"
|
||||
SEG2_EXP_SESSION = "dddd1111222233334444"
|
||||
|
||||
MEDIA_PLAYLIST = (
|
||||
"#EXTM3U\n"
|
||||
"#EXT-X-VERSION:3\n"
|
||||
"#EXTINF:10.0,\n"
|
||||
"seg0.ts\n"
|
||||
"#EXTINF:10.0,\n"
|
||||
"seg1.ts\n"
|
||||
"#EXTINF:10.0,\n"
|
||||
"seg2.ts\n"
|
||||
"#EXT-X-ENDLIST\n"
|
||||
)
|
||||
|
||||
MASTER_PLAYLIST = (
|
||||
"#EXTM3U\n"
|
||||
"#EXT-X-STREAM-INF:BANDWIDTH=1000000\n"
|
||||
"variant.m3u8\n"
|
||||
)
|
||||
|
||||
ENCRYPTED_PLAYLIST = (
|
||||
"#EXTM3U\n"
|
||||
"#EXT-X-VERSION:3\n"
|
||||
'#EXT-X-KEY:METHOD=AES-128,URI="key.bin"\n'
|
||||
"#EXTINF:10.0,\n"
|
||||
"seg0.ts\n"
|
||||
"#EXT-X-ENDLIST\n"
|
||||
)
|
||||
|
||||
|
||||
def _write_object(tmp_path, session_id, filename, content):
|
||||
obj_dir = tmp_path / session_id
|
||||
obj_dir.mkdir()
|
||||
if isinstance(content, str):
|
||||
content = content.encode("utf-8")
|
||||
(obj_dir / filename).write_bytes(content)
|
||||
|
||||
|
||||
def _hls_record(rec_id, session_id, url, kind="segment", host="h",
|
||||
mac_hash="m1", ctype="video/mp2t", expired=False,
|
||||
buffer_ref=None):
|
||||
return {"id": rec_id, "session_id": session_id, "first_ts": 3, "last_ts": 3,
|
||||
"mac_hash": mac_hash, "host": host, "url": url, "direction": "down",
|
||||
"kind": kind, "ctype": ctype, "bytes": 5, "segments": 0,
|
||||
"truncated": False,
|
||||
"buffer_ref": buffer_ref if buffer_ref is not None else (None if expired else session_id),
|
||||
"expired": expired}
|
||||
|
||||
|
||||
def _seed_hls_buffer(tmp_path, manifest_text=MEDIA_PLAYLIST):
|
||||
"""Seed a manifest + 2 live matching segments + 1 expired segment whose
|
||||
url would otherwise match seg2.ts."""
|
||||
log = tmp_path / "media-buffer.jsonl"
|
||||
manifest = _hls_record(MANIFEST_ID, MANIFEST_SESSION,
|
||||
"https://h/hls/index.m3u8", kind="manifest",
|
||||
ctype="application/vnd.apple.mpegurl")
|
||||
seg0 = _hls_record(SEG0_ID, SEG0_SESSION, "https://h/hls/seg0.ts")
|
||||
seg1 = _hls_record(SEG1_ID, SEG1_SESSION, "https://h/hls/seg1.ts")
|
||||
seg2_expired = _hls_record(SEG2_EXP_ID, SEG2_EXP_SESSION,
|
||||
"https://h/hls/seg2.ts", expired=True)
|
||||
lines = [json.dumps(r) for r in (manifest, seg0, seg1, seg2_expired)]
|
||||
log.write_text("\n".join(lines) + "\n")
|
||||
|
||||
_write_object(tmp_path, MANIFEST_SESSION, "object-0.m3u8", manifest_text)
|
||||
_write_object(tmp_path, SEG0_SESSION, "object-0.ts", b"seg0-bytes")
|
||||
_write_object(tmp_path, SEG1_SESSION, "object-0.ts", b"seg1-bytes")
|
||||
# No object written for the expired segment — it's metatag-only.
|
||||
return log
|
||||
|
||||
|
||||
def test_manifest_replay_rewrites_live_segments_only(tmp_path, monkeypatch):
|
||||
_seed_hls_buffer(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
resp = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert not isinstance(resp, FileResponse)
|
||||
assert resp.media_type == "application/vnd.apple.mpegurl"
|
||||
body = resp.body.decode("utf-8")
|
||||
|
||||
assert f"/api/v1/dpi/media/replay/{SEG0_ID}" in body
|
||||
assert f"/api/v1/dpi/media/replay/{SEG1_ID}" in body
|
||||
assert "seg0.ts" not in body
|
||||
assert "seg1.ts" not in body
|
||||
# Expired segment has no live index entry -> left unchanged.
|
||||
assert "seg2.ts" in body
|
||||
|
||||
header = resp.headers.get("x-secubox-media") or resp.headers.get("X-SecuBox-Media")
|
||||
assert header is not None
|
||||
assert "hls-reassembled" in header
|
||||
assert "matched=2" in header
|
||||
assert "total=3" in header
|
||||
|
||||
|
||||
def test_manifest_replay_master_playlist_returns_raw_unsupported(tmp_path, monkeypatch):
|
||||
_seed_hls_buffer(tmp_path, manifest_text=MASTER_PLAYLIST)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
resp = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert not isinstance(resp, FileResponse)
|
||||
assert resp.media_type == "application/vnd.apple.mpegurl"
|
||||
assert resp.body.decode("utf-8") == MASTER_PLAYLIST
|
||||
header = resp.headers.get("x-secubox-media") or resp.headers.get("X-SecuBox-Media")
|
||||
assert header == "unsupported-variant"
|
||||
|
||||
|
||||
def test_manifest_replay_encrypted_returns_raw_unsupported(tmp_path, monkeypatch):
|
||||
_seed_hls_buffer(tmp_path, manifest_text=ENCRYPTED_PLAYLIST)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
resp = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert not isinstance(resp, FileResponse)
|
||||
assert resp.body.decode("utf-8") == ENCRYPTED_PLAYLIST
|
||||
header = resp.headers.get("x-secubox-media") or resp.headers.get("X-SecuBox-Media")
|
||||
assert header == "unsupported-variant"
|
||||
|
||||
|
||||
def test_segment_kind_replay_still_uses_phase1_fileresponse(tmp_path, monkeypatch):
|
||||
_seed_hls_buffer(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
resp = m.media_replay(SEG0_ID, request=None, user=ADMIN)
|
||||
assert isinstance(resp, FileResponse)
|
||||
assert resp.path.endswith("object-0.ts")
|
||||
assert Path(resp.path).read_bytes() == b"seg0-bytes"
|
||||
|
||||
|
||||
def test_manifest_replay_phase1_invariants_hold(tmp_path, monkeypatch):
|
||||
"""Bad id -> 400, expired manifest/segment -> 410 still hold for
|
||||
manifest/segment records exactly as for any other kind. The admin/owner
|
||||
gate itself (`require_admin_or_owner`) is dependency-injected by FastAPI
|
||||
ahead of the handler body — its 403-on-non-admin behavior is kind-agnostic
|
||||
and already covered by test_require_admin_or_owner_rejects_nonadmin; it
|
||||
applies identically here since the route declares no kind-specific
|
||||
override."""
|
||||
_seed_hls_buffer(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
with pytest.raises(HTTPException) as e:
|
||||
m.media_replay("not-hex!", request=None, user=ADMIN)
|
||||
assert e.value.status_code == 400
|
||||
|
||||
with pytest.raises(HTTPException) as e2:
|
||||
m.media_replay(SEG2_EXP_ID, request=None, user=ADMIN)
|
||||
assert e2.value.status_code == 410
|
||||
|
||||
with pytest.raises(HTTPException) as e3:
|
||||
m.require_admin_or_owner(user=NONADMIN)
|
||||
assert e3.value.status_code == 403
|
||||
|
||||
|
||||
def test_manifest_replay_fallback_on_read_error(tmp_path, monkeypatch):
|
||||
"""A read/parse error inside `_replay_manifest` (e.g. the manifest object
|
||||
becomes unreadable after the record/path lookup already succeeded) must
|
||||
NEVER bubble up as a 500 — `_replay_manifest` catches it and returns
|
||||
None, and `media_replay` falls back to the Phase-1 raw FileResponse."""
|
||||
_seed_hls_buffer(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise OSError("simulated read failure")
|
||||
|
||||
# `_replay_manifest` calls builtin `open()` directly; shadowing it on the
|
||||
# module namespace only affects that call site, not media_buffer's own
|
||||
# (separate-module) reads of the metatag log.
|
||||
monkeypatch.setattr(m, "open", _boom, raising=False)
|
||||
|
||||
resp = m.media_replay(MANIFEST_ID, request=None, user=ADMIN)
|
||||
assert isinstance(resp, FileResponse)
|
||||
assert resp.path.endswith("object-0.m3u8")
|
||||
|
||||
|
||||
def test_manifest_branch_does_not_swallow_denials(tmp_path, monkeypatch):
|
||||
"""Guard: `_replay_manifest`'s broad `except Exception: return None`
|
||||
fail-safe must never turn a *denial* into a 200. A bad `rec_id` and an
|
||||
expired/evicted manifest-or-segment record both raise their
|
||||
HTTPException BEFORE `media_replay` ever calls `_replay_manifest`, so
|
||||
the try/except added for the read-truncation logging cannot mask them."""
|
||||
_seed_hls_buffer(tmp_path)
|
||||
monkeypatch.setattr(m, "MEDIA_BUFFER_ROOT", str(tmp_path))
|
||||
|
||||
with pytest.raises(HTTPException) as e:
|
||||
m.media_replay("not-hex!", request=None, user=ADMIN)
|
||||
assert e.value.status_code == 400
|
||||
|
||||
with pytest.raises(HTTPException) as e2:
|
||||
m.media_replay(SEG2_EXP_ID, request=None, user=ADMIN)
|
||||
assert e2.value.status_code == 410
|
||||
|
|
|
|||
|
|
@ -335,6 +335,18 @@
|
|||
color: var(--text-dim);
|
||||
font-style: italic;
|
||||
}
|
||||
.media-note {
|
||||
font-size: 0.68rem;
|
||||
color: var(--text-dim);
|
||||
font-style: italic;
|
||||
margin-top: 0.15rem;
|
||||
}
|
||||
.media-note.warn { color: var(--red); }
|
||||
.media-actions .btn.disabled {
|
||||
opacity: 0.45;
|
||||
pointer-events: none;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.main { margin-left: 0; }
|
||||
|
|
@ -845,6 +857,18 @@
|
|||
metaRow2.appendChild(ageSpan);
|
||||
card.appendChild(metaRow2);
|
||||
|
||||
// #812 Phase 2: manifest segment-count hint. `item.segments` is a
|
||||
// display hint set by the replay-time join (hls.py/media_replay),
|
||||
// not authoritative — never rendered except as plain text.
|
||||
if (item.kind === 'manifest' && typeof item.segments === 'number' && item.segments > 0) {
|
||||
const segRow = document.createElement('div');
|
||||
segRow.className = 'media-meta-row';
|
||||
const segSpan = document.createElement('span');
|
||||
segSpan.textContent = '🧩 ' + item.segments + ' segment' + (item.segments === 1 ? '' : 's');
|
||||
segRow.appendChild(segSpan);
|
||||
card.appendChild(segRow);
|
||||
}
|
||||
|
||||
const actions = document.createElement('div');
|
||||
actions.className = 'media-actions';
|
||||
if (item.expired) {
|
||||
|
|
@ -856,30 +880,91 @@
|
|||
// `id` is hex-validated server-side, but encodeURIComponent defensively
|
||||
// anyway — never build this URL by string-concatenating raw input.
|
||||
const replayUrl = API + '/media/replay/' + encodeURIComponent(item.id);
|
||||
const playBtn = document.createElement('a');
|
||||
playBtn.className = 'btn primary';
|
||||
playBtn.href = replayUrl;
|
||||
playBtn.target = '_blank';
|
||||
playBtn.rel = 'noopener';
|
||||
playBtn.textContent = '▶ Play';
|
||||
const dlBtn = document.createElement('a');
|
||||
dlBtn.className = 'btn';
|
||||
dlBtn.href = replayUrl;
|
||||
dlBtn.setAttribute('download', '');
|
||||
dlBtn.textContent = '⬇ Download';
|
||||
actions.appendChild(playBtn);
|
||||
actions.appendChild(dlBtn);
|
||||
if (item.kind === 'manifest') {
|
||||
buildManifestActions(card, actions, replayUrl);
|
||||
} else {
|
||||
const playBtn = document.createElement('a');
|
||||
playBtn.className = 'btn primary';
|
||||
playBtn.href = replayUrl;
|
||||
playBtn.target = '_blank';
|
||||
playBtn.rel = 'noopener';
|
||||
playBtn.textContent = '▶ Play';
|
||||
const dlBtn = document.createElement('a');
|
||||
dlBtn.className = 'btn';
|
||||
dlBtn.href = replayUrl;
|
||||
dlBtn.setAttribute('download', '');
|
||||
dlBtn.textContent = '⬇ Download';
|
||||
actions.appendChild(playBtn);
|
||||
actions.appendChild(dlBtn);
|
||||
}
|
||||
}
|
||||
card.appendChild(actions);
|
||||
return card;
|
||||
}
|
||||
|
||||
// #812 Phase 2 — a manifest's Play action targets the rewritten playlist
|
||||
// (`/media/replay/{id}`, the manifest branch of the replay endpoint).
|
||||
// No external hls.js is bundled here (self-contained CSP, no CDN scripts
|
||||
// allowed), so a native <video>/in-tab HLS view only works reliably on
|
||||
// Safari; every other browser gets a Download link for the rewritten
|
||||
// .m3u8 plus a short hint. The Play button also best-effort probes the
|
||||
// response for the `X-SecuBox-Media: unsupported-variant` header (set
|
||||
// by the backend when the manifest is master/ABR or AES-encrypted and
|
||||
// therefore cannot be reassembled) and, if present, disables further
|
||||
// inline play attempts while leaving the raw-manifest Download intact.
|
||||
function buildManifestActions(card, actions, replayUrl) {
|
||||
const playBtn = document.createElement('a');
|
||||
playBtn.className = 'btn primary';
|
||||
playBtn.href = replayUrl;
|
||||
playBtn.target = '_blank';
|
||||
playBtn.rel = 'noopener';
|
||||
playBtn.textContent = '🎞️ Ouvrir (Safari/HLS)';
|
||||
|
||||
const dlBtn = document.createElement('a');
|
||||
dlBtn.className = 'btn';
|
||||
dlBtn.href = replayUrl;
|
||||
dlBtn.setAttribute('download', '');
|
||||
dlBtn.textContent = '⬇ Download';
|
||||
|
||||
const note = document.createElement('div');
|
||||
note.className = 'media-note';
|
||||
note.textContent = 'Lecture HLS native limitée à Safari — sinon, téléchargez le manifeste réécrit.';
|
||||
|
||||
let checked = false; // avoid re-probing on every click once resolved
|
||||
playBtn.addEventListener('click', function (ev) {
|
||||
if (checked) return;
|
||||
ev.preventDefault();
|
||||
checked = true;
|
||||
fetch(replayUrl).then(function (res) {
|
||||
if (res.headers.get('X-SecuBox-Media') === 'unsupported-variant') {
|
||||
playBtn.classList.add('disabled');
|
||||
playBtn.removeAttribute('href');
|
||||
note.classList.add('warn');
|
||||
note.textContent = 'Flux non réassemblable (ABR/chiffré) — téléchargement du manifeste brut uniquement.';
|
||||
} else {
|
||||
window.open(replayUrl, '_blank', 'noopener');
|
||||
}
|
||||
}).catch(function () {
|
||||
// network hiccup on the probe — fall back to opening it directly
|
||||
window.open(replayUrl, '_blank', 'noopener');
|
||||
});
|
||||
});
|
||||
|
||||
actions.appendChild(playBtn);
|
||||
actions.appendChild(dlBtn);
|
||||
card.appendChild(note);
|
||||
}
|
||||
|
||||
async function loadMediaBuffer() {
|
||||
const box = document.getElementById('media-buffer-gallery');
|
||||
const badge = document.getElementById('media-buffer-badge');
|
||||
if (!box) return;
|
||||
const data = await apiAuthed(API + '/media/buffer');
|
||||
const items = (data && Array.isArray(data.items)) ? data.items : [];
|
||||
const allItems = (data && Array.isArray(data.items)) ? data.items : [];
|
||||
// #812 Phase 2: HLS segments are join fodder for manifest replay,
|
||||
// never a gallery entry on their own — hide them here, not server-side,
|
||||
// so the count badge below still reflects the API's raw total.
|
||||
const items = allItems.filter(item => item.kind !== 'segment');
|
||||
if (badge) {
|
||||
badge.textContent = (data && typeof data.count === 'number')
|
||||
? (data.count + ' capture' + (data.count === 1 ? '' : 's')) : '-';
|
||||
|
|
|
|||
|
|
@ -128,12 +128,47 @@ func NewMediaBuffer(root string, enabled bool, perObjectCeil int64) *MediaBuffer
|
|||
// should be considered capturable media. It reuses mediaKind (mediacatch.go),
|
||||
// which already covers ctype prefixes ("video/", "audio/"), manifest
|
||||
// mimetypes/extensions (HLS .m3u8, DASH .mpd) and known media file
|
||||
// extensions — any non-empty classification counts as media. nil-safe.
|
||||
// extensions — any non-empty classification counts as media. Also true for
|
||||
// segmentKind (Phase 2, #812): an HLS/DASH segment chunk is media that should
|
||||
// be buffered even in the ambiguous-ctype cases (e.g. a .m4s served as
|
||||
// application/octet-stream) that mediaKind's coarser rules don't recognise on
|
||||
// their own. nil-safe.
|
||||
func (b *MediaBuffer) IsMedia(ctype, path string) bool {
|
||||
if b == nil {
|
||||
return false
|
||||
}
|
||||
return mediaKind(path, ctype) != ""
|
||||
return mediaKind(path, ctype) != "" || segmentKind(path, ctype)
|
||||
}
|
||||
|
||||
// segmentKind reports whether path/ctype identify an HLS/DASH segment CHUNK
|
||||
// (as opposed to a whole-file video/audio download or a manifest). Segments
|
||||
// are ordinary 200 downloads like any other media object (Phase 1's ==200
|
||||
// gate already applies) but must be tagged kind="segment" — not "video" — so
|
||||
// the DPI gallery can hide them and Phase 2's replay-time URL join can find
|
||||
// them (see Capture below, which gives segmentKind precedence over
|
||||
// mediaKind's video/audio classification).
|
||||
//
|
||||
// True for: path ending .ts, .m4s or .m4v (HLS transport-stream / fMP4 /
|
||||
// DASH chunk extensions), or ctype video/mp2t (a.k.a. video/MP2T) or
|
||||
// video/iso.segment. Deliberately FALSE for whole-file .mp4/.webm/.mkv/.mov
|
||||
// and for .m3u8/.mpd manifests — those must keep classifying as "video" /
|
||||
// "manifest" via mediaKind, unchanged. When ctype is ambiguous (e.g.
|
||||
// application/octet-stream, or empty) only the path extension decides — a
|
||||
// bare "some binary blob" ctype must NOT be guessed into "segment" without a
|
||||
// segment-shaped path.
|
||||
func segmentKind(path, ctype string) bool {
|
||||
p := strings.ToLower(path)
|
||||
if i := strings.IndexByte(p, '?'); i >= 0 {
|
||||
p = p[:i]
|
||||
}
|
||||
if strings.HasSuffix(p, ".ts") || strings.HasSuffix(p, ".m4s") || strings.HasSuffix(p, ".m4v") {
|
||||
return true
|
||||
}
|
||||
switch strings.ToLower(ctype) {
|
||||
case "video/mp2t", "video/iso.segment":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Capture decides whether to start recording a flow's body and, if so,
|
||||
|
|
@ -185,6 +220,17 @@ func (b *MediaBuffer) Capture(mac, host, url, path, ctype, direction string, con
|
|||
qcap = mediaBufferChanCap
|
||||
}
|
||||
|
||||
// kind: mediaKind's coarser ctype-prefix rule ("video/" → "video") would
|
||||
// otherwise misclassify an HLS/DASH segment (e.g. .ts served video/mp2t)
|
||||
// as a whole video. segmentKind takes precedence when it matches — Phase
|
||||
// 2 (#812) needs kind="segment" so the DPI gallery can hide it and the
|
||||
// replay-time manifest join can find it — but whole-file .mp4/video/*
|
||||
// and .m3u8/.mpd manifests are unaffected (segmentKind is false for both).
|
||||
kind := mediaKind(path, ctype)
|
||||
if segmentKind(path, ctype) {
|
||||
kind = "segment"
|
||||
}
|
||||
|
||||
w := &ObjectWriter{
|
||||
buf: b,
|
||||
sink: sink,
|
||||
|
|
@ -196,7 +242,7 @@ func (b *MediaBuffer) Capture(mac, host, url, path, ctype, direction string, con
|
|||
host: host,
|
||||
url: url,
|
||||
direction: direction,
|
||||
kind: mediaKind(path, ctype),
|
||||
kind: kind,
|
||||
ctype: ctype,
|
||||
firstTS: time.Now().Unix(),
|
||||
perObjectCeil: b.perObjectCeil,
|
||||
|
|
|
|||
|
|
@ -528,3 +528,60 @@ func TestDownloadTeeSkips206(t *testing.T) {
|
|||
t.Fatalf("expected 200 download captured with mac_hash=mac200, got %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSegmentClassification is the Phase 2 (#812) unit test for segmentKind:
|
||||
// HLS transport-stream (.ts) and fMP4/DASH (.m4s) chunks — by extension OR by
|
||||
// a segment-shaped Content-Type (video/mp2t) — must classify as a segment,
|
||||
// while whole-file video (.mp4) and manifests (.m3u8) must NOT, and an
|
||||
// ambiguous Content-Type (application/octet-stream) with a non-segment path
|
||||
// must not be guessed into a segment either.
|
||||
func TestSegmentClassification(t *testing.T) {
|
||||
cases := []struct {
|
||||
path, ctype string
|
||||
want bool
|
||||
}{
|
||||
{"/x.ts", "video/mp2t", true},
|
||||
{"/x.m4s", "", true},
|
||||
{"/v.mp4", "video/mp4", false},
|
||||
{"/i.m3u8", "application/vnd.apple.mpegurl", false},
|
||||
{"/x.ts", "", true},
|
||||
{"/data", "application/octet-stream", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := segmentKind(c.path, c.ctype); got != c.want {
|
||||
t.Errorf("segmentKind(%q,%q)=%v want %v", c.path, c.ctype, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCaptureTagsSegmentKind is the Phase 2 (#812) Capture-level test: a
|
||||
// captured HLS segment (.ts served video/mp2t) must be tagged kind="segment"
|
||||
// in its metatag — NOT "video", which mediaKind's ctype-prefix rule would
|
||||
// otherwise assign — while a whole-file video download and an HLS manifest
|
||||
// keep their Phase 1 classifications unchanged.
|
||||
func TestCaptureTagsSegmentKind(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, path, ctype, wantKind string
|
||||
}{
|
||||
{"segment", "/seg0.ts", "video/mp2t", "segment"},
|
||||
{"video", "/movie.mp4", "video/mp4", "video"},
|
||||
{"manifest", "/index.m3u8", "application/vnd.apple.mpegurl", "manifest"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
b := NewMediaBuffer(root, true, 512<<20)
|
||||
w := b.Capture("mac1", "cdn.example", "https://cdn.example"+c.path, c.path, c.ctype, "down", 16)
|
||||
if w == nil {
|
||||
t.Fatalf("Capture returned nil for %s (%s)", c.path, c.ctype)
|
||||
}
|
||||
w.Write([]byte("0123456789ABCDEF"))
|
||||
w.Close(16)
|
||||
|
||||
m := lastJSONL(t, filepath.Join(root, "media-buffer.jsonl"))
|
||||
if m["kind"] != c.wantKind {
|
||||
t.Errorf("kind=%v want %q", m["kind"], c.wantKind)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,17 @@
|
|||
secubox-toolbox-ng (0.1.28-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* #812 R4 media buffer (Phase 2): HLS reassembly — segments (.ts/.m4s) are
|
||||
captured tagged kind=segment alongside whole-file media/manifest capture;
|
||||
the DPI manifest replay branch parses a captured .m3u8, joins each segment
|
||||
URI to its captured segment by resolved absolute URL (read-time join, no
|
||||
live session state in sbxmitm), and serves a rewritten playlist whose
|
||||
segment URIs point at their captured segments' replay URLs. Master/ABR
|
||||
(multivariant) and AES-128-encrypted playlists, and DASH .mpd, are
|
||||
detected and served raw with an X-SecuBox-Media: unsupported-variant
|
||||
header rather than a broken rewrite — deferred to Phase 2b.
|
||||
|
||||
-- Gerald KERMA <devel@cybermind.fr> Sat, 04 Jul 2026 16:00:00 +0000
|
||||
|
||||
secubox-toolbox-ng (0.1.27-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* #812 R4 media buffer (Phase 1): sbxmitm tees whole-file media up/downloads on
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user