Admin Replay: re-run the pipeline over past calls in a sandbox (#170)

Correlation has only ever been measured through live AI windows: days of
wall time per change, and the 09-20→22 window was invalidated outright by
unfunded AI accounts (#169). Recordings are kept regardless of AI, so the
traffic to measure against already exists.

- internal/replay.py: runs a time range of real calls through the live
  pipeline code in original order, clock pinned per call, into
  replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe),
  transcripts (re-extract), reuse (correlation only from a prior run's
  scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on
  virtual time. No alerts, summaries, vocab, AI-health alerts or pending
  terms. One run at a time, <=5000 calls, <=7 days.
- firestore.py: ContextVar sandbox redirect for calls/incidents.
- clock.py: ContextVar-pinnable now(), used on the correlation path.
- feature_flags.py: ContextVar flag override so replay runs with live AI off.
- upload.py: scene loop extracted to _extract_and_correlate, shared by the
  live pipeline and replay so replay measures the code that runs live.
- resolved_via on every incident resolve, so a real clear can be told
  from the idle timeout — live and in replay.
- routers/replay.py + /admin Replay tab: estimate, start, compare runs,
  drill into incidents with audio.

Reviewed by drb-correlation-review; its leak and fidelity findings are
fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run
(no Node on the authoring box).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-09-26 15:23:57 -04:00
co-authored by Claude Opus 5.5
parent e79b8bc37d
commit aff3f16d32
19 changed files with 1976 additions and 101 deletions
+32
View File
@@ -0,0 +1,32 @@
"""
Pipeline clock.
`now()` is `datetime.now(timezone.utc)` everywhere except inside a replay run
(app/internal/replay.py), which pins it to the replayed call's own time so the
correlator's recency windows, the idle-resolve sweep and every started_at /
updated_at / resolved_at it writes behave the way they did live.
A ContextVar rather than a module global: a replay runs as a background task
alongside real uploads, and each asyncio task (and every asyncio.to_thread it
spawns) carries its own copy of the context, so a pinned clock can never leak
into a live call's pipeline.
"""
from contextvars import ContextVar
from datetime import datetime, timezone
from typing import Optional
_pinned: ContextVar[Optional[datetime]] = ContextVar("drb_clock_pinned", default=None)
def now() -> datetime:
pinned = _pinned.get()
return pinned if pinned is not None else datetime.now(timezone.utc)
def pin(when: Optional[datetime]):
"""Pin the clock for the current context. Returns a token for `unpin`."""
return _pinned.set(when)
def unpin(token) -> None:
_pinned.reset(token)