Admin Replay: re-run the pipeline over past calls in a sandbox, so correlation changes are measured without live AI windows #170

Open
opened 2026-09-26 15:15:54 -04:00 by logan · 0 comments
Owner

Owner ruling 2026-09-26: correlation work can't keep depending on live AI windows (days of wall time, burns credits, and the 09-20→22 window was invalidated by #169). Recordings are kept regardless of AI, so replay past traffic instead — as a permanent admin section, not a one-off script.

Built on branch feat/replay (Server):

  • app/internal/replay.py — runs a time range of real calls through the live pipeline code (upload._extract_and_correlate, now shared with the live path), original order, clock pinned per call, into replay_runs/{run_id}/calls|incidents (ContextVar redirect in firestore.py). Modes: audio (re-Whisper), transcripts (re-extract), reuse (correlation only, reuses a prior run's scenes — isolates correlator changes from extraction noise). Simulates idle-resolve + recorrelation sweeps on virtual time. No alerts/summaries/vocab. One run at a time, ≤5000 calls, ≤7 days.
  • app/routers/replay.py — /admin/replay estimate/start/list/get/cancel/delete/incidents.
  • New resolved_via field on every incident resolve (units_cleared, llm_closure, reassignment, children_resolved, idle_timeout, emptied_by_*) — also useful live: first time a real clear can be told from a timeout without inference.
  • Frontend: /admin → Replay tab (form + estimate, runs table with comparable metrics, incident drill-down with audio).

Next after merge: replay 09-22 11:30–19:00Z (transcripts mode) once Gemini/OpenAI are funded, as the baseline row. Follow-up worth filing separately: hand-labelled ground truth for one afternoon so runs get a real score, not just shape metrics.

Owner ruling 2026-09-26: correlation work can't keep depending on live AI windows (days of wall time, burns credits, and the 09-20→22 window was invalidated by #169). Recordings are kept regardless of AI, so replay past traffic instead — as a permanent admin section, not a one-off script. Built on branch `feat/replay` (Server): - `app/internal/replay.py` — runs a time range of real calls through the live pipeline code (`upload._extract_and_correlate`, now shared with the live path), original order, clock pinned per call, into `replay_runs/{run_id}/calls|incidents` (ContextVar redirect in `firestore.py`). Modes: audio (re-Whisper), transcripts (re-extract), reuse (correlation only, reuses a prior run's scenes — isolates correlator changes from extraction noise). Simulates idle-resolve + recorrelation sweeps on virtual time. No alerts/summaries/vocab. One run at a time, ≤5000 calls, ≤7 days. - `app/routers/replay.py` — `/admin/replay` estimate/start/list/get/cancel/delete/incidents. - New `resolved_via` field on every incident resolve (units_cleared, llm_closure, reassignment, children_resolved, idle_timeout, emptied_by_*) — also useful live: first time a real clear can be told from a timeout without inference. - Frontend: /admin → Replay tab (form + estimate, runs table with comparable metrics, incident drill-down with audio). Next after merge: replay 09-22 11:30–19:00Z (transcripts mode) once Gemini/OpenAI are funded, as the baseline row. Follow-up worth filing separately: hand-labelled ground truth for one afternoon so runs get a real score, not just shape metrics.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#170