Admin Replay: re-run the pipeline over past calls in a sandbox (#170)
Correlation has only ever been measured through live AI windows: days of wall time per change, and the 09-20→22 window was invalidated outright by unfunded AI accounts (#169). Recordings are kept regardless of AI, so the traffic to measure against already exists. - internal/replay.py: runs a time range of real calls through the live pipeline code in original order, clock pinned per call, into replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe), transcripts (re-extract), reuse (correlation only from a prior run's scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on virtual time. No alerts, summaries, vocab, AI-health alerts or pending terms. One run at a time, <=5000 calls, <=7 days. - firestore.py: ContextVar sandbox redirect for calls/incidents. - clock.py: ContextVar-pinnable now(), used on the correlation path. - feature_flags.py: ContextVar flag override so replay runs with live AI off. - upload.py: scene loop extracted to _extract_and_correlate, shared by the live pipeline and replay so replay measures the code that runs live. - resolved_via on every incident resolve, so a real clear can be told from the idle timeout — live and in replay. - routers/replay.py + /admin Replay tab: estimate, start, compare runs, drill into incidents with audio. Reviewed by drb-correlation-review; its leak and fidelity findings are fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run (no Node on the authoring box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
e79b8bc37d
commit
aff3f16d32
@@ -108,6 +108,11 @@ async def report_degraded(
|
||||
TRANSIENT_ALERT_THRESHOLD consecutive failures have been reported for
|
||||
this tier, so an ordinary blip never pages anyone.
|
||||
"""
|
||||
from app.internal import firestore as fstore
|
||||
if fstore.in_sandbox():
|
||||
# A replay's rate limits are not a live outage, and must never page
|
||||
# the AI-alert webhook or flip /health/ai (app/internal/replay.py).
|
||||
return
|
||||
if tier not in _state:
|
||||
_state[tier] = _default_state()
|
||||
entry = _state[tier]
|
||||
@@ -145,6 +150,9 @@ async def report_healthy(tier: str) -> None:
|
||||
just after a failure -- it is what lets a degraded tier recover on its
|
||||
own instead of staying red forever after one transient blip.
|
||||
"""
|
||||
from app.internal import firestore as fstore
|
||||
if fstore.in_sandbox():
|
||||
return # nor may a replay's success "recover" a real live outage
|
||||
if tier not in _state:
|
||||
_state[tier] = _default_state()
|
||||
entry = _state[tier]
|
||||
|
||||
Reference in New Issue
Block a user