Replay: fail fast on a dead AI account; extraction reports to ai_health
First replay (290 calls, 09-22 10:00-12:00 ET) produced 0 incidents and no errors: every gpt-4o-mini extraction failed and _sync_extract swallowed it as "no scenes". Same shape as #169 — and the live extraction tier in /health/ai had no reporter at all, so this has been invisible in production too. - intelligence: API failures propagate out of _sync_extract; extract_scenes reports them to ai_health ("extraction" tier, billing/dead-model classified) and still returns [] so the pipeline degrades as before. - ai_health: inside a replay sandbox, failures go to the run's own sink instead of being dropped. - replay: aborts after 5 permanent failures on a tier, naming the cause; run metrics carry ai_failures; UI shows them. - replay estimate: audio minutes from started_at/ended_at (no duration field exists on call docs). - ReplayTab exposes the loaded run on window.__drbReplay for in-page analysis. c2-core: 458 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
aff3f16d32
commit
ec91a9175f
@@ -25,6 +25,7 @@ transcription.py) need the exact same judgment call and must not each grow
|
||||
their own slightly-different copy that drifts.
|
||||
"""
|
||||
import asyncio
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
@@ -59,6 +60,14 @@ def _default_state() -> dict:
|
||||
|
||||
_state: dict[str, dict] = {t: _default_state() for t in TIERS}
|
||||
|
||||
# Set by a replay run to a list it owns; report_degraded appends there instead
|
||||
# of touching _state while inside a sandbox (see app/internal/replay.py).
|
||||
_sandbox_failures: ContextVar[Optional[list]] = ContextVar("drb_ai_sandbox_failures", default=None)
|
||||
|
||||
|
||||
def collect_sandbox_failures(sink: Optional[list]):
|
||||
return _sandbox_failures.set(sink)
|
||||
|
||||
|
||||
def classify(text: str) -> str:
|
||||
"""
|
||||
@@ -112,6 +121,11 @@ async def report_degraded(
|
||||
if fstore.in_sandbox():
|
||||
# A replay's rate limits are not a live outage, and must never page
|
||||
# the AI-alert webhook or flip /health/ai (app/internal/replay.py).
|
||||
# They are the run's own problem, so they go to the run instead.
|
||||
sink = _sandbox_failures.get()
|
||||
if sink is not None:
|
||||
sink.append({"tier": tier, "provider": provider, "model": model,
|
||||
"problem": problem, "permanent": permanent})
|
||||
return
|
||||
if tier not in _state:
|
||||
_state[tier] = _default_state()
|
||||
|
||||
Reference in New Issue
Block a user