Replay: fail fast on a dead AI account; extraction reports to ai_health
First replay (290 calls, 09-22 10:00-12:00 ET) produced 0 incidents and no errors: every gpt-4o-mini extraction failed and _sync_extract swallowed it as "no scenes". Same shape as #169 — and the live extraction tier in /health/ai had no reporter at all, so this has been invisible in production too. - intelligence: API failures propagate out of _sync_extract; extract_scenes reports them to ai_health ("extraction" tier, billing/dead-model classified) and still returns [] so the pipeline degrades as before. - ai_health: inside a replay sandbox, failures go to the run's own sink instead of being dropped. - replay: aborts after 5 permanent failures on a tier, naming the cause; run metrics carry ai_failures; UI shows them. - replay estimate: audio minutes from started_at/ended_at (no duration field exists on call docs). - ReplayTab exposes the loaded run on window.__drbReplay for in-page analysis. c2-core: 458 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
aff3f16d32
commit
ec91a9175f
@@ -15,6 +15,7 @@ import re
|
||||
from typing import Optional
|
||||
from app.internal.logger import logger
|
||||
from app.internal import firestore as fstore
|
||||
from app.internal import ai_health
|
||||
from app.internal import area_context
|
||||
from app.internal.chatter_classifier import classify_chatter
|
||||
# Location validity is defined once, by the module that owns the incident's
|
||||
@@ -268,11 +269,26 @@ async def extract_scenes(
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raw_scenes: list[dict] = await asyncio.to_thread(
|
||||
_sync_extract,
|
||||
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
|
||||
unit_format_hint,
|
||||
)
|
||||
try:
|
||||
raw_scenes: list[dict] = await asyncio.to_thread(
|
||||
_sync_extract,
|
||||
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
|
||||
unit_format_hint,
|
||||
)
|
||||
except Exception as e:
|
||||
text = str(e)
|
||||
kind = ai_health.classify(text)
|
||||
logger.warning(f"GPT-4o-mini extraction failed for call {call_id}: {text}")
|
||||
await ai_health.report_degraded(
|
||||
"extraction", "openai", "gpt-4o-mini",
|
||||
{"billing": "the OpenAI account is out of credit",
|
||||
"dead_model": "model is unavailable"}.get(kind, f"extraction failed: {text[:200]}"),
|
||||
{"billing": "Top up OpenAI billing",
|
||||
"dead_model": "Update the extraction model in intelligence.py"}.get(kind, "Usually transient"),
|
||||
permanent=kind != "transient",
|
||||
)
|
||||
return []
|
||||
await ai_health.report_healthy("extraction")
|
||||
|
||||
if not raw_scenes:
|
||||
return []
|
||||
@@ -806,9 +822,11 @@ def _sync_extract(
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f"GPT-4o-mini returned non-JSON: {e}")
|
||||
return []
|
||||
except Exception as e:
|
||||
logger.warning(f"GPT-4o-mini extraction failed: {e}")
|
||||
return []
|
||||
# Any other exception is the API call itself failing (no credit, rate
|
||||
# limit, outage) and propagates to extract_scenes, which reports it to
|
||||
# ai_health. Swallowing it here made "OpenAI is down" indistinguishable
|
||||
# from "nothing happened on the radio" — the extraction tier existed in
|
||||
# /health/ai but nothing ever reported to it.
|
||||
|
||||
|
||||
def _sync_embed(text: str) -> Optional[list[float]]:
|
||||
|
||||
Reference in New Issue
Block a user