Say so loudly when the OpenAI account can no longer be billed
Transcription is the top of the pipeline and it fails soft: any exception logs a WARNING, returns None, and upload.py carries on. That is the right behaviour for a network blip and exactly the wrong behaviour for an unpayable account, because with no transcript there is no extraction, no correlation and no incident -- the system keeps accepting calls and quietly stores empty ones, which looks like quiet radio traffic rather than an outage. This is the third instance of the same failure mode today. The Gemini correlator was down first on a retired model ID and then on a depleted balance, and in both cases the only signal was a per-call WARNING that read as noise. The OpenAI balance is low enough that this one is a matter of when. Billing-shaped errors (insufficient_quota, billing, credit, quota exceeded) now log once at ERROR, name what is dead downstream, and link the top-up page. Everything else keeps the existing per-call WARNING. No new environment variables, so CI deploys this without an ansible run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
83416fe169
commit
427d2a9f37
@@ -93,6 +93,42 @@ def _is_degenerate(text: str, segments: list[dict]) -> bool:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
_billing_reported = False
|
||||||
|
|
||||||
|
|
||||||
|
def _log_transcribe_failure(call_id: str, exc: Exception) -> None:
|
||||||
|
"""
|
||||||
|
Log a transcription failure, escalating an unpayable account to ERROR once.
|
||||||
|
|
||||||
|
Transcription failing returns None and the pipeline carries on by design, so
|
||||||
|
a per-call WARNING is invisible: no transcript means no extraction, which
|
||||||
|
means no incident, and the only symptom is calls quietly arriving empty. A
|
||||||
|
network blip is genuinely a warning. An exhausted balance is not -- it will
|
||||||
|
not fix itself and it takes the whole pipeline down with it, so it says so
|
||||||
|
once, loudly, and names the fix.
|
||||||
|
|
||||||
|
The same failure mode already bit the Gemini correlator twice (a retired
|
||||||
|
model ID, then a depleted balance), which is why this is worth the code.
|
||||||
|
"""
|
||||||
|
global _billing_reported
|
||||||
|
text = str(exc)
|
||||||
|
low = text.lower()
|
||||||
|
|
||||||
|
if ("insufficient_quota" in low or "billing" in low
|
||||||
|
or "credit" in low or "exceeded your current quota" in low):
|
||||||
|
if not _billing_reported:
|
||||||
|
_billing_reported = True
|
||||||
|
logger.error(
|
||||||
|
"Transcription: the OpenAI account cannot be billed -- EVERY call is "
|
||||||
|
"now stored with no transcript, so extraction, correlation and "
|
||||||
|
"incidents are all dead downstream. Top up at "
|
||||||
|
f"https://platform.openai.com/settings/organization/billing. API said: {text}"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
logger.warning(f"Transcription failed for call {call_id}: {text}")
|
||||||
|
|
||||||
|
|
||||||
async def transcribe_call(
|
async def transcribe_call(
|
||||||
call_id: str,
|
call_id: str,
|
||||||
gcs_uri: str,
|
gcs_uri: str,
|
||||||
@@ -114,7 +150,7 @@ async def transcribe_call(
|
|||||||
_sync_transcribe, gcs_uri, talkgroup_name
|
_sync_transcribe, gcs_uri, talkgroup_name
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f"Transcription failed for call {call_id}: {e}")
|
_log_transcribe_failure(call_id, e)
|
||||||
return None, []
|
return None, []
|
||||||
|
|
||||||
if transcript:
|
if transcript:
|
||||||
|
|||||||
Reference in New Issue
Block a user