Say so loudly when the OpenAI account can no longer be billed
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Failing after 2m56s

Transcription is the top of the pipeline and it fails soft: any exception logs
a WARNING, returns None, and upload.py carries on. That is the right behaviour
for a network blip and exactly the wrong behaviour for an unpayable account,
because with no transcript there is no extraction, no correlation and no
incident -- the system keeps accepting calls and quietly stores empty ones,
which looks like quiet radio traffic rather than an outage.

This is the third instance of the same failure mode today. The Gemini
correlator was down first on a retired model ID and then on a depleted
balance, and in both cases the only signal was a per-call WARNING that read as
noise. The OpenAI balance is low enough that this one is a matter of when.

Billing-shaped errors (insufficient_quota, billing, credit, quota exceeded)
now log once at ERROR, name what is dead downstream, and link the top-up page.
Everything else keeps the existing per-call WARNING.

No new environment variables, so CI deploys this without an ansible run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-18 20:41:45 -04:00
co-authored by Claude Opus 5
parent 83416fe169
commit 427d2a9f37
+37 -1
View File
@@ -93,6 +93,42 @@ def _is_degenerate(text: str, segments: list[dict]) -> bool:
return False
_billing_reported = False
def _log_transcribe_failure(call_id: str, exc: Exception) -> None:
"""
Log a transcription failure, escalating an unpayable account to ERROR once.
Transcription failing returns None and the pipeline carries on by design, so
a per-call WARNING is invisible: no transcript means no extraction, which
means no incident, and the only symptom is calls quietly arriving empty. A
network blip is genuinely a warning. An exhausted balance is not -- it will
not fix itself and it takes the whole pipeline down with it, so it says so
once, loudly, and names the fix.
The same failure mode already bit the Gemini correlator twice (a retired
model ID, then a depleted balance), which is why this is worth the code.
"""
global _billing_reported
text = str(exc)
low = text.lower()
if ("insufficient_quota" in low or "billing" in low
or "credit" in low or "exceeded your current quota" in low):
if not _billing_reported:
_billing_reported = True
logger.error(
"Transcription: the OpenAI account cannot be billed -- EVERY call is "
"now stored with no transcript, so extraction, correlation and "
"incidents are all dead downstream. Top up at "
f"https://platform.openai.com/settings/organization/billing. API said: {text}"
)
return
logger.warning(f"Transcription failed for call {call_id}: {text}")
async def transcribe_call(
call_id: str,
gcs_uri: str,
@@ -114,7 +150,7 @@ async def transcribe_call(
_sync_transcribe, gcs_uri, talkgroup_name
)
except Exception as e:
logger.warning(f"Transcription failed for call {call_id}: {e}")
_log_transcribe_failure(call_id, e)
return None, []
if transcript: