Escalate a depleted Gemini balance the same way as a dead model ID
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Successful in 2m2s

Correcting the model IDs got past the 404s and straight into 429 "Your
prepayment credits are depleted" on every call, so the LLM correlation tier is
still down -- same symptom, different cause, and the previous commit would have
logged it as an ordinary per-call WARNING and buried it exactly like the last
one.

An empty balance shares a status code with an ordinary rate limit but is the
opposite kind of problem: a rate limit clears on its own, a dead account never
does. The match is on the billing wording ("credits are depleted",
"prepayment", "billing") rather than on 429, so a burst of rate limiting still
reads as WARNING while an unpayable account escalates to the once-per-model
ERROR that names the fix.

The two escalation paths now share _log_tier_down, which is also where the
once-per-model suppression lives -- this code runs on every call at radio
traffic volume, so an ERROR per call would be its own kind of noise.

38 correlator tests still pass. No new environment variables, so CI deploys
this without an ansible run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-18 20:37:27 -04:00
co-authored by Claude Opus 5
parent 2a1d52b7af
commit 4842725a03
+25 -7
View File
@@ -260,16 +260,34 @@ def _log_llm_failure(where: str, call_id: str, model: str, exc: Exception) -> No
that will never fix itself, so it gets ERROR and says what to do.
"""
text = str(exc)
if "404" in text or "not found" in text.lower() or "no longer available" in text.lower():
if model not in _dead_models:
low = text.lower()
if "404" in text or "not found" in low or "no longer available" in low:
_log_tier_down(where, model, "model is unavailable",
"Update CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py", text)
return
# A depleted balance reads as 429, the same status as an ordinary rate limit,
# but it is the opposite kind of problem: a rate limit clears on its own and a
# dead account never does. Matching on the billing wording keeps a burst of
# rate limits at WARNING while an empty account escalates like a bad model ID.
if "credits are depleted" in low or "prepayment" in low or "billing" in low:
_log_tier_down(where, model, "the Gemini account is out of credit",
"Top up billing at https://ai.studio/projects", text)
return
logger.warning(f"{where} failed for call {call_id}: {text}")
def _log_tier_down(where: str, model: str, problem: str, fix: str, text: str) -> None:
"""ERROR once per model, not once per call — this runs at radio-traffic volume."""
if model in _dead_models:
return
_dead_models.add(model)
logger.error(
f"{where}: model {model!r} is unavailable -- the LLM correlation tier "
f"is DISABLED and every call is falling back to rules-only. Update "
f"CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py. Google said: {text}"
f"{where}: {problem} ({model!r}) -- the LLM correlation tier is DISABLED "
f"and every call is falling back to rules-only. {fix}. API said: {text}"
)
return
logger.warning(f"{where} failed for call {call_id}: {text}")
async def tiebreak(rules_decision: dict, llm_decision: dict, ctx: dict) -> dict: