Escalate a depleted Gemini balance the same way as a dead model ID
Correcting the model IDs got past the 404s and straight into 429 "Your
prepayment credits are depleted" on every call, so the LLM correlation tier is
still down -- same symptom, different cause, and the previous commit would have
logged it as an ordinary per-call WARNING and buried it exactly like the last
one.
An empty balance shares a status code with an ordinary rate limit but is the
opposite kind of problem: a rate limit clears on its own, a dead account never
does. The match is on the billing wording ("credits are depleted",
"prepayment", "billing") rather than on 429, so a burst of rate limiting still
reads as WARNING while an unpayable account escalates to the once-per-model
ERROR that names the fix.
The two escalation paths now share _log_tier_down, which is also where the
once-per-model suppression lives -- this code runs on every call at radio
traffic volume, so an ERROR per call would be its own kind of noise.
38 correlator tests still pass. No new environment variables, so CI deploys
this without an ansible run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
2a1d52b7af
commit
4842725a03
@@ -260,18 +260,36 @@ def _log_llm_failure(where: str, call_id: str, model: str, exc: Exception) -> No
|
|||||||
that will never fix itself, so it gets ERROR and says what to do.
|
that will never fix itself, so it gets ERROR and says what to do.
|
||||||
"""
|
"""
|
||||||
text = str(exc)
|
text = str(exc)
|
||||||
if "404" in text or "not found" in text.lower() or "no longer available" in text.lower():
|
low = text.lower()
|
||||||
if model not in _dead_models:
|
|
||||||
_dead_models.add(model)
|
if "404" in text or "not found" in low or "no longer available" in low:
|
||||||
logger.error(
|
_log_tier_down(where, model, "model is unavailable",
|
||||||
f"{where}: model {model!r} is unavailable -- the LLM correlation tier "
|
"Update CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py", text)
|
||||||
f"is DISABLED and every call is falling back to rules-only. Update "
|
|
||||||
f"CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py. Google said: {text}"
|
|
||||||
)
|
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# A depleted balance reads as 429, the same status as an ordinary rate limit,
|
||||||
|
# but it is the opposite kind of problem: a rate limit clears on its own and a
|
||||||
|
# dead account never does. Matching on the billing wording keeps a burst of
|
||||||
|
# rate limits at WARNING while an empty account escalates like a bad model ID.
|
||||||
|
if "credits are depleted" in low or "prepayment" in low or "billing" in low:
|
||||||
|
_log_tier_down(where, model, "the Gemini account is out of credit",
|
||||||
|
"Top up billing at https://ai.studio/projects", text)
|
||||||
|
return
|
||||||
|
|
||||||
logger.warning(f"{where} failed for call {call_id}: {text}")
|
logger.warning(f"{where} failed for call {call_id}: {text}")
|
||||||
|
|
||||||
|
|
||||||
|
def _log_tier_down(where: str, model: str, problem: str, fix: str, text: str) -> None:
|
||||||
|
"""ERROR once per model, not once per call — this runs at radio-traffic volume."""
|
||||||
|
if model in _dead_models:
|
||||||
|
return
|
||||||
|
_dead_models.add(model)
|
||||||
|
logger.error(
|
||||||
|
f"{where}: {problem} ({model!r}) -- the LLM correlation tier is DISABLED "
|
||||||
|
f"and every call is falling back to rules-only. {fix}. API said: {text}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def tiebreak(rules_decision: dict, llm_decision: dict, ctx: dict) -> dict:
|
async def tiebreak(rules_decision: dict, llm_decision: dict, ctx: dict) -> dict:
|
||||||
"""
|
"""
|
||||||
Run the smart tiebreaker (corr_smart_model) when rules and LLM disagree.
|
Run the smart tiebreaker (corr_smart_model) when rules and LLM disagree.
|
||||||
|
|||||||
Reference in New Issue
Block a user