Three separate outages in one night, all with the same shape: a dependency died, the code fell back gracefully, and nothing said so.
corr_cheap_model / corr_smart_model pointed at retired Gemini model IDs. Every correlation call 404'd for an unknown number of days. The LLM tier and the consensus tiebreak were dead in production while correlation was being tuned against rules-only output.
Once the IDs were corrected, every call returned 429 "prepayment credits are depleted".
Transcription has the same structure: transcription.py returns None on any failure and the pipeline carries on, so an unpayable OpenAI account would silently store every call with no transcript.
All three now log once at ERROR naming the fix, which is a floor, not a solution — nobody reads container logs continuously. What is missing is an actual alert (Discord webhook to the operator, or a health endpoint that reports degraded AI tiers) so a dead dependency surfaces without someone happening to grep for it.
The deeper pattern worth fixing: every AI call site fails soft, which is right for a transient blip and wrong for a permanent condition. Those two cases need to be distinguishable everywhere, not just where they have been patched reactively.
Three separate outages in one night, all with the same shape: a dependency died, the code fell back gracefully, and nothing said so.
1. `corr_cheap_model` / `corr_smart_model` pointed at retired Gemini model IDs. Every correlation call 404'd for an unknown number of days. The LLM tier and the consensus tiebreak were dead in production while correlation was being tuned against rules-only output.
2. Once the IDs were corrected, every call returned 429 "prepayment credits are depleted".
3. Transcription has the same structure: `transcription.py` returns `None` on any failure and the pipeline carries on, so an unpayable OpenAI account would silently store every call with no transcript.
All three now log once at ERROR naming the fix, which is a floor, not a solution — nobody reads container logs continuously. What is missing is an actual alert (Discord webhook to the operator, or a health endpoint that reports degraded AI tiers) so a dead dependency surfaces without someone happening to grep for it.
The deeper pattern worth fixing: every AI call site fails soft, which is right for a transient blip and wrong for a permanent condition. Those two cases need to be distinguishable everywhere, not just where they have been patched reactively.
New app/internal/ai_health.py: a per-tier degradation registry (transcription, correlation_cheap, correlation_smart, extraction). classify(text) is now the single place that disambiguates a depleted balance from an ordinary rate limit — both arrive as 429 — moved out of llm_correlator.py so it cannot drift into two copies.
report_degraded(..., permanent=) alerts immediately for a permanent condition (retired model, dead billing) and only after 5 consecutive failures for a transient one. Once per episode, never per call — this runs at radio-traffic volume.
report_healthy(tier) resets and posts a recovery notice, so a tier that comes back does not stay flagged.
GET /health/ai exposes the snapshot. Deliberately separate from /health so the CI deploy's curl -f /health liveness check keeps its existing meaning.
Alerts POST to ai_alert_webhook_url (new in config.py); empty means silently skip, since not every deployment will set one.
transcription.py and llm_correlator.py now report through the registry on both failure and success; the ERROR logs are kept alongside, not replaced.
14 new tests in tests/test_ai_health.py. Suite: 107 passed, 0 failed.
Set AI_ALERT_WEBHOOK_URL on the server — without it this only improves /health/ai, and the whole point of the issue was getting the signal out of the box.
Note the adjacent gap found the same night: deploys were failing silently for two days with nobody notified either (server-26#21). Same shape of problem, different subsystem.
Implemented in `a250c29`.
- New `app/internal/ai_health.py`: a per-tier degradation registry (`transcription`, `correlation_cheap`, `correlation_smart`, `extraction`). `classify(text)` is now the single place that disambiguates a depleted balance from an ordinary rate limit — both arrive as 429 — moved out of `llm_correlator.py` so it cannot drift into two copies.
- `report_degraded(..., permanent=)` alerts immediately for a permanent condition (retired model, dead billing) and only after 5 consecutive failures for a transient one. Once per episode, never per call — this runs at radio-traffic volume.
- `report_healthy(tier)` resets and posts a recovery notice, so a tier that comes back does not stay flagged.
- `GET /health/ai` exposes the snapshot. Deliberately separate from `/health` so the CI deploy's `curl -f /health` liveness check keeps its existing meaning.
- Alerts POST to `ai_alert_webhook_url` (new in `config.py`); empty means silently skip, since not every deployment will set one.
- `transcription.py` and `llm_correlator.py` now report through the registry on both failure and success; the ERROR logs are kept alongside, not replaced.
14 new tests in `tests/test_ai_health.py`. Suite: **107 passed, 0 failed**.
**Set `AI_ALERT_WEBHOOK_URL` on the server** — without it this only improves `/health/ai`, and the whole point of the issue was getting the signal out of the box.
Note the adjacent gap found the same night: deploys were failing silently for two days with nobody notified either (server-26#21). Same shape of problem, different subsystem.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Three separate outages in one night, all with the same shape: a dependency died, the code fell back gracefully, and nothing said so.
corr_cheap_model/corr_smart_modelpointed at retired Gemini model IDs. Every correlation call 404'd for an unknown number of days. The LLM tier and the consensus tiebreak were dead in production while correlation was being tuned against rules-only output.transcription.pyreturnsNoneon any failure and the pipeline carries on, so an unpayable OpenAI account would silently store every call with no transcript.All three now log once at ERROR naming the fix, which is a floor, not a solution — nobody reads container logs continuously. What is missing is an actual alert (Discord webhook to the operator, or a health endpoint that reports degraded AI tiers) so a dead dependency surfaces without someone happening to grep for it.
The deeper pattern worth fixing: every AI call site fails soft, which is right for a transient blip and wrong for a permanent condition. Those two cases need to be distinguishable everywhere, not just where they have been patched reactively.
Implemented in
a250c29.app/internal/ai_health.py: a per-tier degradation registry (transcription,correlation_cheap,correlation_smart,extraction).classify(text)is now the single place that disambiguates a depleted balance from an ordinary rate limit — both arrive as 429 — moved out ofllm_correlator.pyso it cannot drift into two copies.report_degraded(..., permanent=)alerts immediately for a permanent condition (retired model, dead billing) and only after 5 consecutive failures for a transient one. Once per episode, never per call — this runs at radio-traffic volume.report_healthy(tier)resets and posts a recovery notice, so a tier that comes back does not stay flagged.GET /health/aiexposes the snapshot. Deliberately separate from/healthso the CI deploy'scurl -f /healthliveness check keeps its existing meaning.ai_alert_webhook_url(new inconfig.py); empty means silently skip, since not every deployment will set one.transcription.pyandllm_correlator.pynow report through the registry on both failure and success; the ERROR logs are kept alongside, not replaced.14 new tests in
tests/test_ai_health.py. Suite: 107 passed, 0 failed.Set
AI_ALERT_WEBHOOK_URLon the server — without it this only improves/health/ai, and the whole point of the issue was getting the signal out of the box.Note the adjacent gap found the same night: deploys were failing silently for two days with nobody notified either (server-26#21). Same shape of problem, different subsystem.