Add AI provider degradation registry and alerting (server-26#14)
Build & Deploy / Build & push images (push) Successful in 4m18s
Build & Deploy / Deploy to VM (push) Successful in 1m35s

Three AI dependency failures in one night (retired Gemini model IDs,
depleted Gemini balance, unpayable OpenAI account) each surfaced only
as a single ERROR log line that nobody was watching. Add
app/internal/ai_health.py, a shared in-memory registry that
transcription.py and llm_correlator.py report into on every call
(success and failure), distinguishing permanent conditions (dead
model, dead billing) which alert immediately from transient ones
(rate limits, network blips) which only alert after they persist.
Alerts POST once per degradation episode and once on recovery to an
optional Discord webhook (AI_ALERT_WEBHOOK_URL), reusing alerter.py's
httpx pattern. State is exposed unauthenticated at GET /health/ai
alongside the existing /health.

Closes logan/server-26#14
This commit is contained in:
Logan Cusano
2026-08-20 03:14:22 -04:00
parent 5355095c48
commit a250c29e3c
6 changed files with 477 additions and 32 deletions
+6
View File
@@ -101,6 +101,12 @@ class Settings(BaseSettings):
# Defaults to "*" for local development only.
cors_origins: list[str] = ["*"]
# Discord webhook URL that app/internal/ai_health.py posts to when an AI
# tier (transcription/correlation) transitions into or out of degraded
# state. Empty disables the POST entirely — not every self-hosted
# deployment will set this up, and skipping it must be silent.
ai_alert_webhook_url: str = ""
class Config:
env_file = ".env"