Add AI provider degradation registry and alerting (server-26#14)
Three AI dependency failures in one night (retired Gemini model IDs, depleted Gemini balance, unpayable OpenAI account) each surfaced only as a single ERROR log line that nobody was watching. Add app/internal/ai_health.py, a shared in-memory registry that transcription.py and llm_correlator.py report into on every call (success and failure), distinguishing permanent conditions (dead model, dead billing) which alert immediately from transient ones (rate limits, network blips) which only alert after they persist. Alerts POST once per degradation episode and once on recovery to an optional Discord webhook (AI_ALERT_WEBHOOK_URL), reusing alerter.py's httpx pattern. State is exposed unauthenticated at GET /health/ai alongside the existing /health. Closes logan/server-26#14
This commit is contained in:
@@ -8,6 +8,7 @@ from app.internal.node_sweeper import sweeper_loop
|
||||
from app.internal.summarizer import summarizer_loop
|
||||
from app.internal.vocabulary_learner import vocabulary_induction_loop
|
||||
from app.internal.recorrelation_sweep import recorrelation_loop
|
||||
from app.internal import ai_health
|
||||
from app.config import settings
|
||||
from app.internal.auth import (
|
||||
require_firebase_token,
|
||||
@@ -120,3 +121,12 @@ app.include_router(media.router)
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"ok": True, "mqtt_connected": mqtt_handler.is_connected}
|
||||
|
||||
|
||||
# Deliberately unauthenticated, same as /health above: the CI deploy step
|
||||
# curls /health with no credentials, and this is diagnostic state (which AI
|
||||
# tier is degraded and why), not a secret — no API keys or tokens appear in
|
||||
# it. Keeping it auth-free means an external uptime check can watch it too.
|
||||
@app.get("/health/ai")
|
||||
async def health_ai():
|
||||
return {"tiers": ai_health.snapshot()}
|
||||
|
||||
Reference in New Issue
Block a user