No AI client set a timeout: OpenAI's SDK waits 600s per attempt and retries
twice (timeouts included); google-genai waits forever. A hung provider call
had no upper bound, so recorrelation_sweep's 15-minute pipeline hold-back was
a guess against an unbounded tail.
- config: stt_timeout_seconds (120), openai_timeout_seconds (60),
openai_max_retries (1), gemini_timeout_seconds (60).
- app/internal/ai_clients.py builds every OpenAI/AsyncOpenAI client with them
(transcription, extraction, embeddings, summarizer, vocabulary learner,
trip chat). gemini.py's shared client gets HttpOptions(timeout=ms).
- A timeout raises like any other provider error, so existing fallbacks are
unchanged: rules-only correlation, rules decision on tiebreak, raw
transcript on correction, ai_health "transient".
- MIN_MINUTES_SINCE_PIPELINE_START is now derived from those timeouts
(pipeline_worst_case_seconds, 3-scene budget + 2 min non-AI slack), floor
15: 21 min with the defaults. Loop warns if it reaches the scan window.
- tests/test_ai_timeouts.py: client limits, a real hung-socket request that
gives up in <5s, timeout fallback per tier, the derived bound, and a guard
against any new OpenAI( outside ai_clients.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A day of replay runs (server-26#170) spent ~$5 of Gemini on ~7 two-hour
windows (~$0.70 per 290 calls) — several dollars a day per live deployment
for correlation alone — and nothing could say where it went (#45). Gemini
3.x thinks by default and bills it as output; the deprecated
google-generativeai SDK these calls used cannot set a thinking level.
- app/internal/gemini.py: every Gemini call (correlation + transcript
correction) goes through google-genai with JSON mode, an explicit
thinking level, and logs in/out/thinking tokens. A model that rejects
the level is retried without it once and remembered, so the tier is
never lost to a config param. API failures still raise for ai_health.
- correlator: thinking_level "minimal" (a link/new/orphan choice).
transcript correction: "low" until a replay shows minimal is safe.
- replay: runs record real Gemini token usage (metrics.gemini_usage),
shown in the Replay tab.
- requirements: google-genai.
c2-core: 474 pass. Frontend typecheck not run (no Node on this box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Logan CusanoandClaude Opus 5.5