OpenAI/Gemini clients have no request timeout -- sweep hold-back constants are guessed, not derived #153

New Issue

2026-09-14T00:25:33-04:00

logan commented

2026-09-14 00:25:33 -04:00

Found during drb-correlation-review on server-26#131's fix: neither transcription.py's OpenAI Whisper client (OpenAI(api_key=...), transcription.py:318) nor llm_correlator.py's Gemini call (generate_content(prompt), llm_correlator.py:196) sets a request_options/timeout. Both fall back to SDK defaults -- OpenAI's client default is 600s, doubled by stt_retry_on_degenerate (transcription.py:179) retrying once on a degenerate transcript.

Consequence: recorrelation_sweep.py's MIN_MINUTES_SINCE_PIPELINE_START (server-26#131, currently 15 min) is a guess against this unbounded tail, not a measured bound. A genuinely hung STT or Gemini call can run past any hold-back constant we pick, and the re-correlation sweep would race it again on its next tick -- reopening the exact duplicate-link bug #131 fixed, just for the pipeline's slow-tail population instead of its typical-case population.

Fix shape: set an explicit timeout on both clients (something well inside MAX_RECORDING_SECONDS territory, not SDK defaults), so MIN_MINUTES_SINCE_PIPELINE_START can be derived from a real upper bound instead of picked. Also caps spend on a hung request that would otherwise retry against OpenAI's/Google's default backoff indefinitely.

Not blocking #131 -- filed as the honest follow-up per drb-correlation-review's finding, not fixed in the same pass (matches the project's own precedent of separating instrument fixes from behavior fixes).

Found during `drb-correlation-review` on server-26#131's fix: neither `transcription.py`'s OpenAI Whisper client (`OpenAI(api_key=...)`, `transcription.py:318`) nor `llm_correlator.py`'s Gemini call (`generate_content(prompt)`, `llm_correlator.py:196`) sets a `request_options`/timeout. Both fall back to SDK defaults -- OpenAI's client default is 600s, doubled by `stt_retry_on_degenerate` (`transcription.py:179`) retrying once on a degenerate transcript. Consequence: `recorrelation_sweep.py`'s `MIN_MINUTES_SINCE_PIPELINE_START` (server-26#131, currently 15 min) is a guess against this unbounded tail, not a measured bound. A genuinely hung STT or Gemini call can run past any hold-back constant we pick, and the re-correlation sweep would race it again on its next tick -- reopening the exact duplicate-link bug #131 fixed, just for the pipeline's slow-tail population instead of its typical-case population. Fix shape: set an explicit timeout on both clients (something well inside `MAX_RECORDING_SECONDS` territory, not SDK defaults), so `MIN_MINUTES_SINCE_PIPELINE_START` can be derived from a real upper bound instead of picked. Also caps spend on a hung request that would otherwise retry against OpenAI's/Google's default backoff indefinitely. Not blocking #131 -- filed as the honest follow-up per drb-correlation-review's finding, not fixed in the same pass (matches the project's own precedent of separating instrument fixes from behavior fixes).

logan referenced this issue

2026-09-14 00:30:54 -04:00

correlator: stop the re-correlation sweep racing an in-flight upload (#131) #154

logan referenced this issue

2026-09-14 00:31:16 -04:00

Same call_id linked into 2 incident records with identical extracted data (~2% of linked calls) #131

logan referenced this issue from a commit

2026-10-04 18:00:40 -04:00

Explicit timeouts on every OpenAI and Gemini request (#153)

logan referenced a pull request that will close this issue

2026-10-04 18:00:52 -04:00

Explicit timeouts on every OpenAI and Gemini request (#153) #209

logan commented

2026-10-04 18:00:58 -04:00

PR: #209 (commit 278edf4). All OpenAI clients now built via app/internal/ai_clients.py with STT 120s / OpenAI 60s / 1 retry; Gemini client 60s. Timeouts take the existing fallback paths (rules_only, raw transcript, ai_health transient). MIN_MINUTES_SINCE_PIPELINE_START is now derived from those settings: 21 min with defaults (was a guessed 15). c2-core pytest 541 passed / 0 failed.

PR: https://git.vpn.cusano.net/logan/server-26/pulls/209 (commit 278edf4). All OpenAI clients now built via app/internal/ai_clients.py with STT 120s / OpenAI 60s / 1 retry; Gemini client 60s. Timeouts take the existing fallback paths (rules_only, raw transcript, ai_health transient). MIN_MINUTES_SINCE_PIPELINE_START is now derived from those settings: 21 min with defaults (was a guessed 15). c2-core pytest 541 passed / 0 failed.

logan closed this issue

2026-10-04 18:29:10 -04:00

logan referenced this issue from a commit

2026-10-04 18:29:10 -04:00

Merge pull request 'Explicit timeouts on every OpenAI and Gemini request (#153)' (#209) from fix/153-ai-request-timeouts into main

logan added the area/ai-pipeline

type

bug

labels 2026-10-05 02:04:53 -04:00

Sign in to join this conversation.