Gemini correlation tier returned nothing for the whole 09-20→09-22 AI run — 377/377 decisions rules-only #169

New Issue

2026-09-26T14:48:53-04:00

logan commented

2026-09-26 14:48:53 -04:00

Found 2026-09-26 mining the AI-on run (2026-09-20 19:26Z → ~2026-09-22 18:50Z), live SHA e79b8bc.

Correlation debug dump (limit 100, orphan 168h, generated 2026-09-26T18:45:58Z), 377 scene decisions, all from 2026-09-22 11:34–18:50Z:

corr_llm_action: none = 377/377. corr_llm_reasoning empty on all 377.
corr_consensus: rules_only 369, none 8. Zero agreed/tiebreak.
Same metric in prior windows: 9-12 tiebreak 71 / agreed 53; 9-13 65/43; 9-14 74/43. So the Gemini tier worked on 09-14 and was fully dead during this run.
corr_cheap_model/corr_smart_model unchanged since 74faa55 (gemini-3.6-flash / gemini-2.5-pro), so not a config edit in this repo. Suspect key/billing/model retirement — needs c2-core logs (_log_llm_failure ERROR lines) to confirm; /health/ai is in-memory and was reset by later deploys (all tiers clean).
~174 decisions were non-thin with candidates present, so decide() should have called Gemini.

Blast radius: every correlation decision this run was rules-only; 203/377 (54%) linked via fast/thin. transcript_correction_model is also gemini-3.6-flash — if the cause is account-level, transcript correction was dead too, so the STT side of this run is uncorrected Whisper.

Recording vs behavior: this is a behavior outage that also taints the run as a measurement of the consensus correlator — any conclusion about LLM correlation from this window is invalid. Extraction (gpt-4o-mini) was working: 128/373 calls with units, 92 with location.

Also in the same dump (existing issue #155): 100/100 incidents resolved, 86 of them 60–99 min after their last call (idle timeout), only 1/100 with units_cleared.

Fix: pull c2-core logs for the ERROR, fix key/model, and make a dead tier alert survive a restart (ai_health is in-memory only).

Found 2026-09-26 mining the AI-on run (2026-09-20 19:26Z → ~2026-09-22 18:50Z), live SHA e79b8bc. **Correlation debug dump (limit 100, orphan 168h, generated 2026-09-26T18:45:58Z), 377 scene decisions, all from 2026-09-22 11:34–18:50Z:** - `corr_llm_action`: none = **377/377**. `corr_llm_reasoning` empty on all 377. - `corr_consensus`: rules_only 369, none 8. **Zero** `agreed`/`tiebreak`. - Same metric in prior windows: 9-12 tiebreak 71 / agreed 53; 9-13 65/43; 9-14 74/43. So the Gemini tier worked on 09-14 and was fully dead during this run. - `corr_cheap_model`/`corr_smart_model` unchanged since 74faa55 (gemini-3.6-flash / gemini-2.5-pro), so not a config edit in this repo. Suspect key/billing/model retirement — **needs c2-core logs** (`_log_llm_failure` ERROR lines) to confirm; `/health/ai` is in-memory and was reset by later deploys (all tiers clean). - ~174 decisions were non-thin with candidates present, so `decide()` should have called Gemini. **Blast radius:** every correlation decision this run was rules-only; 203/377 (54%) linked via `fast/thin`. `transcript_correction_model` is also gemini-3.6-flash — if the cause is account-level, transcript correction was dead too, so the STT side of this run is uncorrected Whisper. **Recording vs behavior:** this is a behavior outage that also taints the run as a measurement of the consensus correlator — any conclusion about LLM correlation from this window is invalid. Extraction (gpt-4o-mini) was working: 128/373 calls with units, 92 with location. Also in the same dump (existing issue #155): 100/100 incidents resolved, 86 of them 60–99 min after their last call (idle timeout), only 1/100 with `units_cleared`. Fix: pull c2-core logs for the ERROR, fix key/model, and make a dead tier alert survive a restart (ai_health is in-memory only).

logan commented

2026-09-26 14:57:10 -04:00

Owner: OpenAI credits ran out on 2026-09-22. That matches the run ending ~18:50Z that day (Whisper STT + gpt-4o-mini extraction are both OpenAI). It does not explain the Gemini tier: Gemini was already returning nothing from 11:34Z onward while OpenAI extraction was still working. Gemini billing/key not yet checked. Prod SSH not possible from the owner workstation (no key; deploy key only exists as Gitea secret SSH_PRIVATE_KEY). ai_health Discord alert only posts if AI_ALERT_WEBHOOK_URL is set on the VM.

logan commented

2026-09-26 14:59:26 -04:00

Root cause confirmed by owner 2026-09-26: Gemini billing exhausted (and OpenAI billing exhausted 09-22). Not a code or model bug. Remaining code gap: billing death is classified (ai_health.classify -> billing) but the only alert is an optional Discord webhook plus in-memory state wiped on restart, so it went unnoticed for days. Keep this open until a billing-dead tier is loud and durable (e.g., persisted state + shown in /admin AI Features).

Root cause confirmed by owner 2026-09-26: **Gemini billing exhausted** (and OpenAI billing exhausted 09-22). Not a code or model bug. Remaining code gap: billing death is classified (ai_health.classify -> billing) but the only alert is an optional Discord webhook plus in-memory state wiped on restart, so it went unnoticed for days. Keep this open until a billing-dead tier is loud and durable (e.g., persisted state + shown in /admin AI Features).

logan referenced this issue

2026-09-26 15:15:54 -04:00

Admin Replay: re-run the pipeline over past calls in a sandbox, so correlation changes are measured without live AI windows #170

logan referenced this issue from a commit

2026-09-26 15:24:01 -04:00

Admin Replay: re-run the pipeline over past calls in a sandbox (#170)

logan referenced this issue

2026-09-26 15:39:03 -04:00

Replay: fail fast on a dead AI account; extraction reports to ai_health #172

logan referenced this issue from a commit

2026-09-26 15:39:04 -04:00

Replay: fail fast on a dead AI account; extraction reports to ai_health