Same metric in prior windows: 9-12 tiebreak 71 / agreed 53; 9-13 65/43; 9-14 74/43. So the Gemini tier worked on 09-14 and was fully dead during this run.
corr_cheap_model/corr_smart_model unchanged since 74faa55 (gemini-3.6-flash / gemini-2.5-pro), so not a config edit in this repo. Suspect key/billing/model retirement — needs c2-core logs (_log_llm_failure ERROR lines) to confirm; /health/ai is in-memory and was reset by later deploys (all tiers clean).
~174 decisions were non-thin with candidates present, so decide() should have called Gemini.
Blast radius: every correlation decision this run was rules-only; 203/377 (54%) linked via fast/thin. transcript_correction_model is also gemini-3.6-flash — if the cause is account-level, transcript correction was dead too, so the STT side of this run is uncorrected Whisper.
Recording vs behavior: this is a behavior outage that also taints the run as a measurement of the consensus correlator — any conclusion about LLM correlation from this window is invalid. Extraction (gpt-4o-mini) was working: 128/373 calls with units, 92 with location.
Also in the same dump (existing issue #155): 100/100 incidents resolved, 86 of them 60–99 min after their last call (idle timeout), only 1/100 with units_cleared.
Fix: pull c2-core logs for the ERROR, fix key/model, and make a dead tier alert survive a restart (ai_health is in-memory only).
Found 2026-09-26 mining the AI-on run (2026-09-20 19:26Z → ~2026-09-22 18:50Z), live SHA e79b8bc.
**Correlation debug dump (limit 100, orphan 168h, generated 2026-09-26T18:45:58Z), 377 scene decisions, all from 2026-09-22 11:34–18:50Z:**
- `corr_llm_action`: none = **377/377**. `corr_llm_reasoning` empty on all 377.
- `corr_consensus`: rules_only 369, none 8. **Zero** `agreed`/`tiebreak`.
- Same metric in prior windows: 9-12 tiebreak 71 / agreed 53; 9-13 65/43; 9-14 74/43. So the Gemini tier worked on 09-14 and was fully dead during this run.
- `corr_cheap_model`/`corr_smart_model` unchanged since 74faa55 (gemini-3.6-flash / gemini-2.5-pro), so not a config edit in this repo. Suspect key/billing/model retirement — **needs c2-core logs** (`_log_llm_failure` ERROR lines) to confirm; `/health/ai` is in-memory and was reset by later deploys (all tiers clean).
- ~174 decisions were non-thin with candidates present, so `decide()` should have called Gemini.
**Blast radius:** every correlation decision this run was rules-only; 203/377 (54%) linked via `fast/thin`. `transcript_correction_model` is also gemini-3.6-flash — if the cause is account-level, transcript correction was dead too, so the STT side of this run is uncorrected Whisper.
**Recording vs behavior:** this is a behavior outage that also taints the run as a measurement of the consensus correlator — any conclusion about LLM correlation from this window is invalid. Extraction (gpt-4o-mini) was working: 128/373 calls with units, 92 with location.
Also in the same dump (existing issue #155): 100/100 incidents resolved, 86 of them 60–99 min after their last call (idle timeout), only 1/100 with `units_cleared`.
Fix: pull c2-core logs for the ERROR, fix key/model, and make a dead tier alert survive a restart (ai_health is in-memory only).
Owner: OpenAI credits ran out on 2026-09-22. That matches the run ending ~18:50Z that day (Whisper STT + gpt-4o-mini extraction are both OpenAI). It does not explain the Gemini tier: Gemini was already returning nothing from 11:34Z onward while OpenAI extraction was still working. Gemini billing/key not yet checked. Prod SSH not possible from the owner workstation (no key; deploy key only exists as Gitea secret SSH_PRIVATE_KEY). ai_health Discord alert only posts if AI_ALERT_WEBHOOK_URL is set on the VM.
Owner: OpenAI credits ran out on 2026-09-22. That matches the run ending ~18:50Z that day (Whisper STT + gpt-4o-mini extraction are both OpenAI). It does not explain the Gemini tier: Gemini was already returning nothing from 11:34Z onward while OpenAI extraction was still working. Gemini billing/key not yet checked. Prod SSH not possible from the owner workstation (no key; deploy key only exists as Gitea secret SSH_PRIVATE_KEY). ai_health Discord alert only posts if AI_ALERT_WEBHOOK_URL is set on the VM.
Root cause confirmed by owner 2026-09-26: Gemini billing exhausted (and OpenAI billing exhausted 09-22). Not a code or model bug. Remaining code gap: billing death is classified (ai_health.classify -> billing) but the only alert is an optional Discord webhook plus in-memory state wiped on restart, so it went unnoticed for days. Keep this open until a billing-dead tier is loud and durable (e.g., persisted state + shown in /admin AI Features).
Root cause confirmed by owner 2026-09-26: **Gemini billing exhausted** (and OpenAI billing exhausted 09-22). Not a code or model bug. Remaining code gap: billing death is classified (ai_health.classify -> billing) but the only alert is an optional Discord webhook plus in-memory state wiped on restart, so it went unnoticed for days. Keep this open until a billing-dead tier is loud and durable (e.g., persisted state + shown in /admin AI Features).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found 2026-09-26 mining the AI-on run (2026-09-20 19:26Z → ~2026-09-22 18:50Z), live SHA
e79b8bc.Correlation debug dump (limit 100, orphan 168h, generated 2026-09-26T18:45:58Z), 377 scene decisions, all from 2026-09-22 11:34–18:50Z:
corr_llm_action: none = 377/377.corr_llm_reasoningempty on all 377.corr_consensus: rules_only 369, none 8. Zeroagreed/tiebreak.corr_cheap_model/corr_smart_modelunchanged since74faa55(gemini-3.6-flash / gemini-2.5-pro), so not a config edit in this repo. Suspect key/billing/model retirement — needs c2-core logs (_log_llm_failureERROR lines) to confirm;/health/aiis in-memory and was reset by later deploys (all tiers clean).decide()should have called Gemini.Blast radius: every correlation decision this run was rules-only; 203/377 (54%) linked via
fast/thin.transcript_correction_modelis also gemini-3.6-flash — if the cause is account-level, transcript correction was dead too, so the STT side of this run is uncorrected Whisper.Recording vs behavior: this is a behavior outage that also taints the run as a measurement of the consensus correlator — any conclusion about LLM correlation from this window is invalid. Extraction (gpt-4o-mini) was working: 128/373 calls with units, 92 with location.
Also in the same dump (existing issue #155): 100/100 incidents resolved, 86 of them 60–99 min after their last call (idle timeout), only 1/100 with
units_cleared.Fix: pull c2-core logs for the ERROR, fix key/model, and make a dead tier alert survive a restart (ai_health is in-memory only).
Owner: OpenAI credits ran out on 2026-09-22. That matches the run ending ~18:50Z that day (Whisper STT + gpt-4o-mini extraction are both OpenAI). It does not explain the Gemini tier: Gemini was already returning nothing from 11:34Z onward while OpenAI extraction was still working. Gemini billing/key not yet checked. Prod SSH not possible from the owner workstation (no key; deploy key only exists as Gitea secret SSH_PRIVATE_KEY). ai_health Discord alert only posts if AI_ALERT_WEBHOOK_URL is set on the VM.
Root cause confirmed by owner 2026-09-26: Gemini billing exhausted (and OpenAI billing exhausted 09-22). Not a code or model bug. Remaining code gap: billing death is classified (ai_health.classify -> billing) but the only alert is an optional Discord webhook plus in-memory state wiped on restart, so it went unnoticed for days. Keep this open until a billing-dead tier is loud and durable (e.g., persisted state + shown in /admin AI Features).