Correlation has no non-AI fallback: a full AI outage produces a 100% orphan rate for its whole duration #15

New Issue

2026-08-19T22:34:33-04:00

logan commented

2026-08-19 22:34:33 -04:00

Measured via debug_correlation() against production data (100 incidents, 2322 orphaned calls, orphan window 2026-08-19T23:05Z - 2026-08-20T02:22Z).

AI processing was manually switched off overnight and back on at ~2026-08-20 02:14Z. For the full outage window:

0 incidents created (started_at in range: 0 of 100 pulled incidents)
0 calls linked to any incident (calls_detail entries with started_at in range: 0)
2322 calls orphaned, of which the 250 sampled in the debug payload all have incident_type=None; 246/250 have an empty transcript (STT was also off, not just the correlator/Gemini tier); corr_sweep_count=3 (max) on 247/250, i.e. the sweep already gave up retrying every one of them.
Recovery was clean: the first successful correlation lands at 2026-08-20T02:15:48Z (unit-continuity), 94 seconds after the 02:14Z restart, and "new" incidents resume forming within ~4 minutes.

Root cause is structural, not a glitch: the 96625fa creation gate requires severity, a vehicle, a geocode, or a tag before a call can open a new incident, and all four of those come from the (Gemini) intelligence pipeline. With AI off, literally nothing can open an incident, so fast/thin's tier-1/tier-2 attach (which requires an already-open incident on the same TGID) has nothing to attach to either. The two biggest hit talkgroups in the orphan breakdown are TG 9048 (672 orphans) and TG 9600 (349 orphans) - exactly the canary DEFERRED.md already named ("Canary: orphan count climbing on TG 9048/9600") for this exact gate, but the actual failure mode is a full blackout, not a gradual climb.

This is distinct from #14 (no alerting when a provider dies) - even with alerting, the current design has no rules-only path to keep opening/attaching incidents while AI is down; every call during an outage is unrecoverably lost to correlation (the sweep gives up after 3 tries and there's no backfill/reprocess for orphaned calls once AI comes back).

Suggested shape: either (a) let units/location-only calls satisfy the creation gate again specifically when AI is confirmed down (feature-flag-aware), or (b) add a reprocessing sweep that revisits recently-orphaned calls once AI comes back online, similar in spirit to the existing /calls/{id}/reprocess route but bulk and orphan-targeted.

Measured via `debug_correlation()` against production data (100 incidents, 2322 orphaned calls, orphan window 2026-08-19T23:05Z - 2026-08-20T02:22Z). AI processing was manually switched off overnight and back on at ~2026-08-20 02:14Z. For the full outage window: - **0** incidents created (`started_at` in range: 0 of 100 pulled incidents) - **0** calls linked to any incident (`calls_detail` entries with `started_at` in range: 0) - **2322** calls orphaned, of which the 250 sampled in the debug payload all have `incident_type=None`; 246/250 have an **empty transcript** (STT was also off, not just the correlator/Gemini tier); `corr_sweep_count=3` (max) on 247/250, i.e. the sweep already gave up retrying every one of them. - Recovery was clean: the first successful correlation lands at 2026-08-20T02:15:48Z (`unit-continuity`), 94 seconds after the 02:14Z restart, and "new" incidents resume forming within ~4 minutes. Root cause is structural, not a glitch: the 96625fa creation gate requires severity, a vehicle, a geocode, or a tag before a call can open a new incident, and all four of those come from the (Gemini) intelligence pipeline. With AI off, literally nothing can open an incident, so `fast/thin`'s tier-1/tier-2 attach (which requires an *already-open* incident on the same TGID) has nothing to attach to either. The two biggest hit talkgroups in the orphan breakdown are TG 9048 (672 orphans) and TG 9600 (349 orphans) - exactly the canary DEFERRED.md already named ("Canary: orphan count climbing on TG 9048/9600") for this exact gate, but the actual failure mode is a full blackout, not a gradual climb. This is distinct from #14 (no alerting when a provider dies) - even with alerting, the current design has no rules-only path to keep opening/attaching incidents while AI is down; every call during an outage is unrecoverably lost to correlation (the sweep gives up after 3 tries and there's no backfill/reprocess for orphaned calls once AI comes back). Suggested shape: either (a) let `units`/`location`-only calls satisfy the creation gate again specifically when AI is confirmed down (feature-flag-aware), or (b) add a reprocessing sweep that revisits recently-orphaned calls once AI comes back online, similar in spirit to the existing `/calls/{id}/reprocess` route but bulk and orphan-targeted.

logan commented

2026-08-19 22:36:25 -04:00

Closing — this is expected behaviour, not a defect.

routers/upload.py:260-285 gates transcription on stt_enabled and correlation on correlation_enabled. Both flags were deliberately set to False by the owner overnight to control cost. Zero incidents created during that window is exactly what turning AI off means; the creation gate from 96625fa is not implicated.

The one real observation buried in here is that the flags are not independent: correlation_enabled without stt_enabled does nothing at all, because there is no transcript to correlate on. That is worth documenting on the admin toggle so nobody enables correlation alone and expects results, but it is a UI/label matter, not a correlation bug.

The genuine consequence of the off-window is that those calls have audio and no transcript, so they can never correlate unless reprocessed. That is already tracked in #7 (bulk reprocess), whose scope this enlarges — noting it there.

Closing — this is expected behaviour, not a defect. `routers/upload.py:260-285` gates transcription on `stt_enabled` and correlation on `correlation_enabled`. Both flags were deliberately set to False by the owner overnight to control cost. Zero incidents created during that window is exactly what turning AI off means; the creation gate from 96625fa is not implicated. The one real observation buried in here is that the flags are not independent: `correlation_enabled` without `stt_enabled` does nothing at all, because there is no transcript to correlate on. That is worth documenting on the admin toggle so nobody enables correlation alone and expects results, but it is a UI/label matter, not a correlation bug. The genuine consequence of the off-window is that those calls have audio and no transcript, so they can never correlate unless reprocessed. That is already tracked in #7 (bulk reprocess), whose scope this enlarges — noting it there.

logan closed this issue

2026-08-19 22:36:25 -04:00

logan referenced this issue from a commit

2026-10-04 01:10:50 -04:00

approve_node never rotates an existing key; node update-check endpoint; real Update Node button (node-26#14, #15)

logan referenced this issue

2026-10-04 01:11:24 -04:00

approve_node never rotates an existing key; node update-check endpoint; real Update Node button (node-26#14, #15) #194

logan referenced this issue from a commit

2026-10-04 01:17:33 -04:00

Merge pull request 'approve_node never rotates an existing key; node update-check endpoint; real Update Node button (node-26#14, #15)' (#194) from fix/node-14-15-update-endpoint into main

logan added the area/correlation