LLM-orphan gate's "type" veto fires on visibly content-free chatter (18/22 this window) #138

New Issue

2026-09-13T17:21:13-04:00

logan commented

2026-09-13 17:21:13 -04:00

Window #4 measurement (CORRELATION_REVIEW_0913.md, server-26#115 comment): the consensus gate's _call_is_substanceless (routers/upload.py) vetoes on ctx.get("incident_type") truthy 18/22 times this window (82% of all orphan-vs-new disagreements) -- and every one of those 18 transcripts is plainly radio housekeeping with zero incident content:

"Alpha 4, report. First to Alpha 4."
"6-7, this is Hunter. 6-7. 6-7, this is Hunter. 10-4."
"45 Adam, I'm clear with the report. Roger, thank you."

Yet the call's own final persisted incident_type field is None for all 18 -- the opposite of what triggered the veto. ctx["incident_type"] is passed in-memory from the scene's own extraction result at correlation time (not re-read from Firestore), so either:

intelligence.py's scene extraction is assigning an incident_type to content-free chatter it shouldn't, or
On a multi-scene call, an earlier scene's real type reaches this scene's ctx before this scene's own (correctly-None) extraction overwrites the shared top-level field -- a cross-scene contamination bug distinct from #96/#114's last-write-wins on the flat field.

Can't distinguish the two from the dump: see server-26#96 companion issue for capturing incident_type per-scene so the next window can tell them apart directly.

This is the dominant live reason the LLM-orphan gate (#115/#125/#126) has fired 0 times across 2 real measurement windows -- not the escape hatch. Read the extraction prompt in intelligence.py against 5-10 of these transcripts as the next step.

Window #4 measurement (`CORRELATION_REVIEW_0913.md`, server-26#115 comment): the consensus gate's `_call_is_substanceless` (routers/upload.py) vetoes on `ctx.get("incident_type")` truthy 18/22 times this window (82% of all orphan-vs-new disagreements) -- and every one of those 18 transcripts is plainly radio housekeeping with zero incident content: > "Alpha 4, report. First to Alpha 4." > "6-7, this is Hunter. 6-7. 6-7, this is Hunter. 10-4." > "45 Adam, I'm clear with the report. Roger, thank you." Yet the call's own final persisted `incident_type` field is `None` for all 18 -- the opposite of what triggered the veto. `ctx["incident_type"]` is passed in-memory from the scene's own extraction result at correlation time (not re-read from Firestore), so either: 1. `intelligence.py`'s scene extraction is assigning an `incident_type` to content-free chatter it shouldn't, or 2. On a multi-scene call, an earlier scene's real type reaches this scene's `ctx` before this scene's own (correctly-None) extraction overwrites the shared top-level field -- a cross-scene contamination bug distinct from #96/#114's last-write-wins on the flat field. Can't distinguish the two from the dump: see server-26#96 companion issue for capturing `incident_type` per-scene so the next window can tell them apart directly. This is the dominant live reason the LLM-orphan gate (#115/#125/#126) has fired 0 times across 2 real measurement windows -- not the escape hatch. Read the extraction prompt in `intelligence.py` against 5-10 of these transcripts as the next step.

logan referenced this issue

2026-09-13 17:32:43 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CFO draft #140

logan referenced this issue

2026-09-13 17:33:31 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CTO draft #141

logan referenced this issue

2026-09-13 18:49:36 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — FINAL MINUTES #146

logan referenced this issue

2026-09-13 18:50:15 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CTO draft #141

logan referenced this issue from a commit

2026-09-14 00:14:35 -04:00

correlator/admin: capture per-scene incident_type + severity (#139)

logan referenced this issue

2026-09-14 00:14:48 -04:00

correlator/admin: capture per-scene incident_type + severity (#139) #152

logan referenced this issue

2026-09-14 00:15:05 -04:00

Per-scene scenes map (#96/#114) doesn't capture incident_type -- can't audit the gate's "type" veto per scene #139

logan commented

2026-09-14 04:54:53 -04:00

Resolved — window #5 measurement (CORRELATION_REVIEW_0914.md), with #139 live this settles it.

10 true in-window orphan-vs-new disagreements, 9 vetoed on "type". All 9 now show a genuinely truthy incident_type on their OWN scene record (not a sibling-scene contamination artifact) -- e.g. "7 for Post Control. 7 for Post..." typed police; "Bell, do you want an out number?" typed other.

Root cause found in intelligence.py's EXTRACTION_PROMPT (line 69): "When the channel is a police channel and nothing in the transcript contradicts it, return 'police' -- do NOT fall back to 'other' merely because the transmission is administrative." This instruction exists to stop real incidents being under-classified, but its side effect is that nearly all police-channel chatter gets a resolved incident_type -- which is the first thing _call_is_substanceless checks. The gate and the extraction prompt are working against each other by design, not by bug.

Not fixed in this pass -- a prompt change needs its own backtest against real incidents to confirm it doesn't reintroduce #115's original failure (a gate that drops genuine events). Two shapes to weigh: narrow the police-channel default to exclude pure status/administrative traffic, or change _call_is_substanceless to not treat incident_type alone as disqualifying when severity is routine and no event substance exists (riskier -- could re-open #115).

Chatter classifier: 0/9 (0%) of this population flagged -- confirms it doesn't cover this class of chatter at all.

**Resolved — window #5 measurement (`CORRELATION_REVIEW_0914.md`), with #139 live this settles it.** 10 true in-window orphan-vs-new disagreements, 9 vetoed on `"type"`. All 9 now show a genuinely truthy `incident_type` on their OWN scene record (not a sibling-scene contamination artifact) -- e.g. "7 for Post Control. 7 for Post..." typed `police`; "Bell, do you want an out number?" typed `other`. **Root cause found in `intelligence.py`'s EXTRACTION_PROMPT (line 69)**: "When the channel is a police channel and nothing in the transcript contradicts it, return 'police' -- do NOT fall back to 'other' merely because the transmission is administrative." This instruction exists to stop real incidents being under-classified, but its side effect is that nearly all police-channel chatter gets a resolved `incident_type` -- which is the first thing `_call_is_substanceless` checks. The gate and the extraction prompt are working against each other by design, not by bug. **Not fixed in this pass** -- a prompt change needs its own backtest against real incidents to confirm it doesn't reintroduce #115's original failure (a gate that drops genuine events). Two shapes to weigh: narrow the police-channel default to exclude pure status/administrative traffic, or change `_call_is_substanceless` to not treat `incident_type` alone as disqualifying when severity is routine and no event substance exists (riskier -- could re-open #115). Chatter classifier: 0/9 (0%) of this population flagged -- confirms it doesn't cover this class of chatter at all.

logan referenced this issue from a commit

2026-09-20 15:43:02 -04:00

intelligence: stop the police-channel default from typing content-free chatter (#138)

logan commented

2026-09-20 15:51:17 -04:00

Fixed this session, commit 3f69879 (deployed, confirmed via /health). intelligence.py's incident_type rule now checks for real event content before applying the channel default — content-free/administrative-only transmissions return "unknown" (normalizes to None) on every channel instead of defaulting to "police", so the creation gate correctly vetoes them. No correlator/gate code touched, per this issue's own recommendation to fix it in the prompt. Spot-checked immediately post-deploy: too little live traffic yet to confirm (AI features were off since 9/14, just turned back on). Real verification pending a scheduled review in ~2h — will comment again with results. Related: server-26#155 (same session, different mechanism).

Fixed this session, commit 3f69879 (deployed, confirmed via /health). intelligence.py's incident_type rule now checks for real event content before applying the channel default — content-free/administrative-only transmissions return "unknown" (normalizes to None) on every channel instead of defaulting to "police", so the creation gate correctly vetoes them. No correlator/gate code touched, per this issue's own recommendation to fix it in the prompt. Spot-checked immediately post-deploy: too little live traffic yet to confirm (AI features were off since 9/14, just turned back on). Real verification pending a scheduled review in ~2h — will comment again with results. Related: server-26#155 (same session, different mechanism).

logan referenced this issue from a commit

2026-09-26 20:22:04 -04:00

intelligence: traffic stops and self-initiated activity open incidents

logan added the area/ai-pipeline area/correlation

labels 2026-10-05 02:04:49 -04:00

logan referenced this issue