Upstream dispatch-vs-chatter classifier — shadow mode #127

New Issue

2026-09-12T23:58:06-04:00

logan commented

2026-09-12 23:58:06 -04:00

Per #115's decision (comments 2026-09-08 and 2026-09-12) and both CORRELATION_REVIEW_0907b.md / CORRELATION_REVIEW_0912.md: two consensus-layer fixes (#125, #126) have now both measured as no-ops on the top-line new-incident rate. The disagreement shape they target (corr_llm_action==orphan, corr_rules_action==new) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside the gate's scope entirely, and the largest single bucket (15/57) is agreed — both correlator tiers independently called it a new incident, with no disagreement for any consensus-layer fix to arbitrate.

Decision (both review docs): stop iterating the consensus layer. Move the fix upstream — a dispatch-vs-chatter classifier in scene extraction (intelligence.py) so radio housekeeping never produces a scene at all.

What this issue ships

drb-c2-core/app/internal/chatter_classifier.py — a pure function classify_chatter(transcript) -> (bool, reason) that recognizes two shapes, drawn from hand-labeled examples in the review docs cross-referenced against the real dumps (not invented regexes):

roll_call — transcript contains a roll-call announcement or line-up ("stand by for roll call", "Baker 1? Baker 1. Henry 1? ...")
bare_acknowledgement — every token in the transcript reduces to a callsign/10-code/unit-ID (any digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...). Any other word — a location, an action, a description — leaves the transcript unclassified.

Deliberately conservative: when a transcript doesn't cleanly reduce, it returns (False, None) and the existing pipeline runs exactly as today. Precision over recall — a false positive here would (once live) silently mean a real event never gets a scene.

SHIPPED IN SHADOW MODE ONLY. intelligence.extract_scenes now computes the verdict alongside the existing _is_garbage_transcript / transcript_too_short gates and writes chatter_classifier_verdict (bool) and chatter_classifier_reason (str|None) onto the call doc — but does not skip scene extraction. Scene extraction runs exactly as it does today regardless of the verdict. The two fields are now surfaced in /admin's correlation-debug _call_summary, same pattern as corr_gate_veto (#115/#126), so the next live measurement window can report the real-world false-positive rate before anything is wired to actually skip extraction.

Backtest (offline, against the three existing dumps — see PR for script)

dump	total calls	flagged chatter
`corr_dump_9-7_0437am.json` (window #1)	397	55
`corr_dump_9-7_pm.json` (window #2)	333	72
`corr_dump_9-12.json` (window #3)	272	27
total	1002	154

* false positive = a flagged call carrying tags, coords, non-routine severity, or one of the specific transcripts the review docs named as dangerous to drop (the major "extinguishing fire" call, the geocoded 911-hangup call, both pursuit updates, the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call) — all individually re-checked and confirmed (False, None).

Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset of true housekeeping that also includes some mid-conversation follow-up fragments of real events, so this undercounts precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low — the classifier only recognizes the two shapes above, not every housekeeping pattern (garbled spelled-out roll call like "Five units. Six hundred. Eighty-four. Ten-four.", ambiguous check-ins with a stray content word like "7-4, call Central. 6-9, location.", CID/plate/case-number readbacks that mix in other words) were deliberately left uncaught rather than risk a broader net.

Tests

drb-c2-core/tests/test_chatter_classifier.py — real transcripts pulled from the dumps and review docs, both directions: chatter examples that must classify True, and the review docs' own named-dangerous real events that must classify False.

Next step — NOT done here

This is shadow mode only. Nothing skips scene extraction yet. A # TODO(server-26#<this-issue>) at the call site marks exactly what has to be true before flipping it live: run at least one more live shadow-mode measurement window, confirm the false-positive rate against real production traffic is still zero (this backtest only proves it against three already-collected dumps), and only then wire the verdict to actually skip extract_scenes.

Links: #115, CORRELATION_REVIEW_0907.md, CORRELATION_REVIEW_0907b.md, CORRELATION_REVIEW_0912.md.

Per #115's decision (comments 2026-09-08 and 2026-09-12) and both `CORRELATION_REVIEW_0907b.md` / `CORRELATION_REVIEW_0912.md`: two consensus-layer fixes (#125, #126) have now both measured as no-ops on the top-line new-incident rate. The disagreement shape they target (`corr_llm_action==orphan`, `corr_rules_action==new`) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside the gate's scope entirely, and the largest single bucket (15/57) is `agreed` — both correlator tiers independently called it a new incident, with no disagreement for any consensus-layer fix to arbitrate. **Decision (both review docs): stop iterating the consensus layer. Move the fix upstream — a dispatch-vs-chatter classifier in scene extraction (`intelligence.py`) so radio housekeeping never produces a scene at all.** ## What this issue ships `drb-c2-core/app/internal/chatter_classifier.py` — a pure function `classify_chatter(transcript) -> (bool, reason)` that recognizes two shapes, drawn from hand-labeled examples in the review docs cross-referenced against the real dumps (not invented regexes): - **`roll_call`** — transcript contains a roll-call announcement or line-up ("stand by for roll call", "Baker 1? Baker 1. Henry 1? ...") - **`bare_acknowledgement`** — every token in the transcript reduces to a callsign/10-code/unit-ID (any digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...). Any other word — a location, an action, a description — leaves the transcript unclassified. Deliberately conservative: when a transcript doesn't cleanly reduce, it returns `(False, None)` and the existing pipeline runs exactly as today. Precision over recall — a false positive here would (once live) silently mean a real event never gets a scene. **SHIPPED IN SHADOW MODE ONLY.** `intelligence.extract_scenes` now computes the verdict alongside the existing `_is_garbage_transcript` / `transcript_too_short` gates and writes `chatter_classifier_verdict` (bool) and `chatter_classifier_reason` (str|None) onto the call doc — but does **not** skip scene extraction. Scene extraction runs exactly as it does today regardless of the verdict. The two fields are now surfaced in `/admin`'s correlation-debug `_call_summary`, same pattern as `corr_gate_veto` (#115/#126), so the next live measurement window can report the real-world false-positive rate before anything is wired to actually skip extraction. ## Backtest (offline, against the three existing dumps — see PR for script) | dump | total calls | flagged chatter | false positives* | |---|---|---|---| | `corr_dump_9-7_0437am.json` (window #1) | 397 | 55 | **0** | | `corr_dump_9-7_pm.json` (window #2) | 333 | 72 | **0** | | `corr_dump_9-12.json` (window #3) | 272 | 27 | **0** | | **total** | **1002** | **154** | **0** | \* false positive = a flagged call carrying tags, coords, non-routine severity, or one of the specific transcripts the review docs named as dangerous to drop (the major "extinguishing fire" call, the geocoded 911-hangup call, both pursuit updates, the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call) — all individually re-checked and confirmed `(False, None)`. Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset of true housekeeping that also includes some mid-conversation follow-up fragments of real events, so this undercounts precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low — the classifier only recognizes the two shapes above, not every housekeeping pattern (garbled spelled-out roll call like "Five units. Six hundred. Eighty-four. Ten-four.", ambiguous check-ins with a stray content word like "7-4, call Central. 6-9, location.", CID/plate/case-number readbacks that mix in other words) were deliberately left uncaught rather than risk a broader net. ## Tests `drb-c2-core/tests/test_chatter_classifier.py` — real transcripts pulled from the dumps and review docs, both directions: chatter examples that must classify `True`, and the review docs' own named-dangerous real events that must classify `False`. ## Next step — NOT done here **This is shadow mode only. Nothing skips scene extraction yet.** A `# TODO(server-26#<this-issue>)` at the call site marks exactly what has to be true before flipping it live: run at least one more live shadow-mode measurement window, confirm the false-positive rate against real production traffic is still zero (this backtest only proves it against three already-collected dumps), and only then wire the verdict to actually skip `extract_scenes`. Links: #115, `CORRELATION_REVIEW_0907.md`, `CORRELATION_REVIEW_0907b.md`, `CORRELATION_REVIEW_0912.md`.

logan referenced a pull request that will close this issue

2026-09-13 00:00:33 -04:00

intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127) #128

logan referenced this issue from a commit

2026-09-13 12:42:48 -04:00

intelligence: run the chatter classifier before the too-short skip, not after (#127)

logan commented

2026-09-13 12:43:09 -04:00

Review found and fixed two real gaps in the shadow-mode PR (#128, commit 3ae0bb2):

Placement bug (would have made the shadow data itself misleading): the classifier ran after the ≤5-word transcript_too_short skip, but 82% of its backtest flags were on ≤5-word transcripts — a live shadow window would have shown roughly a fifth of the real catch rate. Moved the classification ahead of that check, folded into the existing doc_set (no extra write).
No aggregate tally: admin.py's summary only had per-call fields, same gap corr_gate_veto (#126) fixed for the consensus gate. Added chatter_classifier_flagged/chatter_classifier_reason to the tally, spanning both linked calls and orphans — the target population (non-events) lands in both places, so linked alone undercounts it.

Confirmed still zero behavior change: nothing reads the verdict, extraction runs unconditionally either way. Sandboxed pytest 364 green (unchanged from the PR's own count — this was a placement/observability fix, not new logic).

Not fixed, flagged for the go-live decision instead of touched now: the review found the classifier's shapes are riskier than the offline backtest could show —

roll_call matches "roll call" anywhere in the transcript, not just whole-transcript ("Delayed for roll call, then respond to a structure fire at 12 Main Street" → flagged).
digit-hyphen street addresses match the bare-acknowledgement unit-ID pattern (the dumps already contain "72-Holland" — Whisper does emit this shape for real addresses).
some phonetic-alphabet callsign words are also real street names ("Central to 6 Henry, show me out at Lincoln and Union" → flagged).

All three are harmless in shadow mode (nothing gates on them) but mean a clean shadow-mode window is not sufficient evidence to flip to live — the TODO comment in intelligence.py now says so explicitly. Whoever does the live-flip work should tighten roll_call to whole-transcript and special-case digit-hyphen + phonetic-word-as-street-name before trusting a 0-false-positive window.

Review found and fixed two real gaps in the shadow-mode PR (#128, commit `3ae0bb2`): 1. **Placement bug (would have made the shadow data itself misleading):** the classifier ran *after* the ≤5-word `transcript_too_short` skip, but 82% of its backtest flags were on ≤5-word transcripts — a live shadow window would have shown roughly a fifth of the real catch rate. Moved the classification ahead of that check, folded into the existing `doc_set` (no extra write). 2. **No aggregate tally:** `admin.py`'s `summary` only had per-call fields, same gap `corr_gate_veto` (#126) fixed for the consensus gate. Added `chatter_classifier_flagged`/`chatter_classifier_reason` to the tally, spanning **both** `linked` calls and `orphans` — the target population (non-events) lands in both places, so `linked` alone undercounts it. Confirmed still zero behavior change: nothing reads the verdict, extraction runs unconditionally either way. Sandboxed pytest 364 green (unchanged from the PR's own count — this was a placement/observability fix, not new logic). **Not fixed, flagged for the go-live decision instead of touched now:** the review found the classifier's shapes are riskier than the offline backtest could show — - `roll_call` matches "roll call" **anywhere** in the transcript, not just whole-transcript ("Delayed for roll call, then respond to a structure fire at 12 Main Street" → flagged). - digit-hyphen street addresses match the bare-acknowledgement unit-ID pattern (the dumps already contain "72-Holland" — Whisper does emit this shape for real addresses). - some phonetic-alphabet callsign words are also real street names ("Central to 6 Henry, show me out at Lincoln and Union" → flagged). All three are harmless in shadow mode (nothing gates on them) but mean a clean shadow-mode window is not sufficient evidence to flip to live — the TODO comment in `intelligence.py` now says so explicitly. Whoever does the live-flip work should tighten `roll_call` to whole-transcript and special-case digit-hyphen + phonetic-word-as-street-name before trusting a 0-false-positive window.

logan closed this issue

2026-09-13 12:43:19 -04:00

logan referenced this issue from a commit

2026-09-13 12:43:19 -04:00

Merge pull request 'intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127)' (#128) from feat/115-chatter-classifier-shadow-mode into main

logan referenced this issue

2026-09-13 17:21:12 -04:00

Consensus tiebreaker manufactures incidents — resolves ~all disagreements as "new", overriding the LLM's "link" #115

logan referenced this issue

2026-09-13 17:32:43 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CFO draft #140

logan referenced this issue