intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127) #128

2026-09-13T00:00:33-04:00

logan commented

2026-09-13 00:00:33 -04:00

Closes #127. Related: #115, CORRELATION_REVIEW_0907.md, CORRELATION_REVIEW_0907b.md, CORRELATION_REVIEW_0912.md.

Why

Three live measurement windows and two consensus-layer fixes (#125, #126) all converged on the same conclusion: the disagreement shape those PRs target (corr_llm_action==orphan, corr_rules_action==new) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside its scope, and the largest bucket (15/57, agreed) is structurally unreachable from the consensus layer at all — both correlator tiers independently called it a new incident, no disagreement exists to arbitrate.

Decision in both review docs: stop iterating the consensus layer. Move the fix upstream — recognize non-event radio housekeeping in scene extraction before it ever produces a scene.

What this ships

drb-c2-core/app/internal/chatter_classifier.py — classify_chatter(transcript) -> (bool, reason), a pure function with no I/O. Recognizes two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps:

roll_call — a roll-call announcement or line-up
bare_acknowledgement — every token in the transcript is a callsign/unit-ID (digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...)

Anything else — a location, an action, a description, an unrecognized word — leaves the call unclassified: (False, None). No fuzzy scoring, no thresholding; a transcript either cleanly reduces to one of these two shapes or it doesn't.

This is SHADOW MODE ONLY. Nothing skips scene extraction. intelligence.extract_scenes computes the verdict next to the existing _is_garbage_transcript / transcript_too_short gates and writes chatter_classifier_verdict (bool) + chatter_classifier_reason (str|None) onto the call doc, then runs extraction exactly as it does today regardless of the verdict. admin.py's correlation-debug _call_summary now surfaces both fields, same pattern as corr_gate_veto from #126, so a live window can read the real-world false-positive rate directly instead of it being invisible.

A TODO(server-26#127) at the call site in intelligence.py marks exactly what has to happen before this flips live.

Backtest (offline, against all three existing dumps)

dump	total calls	flagged chatter
`corr_dump_9-7_0437am.json` (window #1)	397	55
`corr_dump_9-7_pm.json` (window #2)	333	72
`corr_dump_9-12.json` (window #3)	272	27
total	1002	154

False positive = a flagged call carrying tags, coords, or non-routine severity — zero across all 1002 calls. Individually re-verified against every transcript the review docs named as dangerous to drop: the major "extinguishing fire" call (severity=major), the geocoded 911-hangup call, both pursuit updates (severity=major), the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call, the MVA call — all confirmed (False, None).

Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset that also includes mid-conversation fragments of real events, so this undercounts precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low. Deliberately left uncaught rather than risk a broader net: garbled spelled-out roll call ("Five units. Six hundred. Eighty-four. Ten-four."), ambiguous check-ins with a stray content word ("7-4, call Central. 6-9, location."), CID/plate/case-number readbacks that mix in other words.

Tests

drb-c2-core/tests/test_chatter_classifier.py — real transcripts from the dumps/review docs, both directions (chatter → True, every named-dangerous real event → False). Sandboxed pytest: 332 → 364, green.

Next step — explicitly NOT done here

This PR ships measurement infrastructure, not a behavior change. Before any live flip:

Run at least one more live shadow-mode measurement window against real production traffic (this backtest only proves zero false positives against three already-collected dumps).
Confirm the false-positive rate is still zero on that live data.
Only then wire the verdict to actually skip extract_scenes and write skip_reason="non_event_chatter".

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

Closes #127. Related: #115, `CORRELATION_REVIEW_0907.md`, `CORRELATION_REVIEW_0907b.md`, `CORRELATION_REVIEW_0912.md`. ## Why Three live measurement windows and two consensus-layer fixes (#125, #126) all converged on the same conclusion: the disagreement shape those PRs target (`corr_llm_action==orphan`, `corr_rules_action==new`) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside its scope, and the largest bucket (15/57, `agreed`) is structurally unreachable from the consensus layer at all — both correlator tiers independently called it a new incident, no disagreement exists to arbitrate. **Decision in both review docs: stop iterating the consensus layer. Move the fix upstream — recognize non-event radio housekeeping in scene extraction before it ever produces a scene.** ## What this ships `drb-c2-core/app/internal/chatter_classifier.py` — `classify_chatter(transcript) -> (bool, reason)`, a pure function with no I/O. Recognizes two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps: - `roll_call` — a roll-call announcement or line-up - `bare_acknowledgement` — every token in the transcript is a callsign/unit-ID (digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...) Anything else — a location, an action, a description, an unrecognized word — leaves the call unclassified: `(False, None)`. No fuzzy scoring, no thresholding; a transcript either cleanly reduces to one of these two shapes or it doesn't. **This is SHADOW MODE ONLY. Nothing skips scene extraction.** `intelligence.extract_scenes` computes the verdict next to the existing `_is_garbage_transcript` / `transcript_too_short` gates and writes `chatter_classifier_verdict` (bool) + `chatter_classifier_reason` (str|None) onto the call doc, then runs extraction exactly as it does today regardless of the verdict. `admin.py`'s correlation-debug `_call_summary` now surfaces both fields, same pattern as `corr_gate_veto` from #126, so a live window can read the real-world false-positive rate directly instead of it being invisible. A `TODO(server-26#127)` at the call site in `intelligence.py` marks exactly what has to happen before this flips live. ## Backtest (offline, against all three existing dumps) | dump | total calls | flagged chatter | false positives | |---|---|---|---| | `corr_dump_9-7_0437am.json` (window #1) | 397 | 55 | **0** | | `corr_dump_9-7_pm.json` (window #2) | 333 | 72 | **0** | | `corr_dump_9-12.json` (window #3) | 272 | 27 | **0** | | **total** | **1002** | **154** | **0** | False positive = a flagged call carrying tags, coords, or non-routine severity — **zero** across all 1002 calls. Individually re-verified against every transcript the review docs named as dangerous to drop: the major "extinguishing fire" call (severity=major), the geocoded 911-hangup call, both pursuit updates (severity=major), the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call, the MVA call — all confirmed `(False, None)`. Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset that also includes mid-conversation fragments of real events, so this *undercounts* precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low. Deliberately left uncaught rather than risk a broader net: garbled spelled-out roll call ("Five units. Six hundred. Eighty-four. Ten-four."), ambiguous check-ins with a stray content word ("7-4, call Central. 6-9, location."), CID/plate/case-number readbacks that mix in other words. ## Tests `drb-c2-core/tests/test_chatter_classifier.py` — real transcripts from the dumps/review docs, both directions (chatter → True, every named-dangerous real event → False). Sandboxed pytest: **332 → 364, green.** ## Next step — explicitly NOT done here This PR ships measurement infrastructure, not a behavior change. Before any live flip: 1. Run at least one more live shadow-mode measurement window against real production traffic (this backtest only proves zero false positives against three already-collected dumps). 2. Confirm the false-positive rate is still zero on that live data. 3. Only then wire the verdict to actually skip `extract_scenes` and write `skip_reason="non_event_chatter"`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

logan added 1 commit 2026-09-13 00:00:34 -04:00

intelligence: shadow-mode upstream dispatch-vs-chatter classifier (server-26#127) 05ddec8284

Three live measurement windows and two consensus-layer fixes (#125, #126)
converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop
iterating the correlator's consensus layer, the actual lever is upstream —
a classifier in scene extraction that recognizes radio housekeeping (roll
call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever
becomes a scene for the correlator to judge.

Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript)
function recognizing two shapes drawn from hand-labeled examples in the
review docs, cross-referenced against the real dumps — not invented regexes.
Deliberately conservative: anything that doesn't cleanly reduce to a known
shape returns (False, None) and the existing pipeline runs unchanged.

SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to
the existing _is_garbage_transcript / transcript_too_short gates and writes
chatter_classifier_verdict / chatter_classifier_reason onto the call doc,
but does not skip extraction. admin.py's correlation-debug _call_summary
surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the
next live window can measure the real-world false-positive rate before
anything is wired to actually skip extraction. TODO(server-26#127) marks
the call site.

Backtest against all three existing dumps (1002 calls): 154 flagged, 0
false positives (no flagged call carries tags, coords, non-routine severity,
or matches any review-doc-named dangerous-to-drop transcript — the major
extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station
subject check, the property-retrieval call, all individually verified).

tests/test_chatter_classifier.py: real transcripts from the dumps/review
docs in both directions. Sandboxed pytest 332 -> 364, green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

logan added 1 commit 2026-09-13 12:42:48 -04:00

intelligence: run the chatter classifier before the too-short skip, not after (#127 ) 3ae0bb2d5b

82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

logan referenced this pull request

2026-09-13 12:43:09 -04:00

Upstream dispatch-vs-chatter classifier — shadow mode #127

logan merged commit b7701b6d49 into main

2026-09-13 12:43:18 -04:00

logan deleted branch feat/115-chatter-classifier-shadow-mode

2026-09-13 12:43:19 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#128