Three live measurement windows and two consensus-layer fixes (#125, #126) all converged on the same conclusion: the disagreement shape those PRs target (corr_llm_action==orphan, corr_rules_action==new) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside its scope, and the largest bucket (15/57, agreed) is structurally unreachable from the consensus layer at all — both correlator tiers independently called it a new incident, no disagreement exists to arbitrate.
Decision in both review docs: stop iterating the consensus layer. Move the fix upstream — recognize non-event radio housekeeping in scene extraction before it ever produces a scene.
What this ships
drb-c2-core/app/internal/chatter_classifier.py — classify_chatter(transcript) -> (bool, reason), a pure function with no I/O. Recognizes two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps:
roll_call — a roll-call announcement or line-up
bare_acknowledgement — every token in the transcript is a callsign/unit-ID (digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...)
Anything else — a location, an action, a description, an unrecognized word — leaves the call unclassified: (False, None). No fuzzy scoring, no thresholding; a transcript either cleanly reduces to one of these two shapes or it doesn't.
This is SHADOW MODE ONLY. Nothing skips scene extraction.intelligence.extract_scenes computes the verdict next to the existing _is_garbage_transcript / transcript_too_short gates and writes chatter_classifier_verdict (bool) + chatter_classifier_reason (str|None) onto the call doc, then runs extraction exactly as it does today regardless of the verdict. admin.py's correlation-debug _call_summary now surfaces both fields, same pattern as corr_gate_veto from #126, so a live window can read the real-world false-positive rate directly instead of it being invisible.
A TODO(server-26#127) at the call site in intelligence.py marks exactly what has to happen before this flips live.
Backtest (offline, against all three existing dumps)
False positive = a flagged call carrying tags, coords, or non-routine severity — zero across all 1002 calls. Individually re-verified against every transcript the review docs named as dangerous to drop: the major "extinguishing fire" call (severity=major), the geocoded 911-hangup call, both pursuit updates (severity=major), the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call, the MVA call — all confirmed (False, None).
Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset that also includes mid-conversation fragments of real events, so this undercounts precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low. Deliberately left uncaught rather than risk a broader net: garbled spelled-out roll call ("Five units. Six hundred. Eighty-four. Ten-four."), ambiguous check-ins with a stray content word ("7-4, call Central. 6-9, location."), CID/plate/case-number readbacks that mix in other words.
Tests
drb-c2-core/tests/test_chatter_classifier.py — real transcripts from the dumps/review docs, both directions (chatter → True, every named-dangerous real event → False). Sandboxed pytest: 332 → 364, green.
Next step — explicitly NOT done here
This PR ships measurement infrastructure, not a behavior change. Before any live flip:
Run at least one more live shadow-mode measurement window against real production traffic (this backtest only proves zero false positives against three already-collected dumps).
Confirm the false-positive rate is still zero on that live data.
Only then wire the verdict to actually skip extract_scenes and write skip_reason="non_event_chatter".
Closes #127. Related: #115, `CORRELATION_REVIEW_0907.md`, `CORRELATION_REVIEW_0907b.md`, `CORRELATION_REVIEW_0912.md`.
## Why
Three live measurement windows and two consensus-layer fixes (#125, #126) all converged on the same conclusion: the disagreement shape those PRs target (`corr_llm_action==orphan`, `corr_rules_action==new`) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside its scope, and the largest bucket (15/57, `agreed`) is structurally unreachable from the consensus layer at all — both correlator tiers independently called it a new incident, no disagreement exists to arbitrate.
**Decision in both review docs: stop iterating the consensus layer. Move the fix upstream — recognize non-event radio housekeeping in scene extraction before it ever produces a scene.**
## What this ships
`drb-c2-core/app/internal/chatter_classifier.py` — `classify_chatter(transcript) -> (bool, reason)`, a pure function with no I/O. Recognizes two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps:
- `roll_call` — a roll-call announcement or line-up
- `bare_acknowledgement` — every token in the transcript is a callsign/unit-ID (digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...)
Anything else — a location, an action, a description, an unrecognized word — leaves the call unclassified: `(False, None)`. No fuzzy scoring, no thresholding; a transcript either cleanly reduces to one of these two shapes or it doesn't.
**This is SHADOW MODE ONLY. Nothing skips scene extraction.** `intelligence.extract_scenes` computes the verdict next to the existing `_is_garbage_transcript` / `transcript_too_short` gates and writes `chatter_classifier_verdict` (bool) + `chatter_classifier_reason` (str|None) onto the call doc, then runs extraction exactly as it does today regardless of the verdict. `admin.py`'s correlation-debug `_call_summary` now surfaces both fields, same pattern as `corr_gate_veto` from #126, so a live window can read the real-world false-positive rate directly instead of it being invisible.
A `TODO(server-26#127)` at the call site in `intelligence.py` marks exactly what has to happen before this flips live.
## Backtest (offline, against all three existing dumps)
| dump | total calls | flagged chatter | false positives |
|---|---|---|---|
| `corr_dump_9-7_0437am.json` (window #1) | 397 | 55 | **0** |
| `corr_dump_9-7_pm.json` (window #2) | 333 | 72 | **0** |
| `corr_dump_9-12.json` (window #3) | 272 | 27 | **0** |
| **total** | **1002** | **154** | **0** |
False positive = a flagged call carrying tags, coords, or non-routine severity — **zero** across all 1002 calls. Individually re-verified against every transcript the review docs named as dangerous to drop: the major "extinguishing fire" call (severity=major), the geocoded 911-hangup call, both pursuit updates (severity=major), the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call, the MVA call — all confirmed `(False, None)`.
Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset that also includes mid-conversation fragments of real events, so this *undercounts* precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low. Deliberately left uncaught rather than risk a broader net: garbled spelled-out roll call ("Five units. Six hundred. Eighty-four. Ten-four."), ambiguous check-ins with a stray content word ("7-4, call Central. 6-9, location."), CID/plate/case-number readbacks that mix in other words.
## Tests
`drb-c2-core/tests/test_chatter_classifier.py` — real transcripts from the dumps/review docs, both directions (chatter → True, every named-dangerous real event → False). Sandboxed pytest: **332 → 364, green.**
## Next step — explicitly NOT done here
This PR ships measurement infrastructure, not a behavior change. Before any live flip:
1. Run at least one more live shadow-mode measurement window against real production traffic (this backtest only proves zero false positives against three already-collected dumps).
2. Confirm the false-positive rate is still zero on that live data.
3. Only then wire the verdict to actually skip `extract_scenes` and write `skip_reason="non_event_chatter"`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Three live measurement windows and two consensus-layer fixes (#125, #126)
converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop
iterating the correlator's consensus layer, the actual lever is upstream —
a classifier in scene extraction that recognizes radio housekeeping (roll
call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever
becomes a scene for the correlator to judge.
Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript)
function recognizing two shapes drawn from hand-labeled examples in the
review docs, cross-referenced against the real dumps — not invented regexes.
Deliberately conservative: anything that doesn't cleanly reduce to a known
shape returns (False, None) and the existing pipeline runs unchanged.
SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to
the existing _is_garbage_transcript / transcript_too_short gates and writes
chatter_classifier_verdict / chatter_classifier_reason onto the call doc,
but does not skip extraction. admin.py's correlation-debug _call_summary
surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the
next live window can measure the real-world false-positive rate before
anything is wired to actually skip extraction. TODO(server-26#127) marks
the call site.
Backtest against all three existing dumps (1002 calls): 154 flagged, 0
false positives (no flagged call carries tags, coords, non-routine severity,
or matches any review-doc-named dangerous-to-drop transcript — the major
extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station
subject check, the property-retrieval call, all individually verified).
tests/test_chatter_classifier.py: real transcripts from the dumps/review
docs in both directions. Sandboxed pytest 332 -> 364, green.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #127. Related: #115,
CORRELATION_REVIEW_0907.md,CORRELATION_REVIEW_0907b.md,CORRELATION_REVIEW_0912.md.Why
Three live measurement windows and two consensus-layer fixes (#125, #126) all converged on the same conclusion: the disagreement shape those PRs target (
corr_llm_action==orphan,corr_rules_action==new) is real but structurally small — backtesting a perfect version of that gate against window #3's own data caps out around 14/57 single-call "junk" incidents in a 4h window. 43/57 are outside its scope, and the largest bucket (15/57,agreed) is structurally unreachable from the consensus layer at all — both correlator tiers independently called it a new incident, no disagreement exists to arbitrate.Decision in both review docs: stop iterating the consensus layer. Move the fix upstream — recognize non-event radio housekeeping in scene extraction before it ever produces a scene.
What this ships
drb-c2-core/app/internal/chatter_classifier.py—classify_chatter(transcript) -> (bool, reason), a pure function with no I/O. Recognizes two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps:roll_call— a roll-call announcement or line-upbare_acknowledgement— every token in the transcript is a callsign/unit-ID (digit-led token) or a small fixed set of procedural filler words ("central", "affirmative", "10-4", "stand by", "show me", "clear", ...)Anything else — a location, an action, a description, an unrecognized word — leaves the call unclassified:
(False, None). No fuzzy scoring, no thresholding; a transcript either cleanly reduces to one of these two shapes or it doesn't.This is SHADOW MODE ONLY. Nothing skips scene extraction.
intelligence.extract_scenescomputes the verdict next to the existing_is_garbage_transcript/transcript_too_shortgates and writeschatter_classifier_verdict(bool) +chatter_classifier_reason(str|None) onto the call doc, then runs extraction exactly as it does today regardless of the verdict.admin.py's correlation-debug_call_summarynow surfaces both fields, same pattern ascorr_gate_vetofrom #126, so a live window can read the real-world false-positive rate directly instead of it being invisible.A
TODO(server-26#127)at the call site inintelligence.pymarks exactly what has to happen before this flips live.Backtest (offline, against all three existing dumps)
corr_dump_9-7_0437am.json(window #1)corr_dump_9-7_pm.json(window #2)corr_dump_9-12.json(window #3)False positive = a flagged call carrying tags, coords, or non-routine severity — zero across all 1002 calls. Individually re-verified against every transcript the review docs named as dangerous to drop: the major "extinguishing fire" call (severity=major), the geocoded 911-hangup call, both pursuit updates (severity=major), the "6 Alpha... Pelham Station" subject check, the property-retrieval call, the Maronex subject check, the milepost-13.7 trespass call, the MVA call — all confirmed
(False, None).Catch rate against a proxy chatter-universe (severity none/routine, no tags, no coords, no resolved incident_type — a superset that also includes mid-conversation fragments of real events, so this undercounts precision on the narrower true-chatter set): 55/343 (16%), 72/271 (27%), 27/201 (13%). Recall is intentionally low. Deliberately left uncaught rather than risk a broader net: garbled spelled-out roll call ("Five units. Six hundred. Eighty-four. Ten-four."), ambiguous check-ins with a stray content word ("7-4, call Central. 6-9, location."), CID/plate/case-number readbacks that mix in other words.
Tests
drb-c2-core/tests/test_chatter_classifier.py— real transcripts from the dumps/review docs, both directions (chatter → True, every named-dangerous real event → False). Sandboxed pytest: 332 → 364, green.Next step — explicitly NOT done here
This PR ships measurement infrastructure, not a behavior change. Before any live flip:
extract_scenesand writeskip_reason="non_event_chatter".🤖 Generated with Claude Code
https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix