intelligence: shadow-mode upstream dispatch-vs-chatter classifier (server-26#127)
Three live measurement windows and two consensus-layer fixes (#125, #126) converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop iterating the correlator's consensus layer, the actual lever is upstream — a classifier in scene extraction that recognizes radio housekeeping (roll call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever becomes a scene for the correlator to judge. Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript) function recognizing two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps — not invented regexes. Deliberately conservative: anything that doesn't cleanly reduce to a known shape returns (False, None) and the existing pipeline runs unchanged. SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to the existing _is_garbage_transcript / transcript_too_short gates and writes chatter_classifier_verdict / chatter_classifier_reason onto the call doc, but does not skip extraction. admin.py's correlation-debug _call_summary surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the next live window can measure the real-world false-positive rate before anything is wired to actually skip extraction. TODO(server-26#127) marks the call site. Backtest against all three existing dumps (1002 calls): 154 flagged, 0 false positives (no flagged call carries tags, coords, non-routine severity, or matches any review-doc-named dangerous-to-drop transcript — the major extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station subject check, the property-retrieval call, all individually verified). tests/test_chatter_classifier.py: real transcripts from the dumps/review docs in both directions. Sandboxed pytest 332 -> 364, green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
This commit is contained in:
Logan Cusano
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
11c98daed0
commit
05ddec8284
@@ -0,0 +1,135 @@
|
||||
"""
|
||||
Upstream dispatch-vs-chatter classifier — SHADOW MODE (server-26#115 follow-up).
|
||||
|
||||
Three live measurement windows (CORRELATION_REVIEW_0907.md, _0907b.md, _0912.md)
|
||||
and two consensus-layer fixes (#125, #126) all converged on the same conclusion:
|
||||
the actual non-event-promotion problem lives upstream of correlation entirely.
|
||||
Radio housekeeping — unit check-ins, roll call, bare 10-4/10-8/98 acknowledgements
|
||||
— has no incident content for `intelligence.extract_scenes` to find, but nothing
|
||||
stops it from being sent to the scene-extraction LLM and coming out the other end
|
||||
as a thin "scene" for the correlator to then judge. See CORRELATION_REVIEW_0912.md
|
||||
("Reminder: the real fix is still unscoped") and issue #115.
|
||||
|
||||
This module is that classifier. It is a PURE function of the transcript text —
|
||||
no Firestore, no LLM call, no side effects — so it is cheap to run on every
|
||||
transcript and cheap to test against real dumps offline.
|
||||
|
||||
SHADOW MODE ONLY. As of this module's introduction, nothing skips scene
|
||||
extraction based on this verdict. `intelligence.extract_scenes` calls
|
||||
`classify_chatter` purely to record the verdict on the call doc
|
||||
(`chatter_classifier_verdict` / `chatter_classifier_reason`) so it becomes
|
||||
observable in the next `/admin` correlation-debug dump, exactly like
|
||||
`corr_gate_veto` (server-26#115 / PR #126). See the TODO at that call site for
|
||||
what has to be true before this flips live.
|
||||
|
||||
Precision over recall, deliberately. A false positive here — flagging a REAL
|
||||
event as chatter — would, once live, silently mean that event never gets a
|
||||
scene, never gets tags/location/severity, and never has a chance to become an
|
||||
incident. That is a much bigger, harder-to-notice failure than a false
|
||||
negative (a housekeeping call that still goes through the existing expensive
|
||||
pipeline and gets judged "not an incident" the same way it is today). When a
|
||||
transcript doesn't clearly match one of the shapes below, this returns
|
||||
(False, None) and the existing pipeline runs exactly as it does today.
|
||||
|
||||
Patterns are drawn from hand-labeled examples in CORRELATION_REVIEW_0907b.md
|
||||
and CORRELATION_REVIEW_0912.md, cross-referenced against the real transcripts
|
||||
in corr_dump_9-7_0437am.json / corr_dump_9-7_pm.json / corr_dump_9-12.json —
|
||||
not invented regexes. See the backtest script referenced in the PR for the
|
||||
per-dump catch rate and false-positive count.
|
||||
"""
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
# Police/law-enforcement phonetic alphabet words (APCO + NATO). Deliberately
|
||||
# duplicated from intelligence.py's `_PHONETIC_ALPHA_WORDS` rather than
|
||||
# imported — intelligence.py imports this module (to write the shadow-mode
|
||||
# verdict onto the call doc), so importing back would be circular. Keep the
|
||||
# two sets in sync if either changes; they're small and rarely touched.
|
||||
_PHONETIC_ALPHA_WORDS = frozenset({
|
||||
# APCO (law enforcement)
|
||||
"adam", "baker", "charles", "david", "edward", "frank", "george", "henry",
|
||||
"ida", "john", "king", "lincoln", "mary", "nora", "ocean", "paul", "queen",
|
||||
"robert", "sam", "tom", "union", "victor", "william", "x-ray", "young", "zebra",
|
||||
# NATO
|
||||
"alpha", "bravo", "charlie", "delta", "echo", "foxtrot", "golf", "hotel",
|
||||
"india", "juliet", "kilo", "lima", "mike", "november", "oscar", "papa",
|
||||
"quebec", "romeo", "sierra", "tango", "uniform", "whiskey", "yankee", "zulu",
|
||||
})
|
||||
|
||||
_TOKEN_RE = re.compile(r"[a-z0-9][a-z0-9\-]*")
|
||||
|
||||
# Bare radio-procedure words that carry zero incident content by themselves.
|
||||
# Deliberately small and literal — this is not a general stopword list, it's
|
||||
# the exact vocabulary observed in hand-labeled chatter transcripts. Words
|
||||
# that are ambiguous outside a pure-procedure context (e.g. "location",
|
||||
# "call", "phone", "number", "go") are left OUT on purpose: including them
|
||||
# risks reducing a real, substantive transcript down to nothing.
|
||||
_FILLER_WORDS = frozenset({
|
||||
"to", "this", "is", "the", "a", "and", "for", "you", "can", "i", "in",
|
||||
"on", "of", "that", "just", "from", "out", "ok", "okay", "at", "be",
|
||||
"show", "me", "mark", "marked", "charge", "standby", "stand", "by",
|
||||
"clear", "available", "affirm", "affirmative", "negative", "copy",
|
||||
"copies", "received", "roger",
|
||||
})
|
||||
|
||||
# Agency/procedural designators — who's being addressed, not what happened.
|
||||
_RADIO_DESIGNATORS = frozenset({
|
||||
"central", "dispatch", "headquarters", "hq", "post", "unit", "sergeant",
|
||||
"sgt", "metro", "mta", "division", "county",
|
||||
})
|
||||
|
||||
_ROLL_CALL_RE = re.compile(r"\broll\s*call\b")
|
||||
|
||||
|
||||
def _tokenize(transcript: str) -> list[str]:
|
||||
return _TOKEN_RE.findall(transcript.lower())
|
||||
|
||||
|
||||
def _is_filler_token(token: str) -> bool:
|
||||
# Any token starting with a digit is a unit ID, 10-code, badge/post
|
||||
# number, or call-number fragment ("10-4", "6-8", "72-holland",
|
||||
# "11-victor", "98", "114") — procedural, not incident content. This is
|
||||
# deliberately broad: a real event transcript that happens to include a
|
||||
# digit-led token (an address number, a case number) still has other,
|
||||
# non-digit descriptive words left over, so this alone never reduces a
|
||||
# real transcript to nothing. See the backtest for confirmation.
|
||||
if token[0].isdigit():
|
||||
return True
|
||||
return (
|
||||
token in _FILLER_WORDS
|
||||
or token in _RADIO_DESIGNATORS
|
||||
or token in _PHONETIC_ALPHA_WORDS
|
||||
)
|
||||
|
||||
|
||||
def classify_chatter(transcript: Optional[str]) -> tuple[bool, Optional[str]]:
|
||||
"""
|
||||
Pure classification of a transcript as non-event radio housekeeping.
|
||||
|
||||
Returns (is_chatter, reason):
|
||||
(True, "roll_call") — contains a roll-call announcement
|
||||
(True, "bare_acknowledgement") — every token is a callsign/10-code/
|
||||
procedural filler word; nothing else
|
||||
(False, None) — not confidently chatter; let the
|
||||
existing pipeline run as today
|
||||
|
||||
Takes only the transcript. Other call metadata (talkgroup, severity, tags)
|
||||
doesn't exist yet at the point this needs to run — this classifier is
|
||||
upstream of the scene-extraction call that produces those fields — so it
|
||||
deliberately doesn't take them as input.
|
||||
"""
|
||||
if not transcript or not transcript.strip():
|
||||
return False, None
|
||||
|
||||
lowered = transcript.lower()
|
||||
if _ROLL_CALL_RE.search(lowered):
|
||||
return True, "roll_call"
|
||||
|
||||
tokens = _tokenize(transcript)
|
||||
if not tokens:
|
||||
return False, None
|
||||
|
||||
if any(not _is_filler_token(t) for t in tokens):
|
||||
return False, None
|
||||
|
||||
return True, "bare_acknowledgement"
|
||||
@@ -16,6 +16,7 @@ from typing import Optional
|
||||
from app.internal.logger import logger
|
||||
from app.internal import firestore as fstore
|
||||
from app.internal import area_context
|
||||
from app.internal.chatter_classifier import classify_chatter
|
||||
# Location validity is defined once, by the module that owns the incident's
|
||||
# location/pin invariant. incident_correlator does not import this module, so
|
||||
# this is not a cycle.
|
||||
@@ -218,6 +219,26 @@ async def extract_scenes(
|
||||
pass
|
||||
return []
|
||||
|
||||
# server-26#127 — SHADOW MODE ONLY. Computes whether this transcript looks
|
||||
# like non-event radio housekeeping (roll call, bare 10-4/10-8/98
|
||||
# acknowledgements, unit check-ins) and records the verdict on the call
|
||||
# doc, but does NOT skip extraction — the scene-extraction call below runs
|
||||
# exactly as it does today regardless of what this says. This is step one
|
||||
# of getting live production data on the classifier's false-positive rate
|
||||
# before trusting it with anything real; see CORRELATION_REVIEW_0912.md.
|
||||
# TODO(server-26#127): flip this from shadow to live (skip extraction and
|
||||
# write skip_reason="non_event_chatter" instead of just recording the
|
||||
# verdict) once a live shadow-mode window confirms 0 false positives on
|
||||
# real production traffic.
|
||||
chatter_is_chatter, chatter_reason = classify_chatter(transcript)
|
||||
try:
|
||||
await fstore.doc_set("calls", call_id, {
|
||||
"chatter_classifier_verdict": chatter_is_chatter,
|
||||
"chatter_classifier_reason": chatter_reason,
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raw_scenes: list[dict] = await asyncio.to_thread(
|
||||
_sync_extract,
|
||||
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
|
||||
|
||||
@@ -141,6 +141,13 @@ async def debug_correlation(
|
||||
# written here specifically so a live measurement window can read
|
||||
# the reason instead of reconstructing it by hand from the dump.
|
||||
"corr_gate_veto": call.get("corr_gate_veto"),
|
||||
# server-26#127 — shadow-mode upstream chatter classifier verdict.
|
||||
# Written by intelligence.extract_scenes on every transcript that
|
||||
# reaches real scene extraction (not on garbage/too-short skips).
|
||||
# Nothing skips extraction on this yet — it's here purely so a
|
||||
# live measurement window can read the false-positive rate.
|
||||
"chatter_classifier_verdict": call.get("chatter_classifier_verdict"),
|
||||
"chatter_classifier_reason": call.get("chatter_classifier_reason"),
|
||||
}
|
||||
|
||||
# ── Determine which systems have AI active ────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user