logan/server-26
1
0
Fork 0
Code Issues 57 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare unit IDs by normalised key, not exact string
Build & Deploy / Build & push images (push) Successful in 4m6s
Details
Build & Deploy / Deploy to VM (push) Successful in 2m32s
Details

Browse Source 
Dispatch audio names the same unit several ways within one conversation, and
every comparison in the correlator used exact string equality, so a follow-up
transmission from a unit already on an incident simply failed to find it. With
the creation gate no longer letting routine traffic open its own incident,
these stopped becoming junk incidents and started becoming orphans instead --
which is how they became visible. In the 01:05Z dump, five of eighteen orphans
were calls belonging to an incident that was open at that moment:

    "K-9A2"     vs "K-9-A-2"     punctuation
    "5-1-6"     vs "516"         digits read out individually
    "37"        vs "37th Post"   ordinal plus role word
    "11-Victor" vs "11 Victor"   hyphen vs space

_normalize_unit lowercases, drops punctuation and role words (post/unit/car),
strips ordinal suffixes, and joins the remaining tokens, so each pair above
collapses to one key. All six comparison sites now go through it: the two
fast-path debug reporters, unit-continuity candidate selection and its
reassignment check, the cross-talkgroup 2+ shared-unit test, and the
disambiguation scorer.

What it deliberately does NOT do is match a bare district letter -- "Adam" is
not treated as "6-Adam". Every district has an Adam, and collapsing them would
merge unrelated incidents across districts. That leaves a couple of the
observed orphans unlinked, which is the right trade: a missed link leaves an
orphan the re-correlation sweep retries three times, while a false link
corrupts an incident permanently and nothing walks it back.

Two smaller things fall out of the shared helper. Matches are reported as the
original spoken strings rather than the normalised keys, so corr_matched_units
stays readable in the debug view. And a unit made only of role words ("Post")
would normalise to the empty string and then compare equal to every other such
unit, so it falls back to the raw text -- tested, because that failure would be
silent and would merge aggressively.

Adds 13 cases: each observed pair, five pairs that must stay distinct, the
empty-key guard, match reporting, and an end-to-end check that the K-9A2 call
now links where it previously orphaned. 38 pass.

No new environment variables, so CI deploys this without an ansible run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-16 21:35:06 -04:00
co-authored by Claude Opus 5
parent 94ce9d48e2
commit c09cb72f66
2 changed files with 128 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
drb-c2-core/app/internal/incident_correlator.py
+68 -14
View File
@@ -135,6 +135,62 @@ _TAG_TYPE_HINTS: dict[str, str] = {
}
# Words that describe a unit's role rather than identifying it. Whisper hears
# the same officer as "Post 5", "5", and "5 post" within one conversation, so
# these carry no distinguishing information and are dropped before comparison.
_UNIT_NOISE_TOKENS = frozenset({"post", "unit", "units", "car"})
_ORDINAL_RE = re.compile(r"^(\d+)(?:st|nd|rd|th)$")
def _normalize_unit(unit: str) -> str:
"""
Reduce a spoken unit ID to a comparison key.
Dispatch audio names the same unit inconsistently and exact string equality
silently drops the follow-ups. Observed on 2026-08-17 in a single hour, each
pair being one unit that failed to match itself:
"K-9A2" vs "K-9-A-2" punctuation
"5-1-6" vs "516" digits read out individually
"37" vs "37th Post" ordinal + role word
"11-Victor" vs "11 Victor"
Case, punctuation and role words go; digit groups join up. What is
deliberately NOT done is matching a bare district letter — "Adam" is not
treated as "6-Adam", because every district has an Adam and collapsing them
would merge unrelated incidents. That costs a few links and is the right
trade: a missed link leaves an orphan the sweep can retry, a false link
corrupts an incident permanently.
"""
tokens = [t for t in re.split(r"[^a-z0-9]+", unit.strip().lower()) if t]
cleaned: list[str] = []
for tok in tokens:
if tok in _UNIT_NOISE_TOKENS:
continue
ordinal = _ORDINAL_RE.match(tok)
cleaned.append(ordinal.group(1) if ordinal else tok)
key = "".join(cleaned)
# A unit made only of noise words ("Post") would normalise to "" and then
# collide with every other such unit, so fall back to the raw text.
return key or unit.strip().lower()
def _unit_keys(units: Optional[list[str]]) -> set[str]:
"""Comparison keys for a unit list, empties dropped."""
return {k for k in (_normalize_unit(u) for u in (units or [])) if k}
def _matching_units(call_units: Optional[list[str]], inc_units: Optional[list[str]]) -> list[str]:
"""
Call-side units that also appear on the incident, compared by normalised key
but returned as the original spoken strings so debug output stays readable.
"""
inc_keys = _unit_keys(inc_units)
if not inc_keys:
return []
return [u for u in (call_units or []) if _normalize_unit(u) in inc_keys]
def _infer_type_from_tags(tags: list[str]) -> Optional[str]:
"""Return an incident type inferred from tags, or None if ambiguous."""
for tag in tags:
@@ -461,8 +517,7 @@ def _run_decision(ctx: dict) -> dict:
"corr_is_dispatch": is_dispatch,
}
if fit_signal == "unit_overlap" and call_units:
inc_unit_set = set(candidate.get("units") or [])
corr_debug["corr_matched_units"] = [u for u in call_units if u in inc_unit_set]
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
logger.info(
f"Correlator fast-path: call {call_id} → {candidate['incident_id']} "
f"(signal={fit_signal}, is_dispatch={is_dispatch})"
@@ -502,8 +557,7 @@ def _run_decision(ctx: dict) -> dict:
"corr_is_dispatch": is_dispatch,
}
if fit_signal == "unit_overlap" and call_units:
inc_unit_set = set(candidate.get("units") or [])
corr_debug["corr_matched_units"] = [u for u in call_units if u in inc_unit_set]
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
logger.info(
f"Correlator fast-path (disambig {len(tg_recent)} candidates): "
f"call {call_id} → {candidate['incident_id']} (signal={fit_signal})"
@@ -524,11 +578,11 @@ def _run_decision(ctx: dict) -> dict:
# incident, the officer has moved on and we don't link back to the old one.
# This correctly handles officers dispatched to a second call mid-shift.
if not matched_incident and call_units and system_id and not reassignment:
call_unit_set = set(call_units)
call_unit_set = _unit_keys(call_units)
unit_candidates = [
inc for inc in all_active
if system_id in (inc.get("system_ids") or [])
and call_unit_set & set(inc.get("units") or [])
and call_unit_set & _unit_keys(inc.get("units"))
]
# Apply idle cap: units get reassigned; a 20+ min gap means the officer
# has almost certainly moved on or the incident closed.
@@ -540,7 +594,7 @@ def _run_decision(ctx: dict) -> dict:
best_unit_inc = max(unit_candidates, key=lambda i: i.get("updated_at", ""))
reassigned_away = any(
inc["incident_id"] != best_unit_inc["incident_id"]
and call_unit_set & set(inc.get("units") or [])
and call_unit_set & _unit_keys(inc.get("units"))
and inc.get("updated_at", "") > best_unit_inc.get("updated_at", "")
for inc in all_active
)
@@ -615,7 +669,7 @@ def _run_decision(ctx: dict) -> dict:
# • embedding similarity >= cross-TG threshold (same subject matter)
# Requiring 2+ shared units prevents single-officer false positives.
if not matched_incident and call_embedding and incident_type and call_units and system_id:
call_unit_set = set(call_units)
call_unit_set = _unit_keys(call_units)
best_cross_score = 0.0
best_cross_inc: Optional[dict] = None
for inc in recent:
@@ -623,7 +677,7 @@ def _run_decision(ctx: dict) -> dict:
continue
if system_id not in (inc.get("system_ids") or []):
continue
inc_units_set = set(inc.get("units") or [])
inc_units_set = _unit_keys(inc.get("units"))
if len(call_unit_set & inc_units_set) < 2:
continue
inc_embedding = inc.get("embedding")
@@ -635,7 +689,7 @@ def _run_decision(ctx: dict) -> dict:
best_cross_inc = inc
if best_cross_inc and best_cross_score >= settings.embedding_cross_tg_threshold:
matched_incident = best_cross_inc
shared = len(call_unit_set & set(best_cross_inc.get("units") or []))
shared = len(call_unit_set & _unit_keys(best_cross_inc.get("units")))
corr_debug = {
"corr_path": "cross-tg",
"corr_score": round(best_cross_score, 4),
@@ -951,8 +1005,8 @@ def _disambiguate(
elif idle_min < 15: score += 1.0
# > 15 min: no bonus — older incidents compete on content/units only
inc_units = set(inc.get("units") or [])
if inc_units and call_units and any(u in inc_units for u in call_units):
inc_units = _unit_keys(inc.get("units"))
if inc_units and call_units and (inc_units & _unit_keys(call_units)):
score += 10.0
inc_vehicles = set(inc.get("vehicles") or [])
@@ -1047,8 +1101,8 @@ def _call_fits_incident(
inc_id = inc.get("incident_id", "?")
# ── 1. Unit overlap ───────────────────────────────────────────────────────
inc_units = set(inc.get("units") or [])
matched_units = [u for u in call_units if u in inc_units] if (inc_units and call_units) else []
inc_units = _unit_keys(inc.get("units"))
matched_units = _matching_units(call_units, inc.get("units"))
if matched_units:
if is_dispatch:
if call_coords: