Stop thin calls fusing a work shift into one incident (server-26#22)
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 1m52s

The 2026-08-20 production dump had 4 of 6 sampled incidents as junk chains,
the worst being f5190670: 68 calls over 4h09m, 44 units, 12 tags, at least
13 genuinely distinct events. 58 of 133 linked calls took the fast/thin
path, which is the one path that attaches a call with no fit test at all.

Three defects combined to produce that, and all three are fixed here.

1. What counted as thin was wrong.

is_thin_call was "not units and not vehicles and not coords". A real
dispatch qualified as thin whenever no unit ID parsed and the geocode
failed - six of them did in that dump, including "All units head over to
the powerhouse, 55 Hyman Hills Road ... she's 87 years old", a brand new
job that attached to the four-hour chain and then overwrote its location
and its title. A call is now substantive if it carries tags, a location
string, a severity above routine, or is a reassignment; only genuinely
content-free housekeeping ("10-4", "Copy") stays thin. Those calls now go
through _call_fits_incident like everything else, which on a dispatch
backbone with no positive signal means they open their own incident or
orphan rather than merging.

The reassignment clause closes a self-defeating guard: upload.py blanks
units when dispatch pulls a unit onto a NEW job, specifically to stop
unit-overlap chaining - and blanking units made the call thin, routing it
to the only path with no fit check. The guard produced the merge it
existed to prevent.

2. The thin path was bounded on dispatch channels only.

Every other talkgroup fell through to "thin_pool = tg_recent": any
incident idle up to tg_fast_path_idle_minutes (90), no single-candidate
requirement, no fit test. The 30-second tier-1 / single-candidate tier-2
structure now applies to all channels. Non-dispatch gets its own window,
TG_THIN_IDLE_MINUTES=15, rather than sharing the dispatch value: a
tactical channel really is dedicated to one scene so it earns longer, but
15 sits inside the 20-minute tactical-default window already used in
_call_fits_incident, so the no-evidence path is never more permissive than
the fit-tested path on the same channel.

Recency gates now compare the magnitude of the idle, not the signed value.
The re-correlation sweep anchors "now" to the call's own started_at, so
idle goes negative routinely - incident 9d376ffe recorded
corr_incident_idle_min: -4.1 - and every "idle <= window" test in this
module reads True for a negative number. Those gates had silently stopped
bounding anything for exactly the calls the sweep re-examines.

3. Nothing capped an incident's total size.

Every fit test in the correlator is pairwise: does this call belong with
that incident. Each of f5190670's 68 links was individually arguable; the
mistake was the accumulated shape, which no pairwise rule can see. Two
hard caps now remove an incident from the candidate pool entirely, before
any path can choose it - including the LLM tier, which reads the same
ctx lists.

INCIDENT_MAX_DURATION_MINUTES=120. The one incident in that dump that was
genuinely a single event ran 63 minutes (06:15 wrong-way driver to 07:18
closeout), so the cap has to clear an hour with real headroom. The four
junk chains ran 3h41m, 3h43m, 4h05m and 4h09m, so it has to sit well under
three hours. 120 also equals correlation_window_hours: the location and
slow paths already refuse a candidate older than that, and the fast path
was the only one exempt, so this removes an inconsistency rather than
inventing a number.

INCIDENT_MAX_CALLS=40. A backstop for a burst that fills up inside the
duration cap, not the primary bound. The worst chain averaged ~16
calls/hour while absorbing an entire dispatch backbone, so 40 calls in
under two hours means one incident is eating most of the channel. Set
deliberately above any plausible single-incident call volume (a
multi-alarm fire on its own tactical channel) so this cap errs toward
keeping real incidents whole and lets the duration cap do the cutting.

Capping is not truncation: the incident keeps every call it has and still
auto-resolves on the normal idle sweep. It just stops being a candidate.

Every ambiguous call here was resolved toward a separate incident rather
than a merge. A wrongly-separate incident is visibly wrong and can be
merged later; a wrongly-merged one silently corrupts every unit, tag,
severity and map pin on the incident it joined, and poisons the AI
summary written from them. The cost is some acknowledgements orphaning
instead of riding along on an incident, which is a small, visible loss.

Deliberately NOT changed, since both push toward more merging while the
current failure mode is entirely over-merging (every incident in the dump
has exactly one "new" call; there is no over-splitting left to trade
against):
  - unit-overlap positive feedback on shared dispatch channels, which is
    now bounded by the caps rather than fixed at its root
  - the sweep retry budget expiring before the target incident exists

Tests: 31 new cases in tests/test_correlator_merge_caps.py, including a
replay of the f5190670 night - 13 unrelated jobs at their real offsets,
plus roster unit traffic and acknowledgements every two minutes. Without
the caps that traffic still builds a 125-call incident spanning 244
minutes; with the old thinness test on top, 153 calls over 247 minutes in
3 incidents. With this commit it is 13 incidents, largest 40 calls over 80
minutes. Each new case was checked to fail when the behaviour it covers is
reverted. Suite: 138 passed.

No AI feature flag was touched; correlation stays off in production.

Closes logan/server-26#22

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-20 03:34:48 -04:00
co-authored by Claude Opus 5
parent baa9d1811f
commit 33a247d306
3 changed files with 737 additions and 43 deletions
@@ -0,0 +1,470 @@
"""
Over-merge guards — server-26#22.
All of this comes from the 2026-08-20 production dump (CORRELATION_REVIEW_0820.md),
where 4 of 6 sampled incidents were junk chains and the worst, `f5190670`, was
68 calls over 4h09m carrying 44 units, 12 tags and at least 13 genuinely distinct
events. That is a work shift filed as one incident.
Three defects combined to produce it, and each has cases below:
1. `is_thin_call` was `not units and not vehicles and not coords`, so a real
dispatch with tags and a street address counted as thin whenever no unit ID
parsed and the geocode failed. Thin calls are the one class that links with
NO `_call_fits_incident` check, so those dispatches were force-merged.
2. The thin path was bounded only on dispatch channels. Everywhere else it
used the full 90-minute fast-path window, any number of candidates, no fit.
3. Nothing capped an incident's total size. Every fit test in the correlator
is pairwise, so each individual link can be defensible while the chain they
accumulate is not — no pairwise rule can see the shape.
`_run_decision` and `_is_thin_call` are pure, so none of this needs Firestore.
"""
import pytest
from datetime import datetime, timedelta, timezone
from app.config import settings
from app.internal.incident_correlator import (
_run_decision, _is_thin_call, _idle_gate_minutes,
_incident_at_capacity, _incident_span_minutes,
)
NOW = datetime(2026, 8, 20, 7, 0, 0, tzinfo=timezone.utc)
# TG 383 from the dump: "Ch 1 (Patched with 155.310)". _DISPATCH_TG_RE matches
# "patched", so this is a shared dispatch backbone carrying the whole department.
DISPATCH_TG = "Ch 1 (Patched with 155.310)"
TACTICAL_TG = "Fireground 2"
def _ctx(**overrides) -> dict:
base = {
"call_id": "call-1",
"all_active": [],
"recent": [],
"call_doc": {},
"call_embedding": None,
"call_units": [],
"call_vehicles": [],
"call_cleared": [],
"call_severity": "routine",
"coords": None,
"is_thin_call": True,
"now": NOW,
"system_id": "sys-1",
"talkgroup_id": 383,
"talkgroup_name": DISPATCH_TG,
"tags": [],
"incident_type": None,
"location": None,
"location_coords": None,
"reassignment": False,
"create_if_new": True,
}
base.update(overrides)
return base
def _incident(idle_minutes: float = 0.2, **overrides) -> dict:
updated = NOW - timedelta(minutes=idle_minutes)
inc = {
"incident_id": "inc-1",
"system_ids": ["sys-1"],
"talkgroup_ids": ["383"],
"updated_at": updated.isoformat(),
"started_at": updated.isoformat(),
"status": "active",
"call_ids": ["seed-call"],
}
inc.update(overrides)
return inc
# ---------------------------------------------------------------------------
# 1. What counts as thin — the misclassification that drove the chains
# ---------------------------------------------------------------------------
def test_content_free_acknowledgement_is_thin():
""""10-4." — no unit, no vehicle, no coords, no tags, no place, routine."""
assert _is_thin_call([], [], None, [], None, "routine", False) is True
@pytest.mark.parametrize("field,value", [
("tags", ["welfare-check"]),
("location", "55 Hyman Hills Road"),
("call_severity", "minor"),
("call_severity", "moderate"),
("call_severity", "major"),
])
def test_extracted_content_makes_a_call_substantive(field, value):
"""
The 07:08 call in `f5190670`: "All units head over to the powerhouse, 55
Hyman Hills Road … she's 87 years old" — a brand new job that was called
thin purely because no unit ID parsed and the geocode failed. It attached
with no fit check and then overwrote the four-hour chain's title and pin.
"""
kwargs = {"tags": [], "location": None, "call_severity": "routine"}
kwargs[field] = value
assert _is_thin_call([], [], None, kwargs["tags"], kwargs["location"],
kwargs["call_severity"], False) is False
def test_reassignment_is_never_thin():
"""
upload.py blanks `units` when dispatch pulls a unit onto a NEW job, to stop
unit-overlap chaining. That made the call thin and routed it to the only
path with no fit check — the guard produced the merge it existed to prevent.
"""
assert _is_thin_call([], [], None, [], None, "routine", True) is False
@pytest.mark.parametrize("units,vehicles,coords", [
(["6-Adam"], [], None),
([], ["black Toyota Camry"], None),
([], [], {"lat": 41.08, "lng": -73.81}),
])
def test_original_thinness_signals_still_apply(units, vehicles, coords):
assert _is_thin_call(units, vehicles, coords, [], None, "routine", False) is False
def test_blank_location_string_does_not_make_a_call_substantive():
assert _is_thin_call([], [], None, [], " ", "routine", False) is True
# ---------------------------------------------------------------------------
# 2. A thin call needs a tight window; a real one needs a fit signal
# ---------------------------------------------------------------------------
def test_thin_call_with_no_overlap_does_not_attach_on_a_dispatch_channel():
inc = _incident(idle_minutes=40)
assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan"
def test_thin_call_with_no_overlap_does_not_attach_on_a_tactical_channel():
"""
The widest version of the bug: non-dispatch talkgroups skipped the tiering
entirely and used the whole 90-minute fast-path window with no
single-candidate requirement, so ANY thin call joined whatever was newest.
"""
inc = _incident(idle_minutes=40)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
def test_tactical_thin_call_still_attaches_inside_its_own_window():
"""Bounded, not removed — a "10-4" on a working channel is still context."""
inc = _incident(idle_minutes=settings.tg_thin_idle_minutes - 1)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "fast/thin"
def test_tactical_thin_call_is_ambiguous_with_two_candidates():
a = _incident(idle_minutes=3.0, incident_id="inc-a")
b = _incident(idle_minutes=4.0, incident_id="inc-b")
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
def test_call_with_unit_overlap_does_attach():
"""
Positive control: real evidence still links. Carrying units also means the
call is not thin, so it reaches _call_fits_incident and passes on
unit_overlap rather than being force-attached.
"""
inc = _incident(idle_minutes=3.0, units=["6-Adam", "K-9A2"])
assert _is_thin_call(["6-Adam"], [], None, [], None, "routine", False) is False
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], call_units=["6-Adam"], is_thin_call=False,
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_fit_signal"] == "unit_overlap"
def test_substantive_call_with_no_signal_opens_its_own_incident():
"""
A tagged dispatch on a shared backbone with no unit/vehicle/geocode match
is a separate job, not a follow-up. Under the old thinness test this exact
call took fast/thin and merged.
"""
inc = _incident(idle_minutes=2.0)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
tags=["welfare-check"], location="55 Hyman Hills Road",
))
assert decision["action"] == "new"
assert decision["incident_type"] == "other"
# ---------------------------------------------------------------------------
# 3. Negative idle — the sweep anchors `now` to the call's own started_at
# ---------------------------------------------------------------------------
def test_idle_gate_uses_distance_not_sign():
"""
Observed on `9d376ffe`: corr_incident_idle_min -4.1, because the sweep
evaluated a 02:45 call against an incident updated at 02:50. Every
`idle <= window` gate reads True for a negative number, so the gates
stopped bounding anything for precisely the calls the sweep re-examines.
"""
future = _incident(idle_minutes=-45)
assert _idle_gate_minutes(future, NOW) == pytest.approx(45.0)
def test_back_dated_thin_call_does_not_sail_through_the_recency_gate():
future = _incident(idle_minutes=-45)
assert _run_decision(_ctx(all_active=[future], recent=[future]))["action"] == "orphan"
# ---------------------------------------------------------------------------
# 4. Hard caps — path-independent, because pairwise fit tests can't see shape
# ---------------------------------------------------------------------------
def _long_running(minutes: float) -> dict:
started = NOW - timedelta(minutes=minutes)
return _incident(idle_minutes=0.2, started_at=started.isoformat())
def test_duration_cap_forces_a_new_incident():
"""
`f5190670` ran 4h09m. The one real incident in the dump ran 63 minutes.
The unit here overlaps the incident's own roster, so without the cap this
links on unit_overlap — the cap is the only thing separating them.
"""
inc = _long_running(settings.incident_max_duration_minutes + 30)
inc["units"] = ["6-Adam"]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
call_units=["6-Adam"], tags=["welfare-check"],
))
assert decision["action"] == "new"
def test_duration_cap_blocks_the_thin_path_too():
"""The cap is checked before any path runs, so 'no fit test' is no escape."""
inc = _long_running(settings.incident_max_duration_minutes + 30)
assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan"
def test_incident_just_under_the_duration_cap_still_accepts_calls():
inc = _long_running(settings.incident_max_duration_minutes - 10)
inc["units"] = ["6-Adam"]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False, call_units=["6-Adam"],
))
assert decision["action"] == "link"
def test_call_count_cap_forces_a_new_incident():
inc = _incident(idle_minutes=0.2, units=["6-Adam"])
inc["call_ids"] = [f"c{i}" for i in range(settings.incident_max_calls)]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
call_units=["6-Adam"], tags=["vehicle-accident"],
))
assert decision["action"] == "new"
def test_incident_one_call_under_the_count_cap_still_accepts_calls():
inc = _incident(idle_minutes=0.2, units=["6-Adam"])
inc["call_ids"] = [f"c{i}" for i in range(settings.incident_max_calls - 1)]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False, call_units=["6-Adam"],
))
assert decision["action"] == "link"
@pytest.mark.parametrize("inc,expect", [
(_incident(), None),
(_long_running(9999), "duration"),
])
def test_capacity_reason_names_the_cap_that_fired(inc, expect):
reason = _incident_at_capacity(inc, NOW)
assert (reason is None) if expect is None else reason.startswith(expect)
def test_span_survives_a_back_dated_reference_time():
"""
The sweep passes the call's own started_at as `now`, which can precede the
incident's last update — the span must still reflect what the incident has
actually accumulated, not go negative and defeat the cap.
"""
started = NOW - timedelta(hours=5)
inc = {"started_at": started.isoformat(), "updated_at": NOW.isoformat()}
past = NOW - timedelta(hours=4)
assert _incident_span_minutes(inc, past) == pytest.approx(300.0, abs=0.1)
def test_unparseable_started_at_is_never_capped_on_duration():
assert _incident_span_minutes({"started_at": "not-a-date"}, NOW) == 0.0
# ---------------------------------------------------------------------------
# 5. Regression: the `f5190670` shape must not reassemble
# ---------------------------------------------------------------------------
# The 13 distinct events visible in `f5190670`, at their real offsets from the
# 03:01 opener. Each arrived exactly as reproduced here: tags and often a place
# name, but no parsed unit and no successful geocode — which is what made the
# old thinness test classify them as chatter.
_F5190670_EVENTS = [
(0, ["vehicle-accident", "telephone-pole-strike"], "Airport Road traffic circle"),
(2, ["uber-passenger", "phone-pinging"], "traffic circle near New King Street"),
(13, ["sign-down"], None),
(26, ["burglary-alarm"], "34 Carlton Drive"),
(67, ["inspection"], "2 Filno River Road"),
(108, ["disabled-vehicle"], "Yonkers Ave"),
(119, ["altercation"], "137 East Main Street"),
(131, ["inspection"], "80 North Grasslands Road"),
(156, ["foot-patrol"], "Tanzania Road"),
(211, [], None), # unit roll call — pure noise
(222, ["vehicle-off-roadway"], None),
(240, ["premises-check"], "Hillcrest Drive"),
(247, ["welfare-check"], "55 Hyman Hills Road"),
]
# The department roster heard on that channel. `f5190670` accumulated 44 units,
# partly from the nine phonetic-alphabet roll calls it absorbed, and once an
# incident holds most of the roster essentially every later call overlaps it —
# mechanism B in the review, unit-overlap positive feedback. Reproduced here so
# the chain has a real engine driving it, not just the thin path.
_ROSTER = ["6-Adam", "7-Baker", "11-Victor", "45-Charlie", "K-9A2", "22-47"]
_START = datetime(2026, 5, 24, 3, 1, 0, tzinfo=timezone.utc)
def _overnight_traffic():
"""
The real shape of that channel: a transmission roughly every two minutes for
4h07m — 13 dispatched jobs, routine unit traffic drawn from one roster, and
acknowledgements in between. Yields (offset_min, units, tags, location).
"""
events = {off: (tags, loc) for off, tags, loc in _F5190670_EVENTS}
# The dispatches, at their real offsets, exactly as they arrived: tags and
# usually a place name, but no parsed unit and no successful geocode.
traffic = [(off, [], tags, loc) for off, (tags, loc) in events.items()]
# Mechanism B, the engine that kept the real chain alive for four hours: one
# job that legitimately opens with units, then keeps producing unit traffic
# all shift. Each follow-up genuinely overlaps on unit, so each link is
# individually defensible and each one refreshes updated_at — which keeps
# the incident permanently inside every recency gate. No pairwise fit test
# can refuse these; only a cap can stop the accumulation.
traffic.append((1, [_ROSTER[0]], ["prisoner-transport"], "Medical Center"))
traffic += [(m, [_ROSTER[0]], [], None) for m in range(5, 249, 4)]
# Everything else on the channel — one transmission every two minutes.
for n, minute in enumerate(range(0, 249, 2)):
if minute in events:
continue
if n % 3 == 0:
traffic.append((minute, [_ROSTER[1 + n % (len(_ROSTER) - 1)]], [], None))
else:
traffic.append((minute, [], [], None)) # "10-4"
return traffic
def _simulate(traffic):
"""
Replay traffic through the real decision engine, applying the same incident
mutations the commit layer would: a link appends the call, merges units, and
(unless thin) refreshes updated_at; "new" opens a doc. Returns
(incidents, placement) where placement maps call_id → incident_id.
"""
incidents: list[dict] = []
placement: dict[str, str] = {}
for i, (offset, units, tags, location) in enumerate(sorted(traffic)):
now = _START + timedelta(minutes=offset)
call_id = f"call-{i}"
thin = _is_thin_call(units, [], None, tags, location, "routine", False)
active = [inc for inc in incidents if inc["status"] == "active"]
decision = _run_decision(_ctx(
call_id=call_id, now=now, all_active=active, recent=active,
tags=tags, location=location, call_units=units, is_thin_call=thin,
))
if decision["action"] == "link":
inc = decision["matched_incident"]
inc["call_ids"].append(call_id)
inc["units"] = list(dict.fromkeys(inc["units"] + units))
inc["_last_call_at"] = now
if decision["corr_debug"].get("corr_path") != "fast/thin":
inc["updated_at"] = now.isoformat()
placement[call_id] = inc["incident_id"]
elif decision["action"] == "new":
incidents.append({
"incident_id": f"inc-{i}", "system_ids": ["sys-1"],
"talkgroup_ids": ["383"], "status": "active",
"started_at": now.isoformat(), "updated_at": now.isoformat(),
"call_ids": [call_id], "units": list(units),
"_started": now, "_last_call_at": now, "_offset": offset,
})
placement[call_id] = incidents[-1]["incident_id"]
return incidents, placement
def _live_span_minutes(inc: dict) -> float:
"""Minutes from an incident's first call to the last one it actually took."""
return (inc["_last_call_at"] - inc["_started"]).total_seconds() / 60
def test_f5190670_does_not_become_one_incident():
"""
The headline regression: a full overnight shift on one patched dispatch
backbone must not end up as a single incident. The real one was 68 calls,
4h09m, 44 units, 12 tags and at least 13 distinct events.
"""
traffic = _overnight_traffic()
incidents, placement = _simulate(traffic)
assert len(incidents) >= 12, f"the shift merged into {len(incidents)} incident(s)"
# Without the caps this same traffic produces a 125-call incident spanning
# 244 minutes; with the old thinness test on top of that, 153 calls over
# 247 minutes in 3 incidents — the `f5190670` shape, reproduced.
biggest = max(len(inc["call_ids"]) for inc in incidents)
assert biggest <= settings.incident_max_calls, (
f"one incident holds {biggest} calls, past the "
f"{settings.incident_max_calls}-call cap"
)
longest = max(_live_span_minutes(inc) for inc in incidents)
assert longest <= settings.incident_max_duration_minutes, (
f"an incident took calls across {longest:.0f}min, past the "
f"{settings.incident_max_duration_minutes}min cap"
)
def test_each_dispatched_job_gets_its_own_incident():
"""
The 13 events are unrelated jobs — a pole strike, a burglar alarm, two
inspections, an altercation, a welfare check. On a dispatch backbone with no
unit or geocode tying them together, none of them may join another's
incident. Under the old thinness test every one of these was "thin" and
force-attached to whatever was most recent.
"""
traffic = _overnight_traffic()
incidents, placement = _simulate(traffic)
dispatch_offsets = {off for off, _, tags, _ in traffic if tags}
dispatch_incidents = {
placement[f"call-{i}"]
for i, (off, _, tags, _) in enumerate(sorted(traffic))
if tags and f"call-{i}" in placement
}
assert len(dispatch_incidents) == len(dispatch_offsets), (
f"{len(dispatch_offsets)} jobs landed in {len(dispatch_incidents)} incident(s)"
)
def test_a_long_run_of_pure_chatter_never_builds_an_incident():
"""
Acknowledgements alone carry no content, so they cannot open an incident and
— with nothing recent to reply to — must not accrete into one either.
"""
incidents, _ = _simulate([(m, [], [], None) for m in range(0, 240, 6)])
assert incidents == []