Let severity, not incident_type, decide what becomes an incident
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 2m29s

The 2026-08-16 correlation dump showed two failures that looked unrelated and
were the same bug. TG 9048 held one incident of 28 calls spanning 49 minutes --
a prisoner transport, a drone retrieval, a records lookup and a canvass, glued
together -- while 32 other calls on that same channel stayed permanently
orphaned.

Creating an incident required a concrete incident_type. Nothing on a transit
police channel produced one: the extraction prompt said to prefer "other" when
uncertain, extraction then collapsed "other" to None, and the tag-based fallback
had no tags to work with because administrative traffic carries none. So the
channel could never open a SECOND incident. Every later call funnelled into
whichever incident happened to exist first, and every call too substantial for
the thin path had nowhere to go at all. The two symptoms were the same missing
value seen from opposite ends.

Severity now decides incident-worthiness. It is a better fit for the question
being asked -- "is this a real event?" -- than a service label ever was, and
unlike incident_type it is always present. The prompt defines four levels with
no escape hatch (routine/minor/moderate/major, "unknown" is gone) and calls
skipped for a too-short transcript are still recorded as routine, because
downstream code reads a missing severity as "not processed yet" rather than
"nothing happened". Anything above routine, or carrying any extracted content,
opens an incident under the neutral "other" type. "other" is also kept as a real
classification now -- rail operations and public works genuinely are not police,
fire or EMS.

Separately, thin calls no longer refresh updated_at; they write last_thin_at.
updated_at drives every recency gate in the fast path, so each "10-4" was
resetting the idle clock on whatever it attached to, keeping that incident
inside the gate for as long as anyone kept acknowledging. An incident now ages
from its last substantive call. This is what made the 49-minute incident
possible even once buckets existed, so it is fixed independently rather than
being left to the gate change.

The re-correlation sweep also now honours skip_reason. /upload has always
refused to correlate garbage and too-short transcripts, but the sweep did not
apply the same filter, so those fragments came back minutes later through the
thin path and attached to whatever was most recent -- a second, quieter route
into the same over-merge.

Adds tests/test_correlator_gate.py (15 cases), the first tests against
incident_correlator.py in its 1,517-line history. tests/conftest.py stubs
firebase-admin only when it is genuinely absent, so the container's real SDK is
never shadowed; this is what makes the correlator importable in the dev venv.
That stub also made test_mqtt_handler and test_node_sweeper collectable for the
first time, revealing 10 pre-existing failures in them -- test-vs-code drift,
untouched here and catalogued in DEFERRED.md.

No new environment variables, so CI deploys this without an ansible run.
This commit is contained in:
Logan Cusano
2026-08-16 18:25:25 -04:00
parent 97013e1505
commit 6d5eb4c5f2
5 changed files with 278 additions and 9 deletions
+60 -1
View File
@@ -1,2 +1,61 @@
# All C2 core settings have defaults — no env setup needed.
# Add any shared fixtures here if required in the future.
#
# firebase-admin and google-cloud-firestore are runtime-only dependencies: they
# are installed in the container but not in the local dev venv, and
# app/internal/firestore.py calls _init_firebase() at import time. Without the
# stubs below, importing ANY module that reaches Firestore fails at collection
# time, which is why test_mqtt_handler and test_node_sweeper could not be run
# outside the container.
#
# The stubs are installed only when the real packages are absent, so the
# container's real SDK is never shadowed.
import sys
from types import ModuleType
from unittest.mock import MagicMock
try: # pragma: no cover - exercised only by which packages are installed
import firebase_admin # noqa: F401
except ModuleNotFoundError:
_firebase = ModuleType("firebase_admin")
# Falsy so _init_firebase() takes the initialize_app() branch rather than the
# already-initialised branch, which is itself broken (see DEFERRED.md).
_firebase._apps = {}
_firebase.initialize_app = MagicMock()
_firebase.credentials = MagicMock()
_firebase.firestore = MagicMock()
_credentials = ModuleType("firebase_admin.credentials")
_credentials.Certificate = MagicMock()
_credentials.ApplicationDefault = MagicMock()
_fs = ModuleType("firebase_admin.firestore")
_fs.client = MagicMock()
# A distinct sentinel rather than a MagicMock: production code writes this
# into dicts that tests compare against, and a MagicMock compares unequal
# to itself across attribute accesses.
_fs.SERVER_TIMESTAMP = "__SERVER_TIMESTAMP__"
_auth = ModuleType("firebase_admin.auth")
_auth.verify_id_token = MagicMock()
_auth.set_custom_user_claims = MagicMock()
_auth.get_user_by_email = MagicMock()
_auth.get_user = MagicMock()
_firebase.auth = _auth
_firebase.credentials = _credentials
_firebase.firestore = _fs
sys.modules["firebase_admin"] = _firebase
sys.modules["firebase_admin.credentials"] = _credentials
sys.modules["firebase_admin.firestore"] = _fs
sys.modules["firebase_admin.auth"] = _auth
try: # pragma: no cover
from google.cloud.firestore_v1.base_query import FieldFilter # noqa: F401
except ModuleNotFoundError:
for _name in (
"google", "google.cloud", "google.cloud.firestore_v1",
"google.cloud.firestore_v1.base_query",
):
sys.modules.setdefault(_name, ModuleType(_name))
sys.modules["google.cloud.firestore_v1.base_query"].FieldFilter = MagicMock()
+148
View File
@@ -0,0 +1,148 @@
"""
Unit tests for the incident-creation gate and the thin-call activity rule.
Both behaviours come from the 2026-08-16 correlation dump, where TG 9048
produced one 28-call / 49-minute incident alongside 32 permanent orphans:
* Requiring a concrete incident_type to create an incident meant a channel
whose traffic never classifies could never open a second incident, so every
later call funnelled into whichever incident existed first.
* Thin ("10-4") calls refreshed updated_at, which kept that incident
permanently inside the fast-path recency gate.
_run_decision is pure — it reads only the context dict — so these cases need no
Firestore. _update_incident writes, so its test patches fstore.
"""
import pytest
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
from app.internal.incident_correlator import _run_decision, _update_incident
NOW = datetime(2026, 8, 16, 21, 0, 0, tzinfo=timezone.utc)
def _ctx(**overrides) -> dict:
"""Context with no active incidents, so the decision reaches the creation gate."""
base = {
"call_id": "call-1",
"all_active": [],
"recent": [],
"call_doc": {},
"call_embedding": None,
"call_units": [],
"call_vehicles": [],
"call_cleared": [],
"call_severity": "routine",
"coords": None,
"is_thin_call": True,
"now": NOW,
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "MTA PD Districts 6/7/11 - Police Dispatch",
"tags": [],
"incident_type": None,
"location": None,
"location_coords": None,
"reassignment": False,
"create_if_new": True,
}
base.update(overrides)
return base
# ---------------------------------------------------------------------------
# Creation gate — severity decides incident-worthiness, not incident_type
# ---------------------------------------------------------------------------
def test_routine_status_traffic_stays_orphaned():
"""A content-free acknowledgement must not open an incident of its own."""
assert _run_decision(_ctx())["action"] == "orphan"
@pytest.mark.parametrize("severity", ["minor", "moderate", "major"])
def test_any_real_severity_opens_an_untyped_incident(severity):
decision = _run_decision(_ctx(call_severity=severity))
assert decision["action"] == "new"
assert decision["incident_type"] == "other"
@pytest.mark.parametrize("field,value", [
("call_units", ["6 Adam"]),
("call_vehicles", ["RMP 22146"]),
("coords", {"lat": 41.0, "lng": -73.8}),
("location", "District 6"),
("tags", ["prisoner-transport"]),
])
def test_concrete_content_opens_an_untyped_incident(field, value):
"""Routine severity is overridden by anything the extractor actually found."""
decision = _run_decision(_ctx(**{field: value}))
assert decision["action"] == "new"
assert decision["incident_type"] == "other"
def test_explicit_type_is_never_downgraded_to_other():
decision = _run_decision(_ctx(incident_type="police", call_severity="moderate"))
assert decision["action"] == "new"
assert decision["incident_type"] == "police"
def test_other_survives_extraction_and_creates_an_incident():
""""other" is a real classification now, not a synonym for unclassifiable."""
decision = _run_decision(_ctx(incident_type="other"))
assert decision["action"] == "new"
assert decision["incident_type"] == "other"
def test_sweep_never_creates_incidents():
"""The re-correlation sweep passes create_if_new=False — it may only link."""
decision = _run_decision(_ctx(call_severity="major", create_if_new=False))
assert decision["action"] == "orphan"
# ---------------------------------------------------------------------------
# Thin calls attach for context but do not count as incident activity
# ---------------------------------------------------------------------------
def _incident(idle_minutes: float) -> dict:
updated = NOW - timedelta(minutes=idle_minutes)
return {
"incident_id": "inc-1",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
"updated_at": updated.isoformat(),
"started_at": updated.isoformat(),
"status": "active",
}
def test_thin_call_links_to_the_active_incident_on_its_talkgroup():
inc = _incident(0.2)
decision = _run_decision(_ctx(all_active=[inc], recent=[inc]))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "fast/thin"
@pytest.mark.asyncio
async def test_thin_link_does_not_refresh_updated_at():
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
_incident(5), "call-1", 9048, "sys-1", [], None, None, [], [], None, NOW,
refresh_activity=False,
)
updates = mock_fstore.doc_set.await_args.args[2]
assert "updated_at" not in updates, "a '10-4' must not reset the incident idle clock"
assert updates["last_thin_at"] == NOW.isoformat()
assert updates["summary_stale"] is True, "the call still belongs in the summary"
@pytest.mark.asyncio
async def test_substantive_link_does_refresh_updated_at():
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
_incident(5), "call-1", 9048, "sys-1", [], None, None, ["6 Adam"], [], None, NOW,
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["updated_at"] == NOW.isoformat()
assert "last_thin_at" not in updates