Let severity, not incident_type, decide what becomes an incident
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 2m29s

The 2026-08-16 correlation dump showed two failures that looked unrelated and
were the same bug. TG 9048 held one incident of 28 calls spanning 49 minutes --
a prisoner transport, a drone retrieval, a records lookup and a canvass, glued
together -- while 32 other calls on that same channel stayed permanently
orphaned.

Creating an incident required a concrete incident_type. Nothing on a transit
police channel produced one: the extraction prompt said to prefer "other" when
uncertain, extraction then collapsed "other" to None, and the tag-based fallback
had no tags to work with because administrative traffic carries none. So the
channel could never open a SECOND incident. Every later call funnelled into
whichever incident happened to exist first, and every call too substantial for
the thin path had nowhere to go at all. The two symptoms were the same missing
value seen from opposite ends.

Severity now decides incident-worthiness. It is a better fit for the question
being asked -- "is this a real event?" -- than a service label ever was, and
unlike incident_type it is always present. The prompt defines four levels with
no escape hatch (routine/minor/moderate/major, "unknown" is gone) and calls
skipped for a too-short transcript are still recorded as routine, because
downstream code reads a missing severity as "not processed yet" rather than
"nothing happened". Anything above routine, or carrying any extracted content,
opens an incident under the neutral "other" type. "other" is also kept as a real
classification now -- rail operations and public works genuinely are not police,
fire or EMS.

Separately, thin calls no longer refresh updated_at; they write last_thin_at.
updated_at drives every recency gate in the fast path, so each "10-4" was
resetting the idle clock on whatever it attached to, keeping that incident
inside the gate for as long as anyone kept acknowledging. An incident now ages
from its last substantive call. This is what made the 49-minute incident
possible even once buckets existed, so it is fixed independently rather than
being left to the gate change.

The re-correlation sweep also now honours skip_reason. /upload has always
refused to correlate garbage and too-short transcripts, but the sweep did not
apply the same filter, so those fragments came back minutes later through the
thin path and attached to whatever was most recent -- a second, quieter route
into the same over-merge.

Adds tests/test_correlator_gate.py (15 cases), the first tests against
incident_correlator.py in its 1,517-line history. tests/conftest.py stubs
firebase-admin only when it is genuinely absent, so the container's real SDK is
never shadowed; this is what makes the correlator importable in the dev venv.
That stub also made test_mqtt_handler and test_node_sweeper collectable for the
first time, revealing 10 pre-existing failures in them -- test-vs-code drift,
untouched here and catalogued in DEFERRED.md.

No new environment variables, so CI deploys this without an ansible run.
This commit is contained in:
Logan Cusano
2026-08-16 18:25:25 -04:00
parent 97013e1505
commit 6d5eb4c5f2
5 changed files with 278 additions and 9 deletions
+60 -1
View File
@@ -1,2 +1,61 @@
# All C2 core settings have defaults — no env setup needed.
# Add any shared fixtures here if required in the future.
#
# firebase-admin and google-cloud-firestore are runtime-only dependencies: they
# are installed in the container but not in the local dev venv, and
# app/internal/firestore.py calls _init_firebase() at import time. Without the
# stubs below, importing ANY module that reaches Firestore fails at collection
# time, which is why test_mqtt_handler and test_node_sweeper could not be run
# outside the container.
#
# The stubs are installed only when the real packages are absent, so the
# container's real SDK is never shadowed.
import sys
from types import ModuleType
from unittest.mock import MagicMock
try: # pragma: no cover - exercised only by which packages are installed
import firebase_admin # noqa: F401
except ModuleNotFoundError:
_firebase = ModuleType("firebase_admin")
# Falsy so _init_firebase() takes the initialize_app() branch rather than the
# already-initialised branch, which is itself broken (see DEFERRED.md).
_firebase._apps = {}
_firebase.initialize_app = MagicMock()
_firebase.credentials = MagicMock()
_firebase.firestore = MagicMock()
_credentials = ModuleType("firebase_admin.credentials")
_credentials.Certificate = MagicMock()
_credentials.ApplicationDefault = MagicMock()
_fs = ModuleType("firebase_admin.firestore")
_fs.client = MagicMock()
# A distinct sentinel rather than a MagicMock: production code writes this
# into dicts that tests compare against, and a MagicMock compares unequal
# to itself across attribute accesses.
_fs.SERVER_TIMESTAMP = "__SERVER_TIMESTAMP__"
_auth = ModuleType("firebase_admin.auth")
_auth.verify_id_token = MagicMock()
_auth.set_custom_user_claims = MagicMock()
_auth.get_user_by_email = MagicMock()
_auth.get_user = MagicMock()
_firebase.auth = _auth
_firebase.credentials = _credentials
_firebase.firestore = _fs
sys.modules["firebase_admin"] = _firebase
sys.modules["firebase_admin.credentials"] = _credentials
sys.modules["firebase_admin.firestore"] = _fs
sys.modules["firebase_admin.auth"] = _auth
try: # pragma: no cover
from google.cloud.firestore_v1.base_query import FieldFilter # noqa: F401
except ModuleNotFoundError:
for _name in (
"google", "google.cloud", "google.cloud.firestore_v1",
"google.cloud.firestore_v1.base_query",
):
sys.modules.setdefault(_name, ModuleType(_name))
sys.modules["google.cloud.firestore_v1.base_query"].FieldFilter = MagicMock()