a3681ea6986e8f66f4019934aaa231baacb598fd
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Logan CusanoandClaude Opus 5
|
c09cb72f66 |
Compare unit IDs by normalised key, not exact string
Dispatch audio names the same unit several ways within one conversation, and
every comparison in the correlator used exact string equality, so a follow-up
transmission from a unit already on an incident simply failed to find it. With
the creation gate no longer letting routine traffic open its own incident,
these stopped becoming junk incidents and started becoming orphans instead --
which is how they became visible. In the 01:05Z dump, five of eighteen orphans
were calls belonging to an incident that was open at that moment:
"K-9A2" vs "K-9-A-2" punctuation
"5-1-6" vs "516" digits read out individually
"37" vs "37th Post" ordinal plus role word
"11-Victor" vs "11 Victor" hyphen vs space
_normalize_unit lowercases, drops punctuation and role words (post/unit/car),
strips ordinal suffixes, and joins the remaining tokens, so each pair above
collapses to one key. All six comparison sites now go through it: the two
fast-path debug reporters, unit-continuity candidate selection and its
reassignment check, the cross-talkgroup 2+ shared-unit test, and the
disambiguation scorer.
What it deliberately does NOT do is match a bare district letter -- "Adam" is
not treated as "6-Adam". Every district has an Adam, and collapsing them would
merge unrelated incidents across districts. That leaves a couple of the
observed orphans unlinked, which is the right trade: a missed link leaves an
orphan the re-correlation sweep retries three times, while a false link
corrupts an incident permanently and nothing walks it back.
Two smaller things fall out of the shared helper. Matches are reported as the
original spoken strings rather than the normalised keys, so corr_matched_units
stays readable in the debug view. And a unit made only of role words ("Post")
would normalise to the empty string and then compare equal to every other such
unit, so it falls back to the raw text -- tested, because that failure would be
silent and would merge aggressively.
Adds 13 cases: each observed pair, five pairs that must stay distinct, the
empty-key guard, match reporting, and an end-to-end check that the K-9A2 call
now links where it previously orphaned. 38 pass.
No new environment variables, so CI deploys this without an ansible run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
7b5258cfdf |
Halve the tier-2 thin-call window, from 10 minutes to 5
With over-creation fixed, the incidents that remain are readable enough to judge, and the ones that still do not make sense all fail the same way. A content-free call attaches to the single active incident on its talkgroup if that incident has been idle under tg_dispatch_thin_idle_minutes, and at 10 minutes that is long enough for the channel to have moved on to something else. In the 00:30Z dump a "72 at Holland Station" incident absorbed a Grand Central train-crew meet 9.6 minutes later, and a status check absorbed a records lookup at 9.7. Being the only candidate is not evidence. It means the channel was quiet, which is exactly when guessing is weakest -- the single-candidate rule was meant to avoid picking wrongly among several, not to license a match no other signal supports. Every correct thin attach in that dump was <= 3.4 minutes idle and every wrong one was >= 8.2, so 5 separates them with room on both sides. Real back-and-forth is unaffected: it runs through the 30-second tier-1 path, and the observed conversational replies sit near zero. Tests pin both sides of the new boundary at 4.9 and 5.1 minutes so a later change to this number has to be deliberate. 23 pass. Also corrects a DEFERRED.md entry written earlier today. It claimed nothing ever closes an incident that goes quiet; summarizer.py has run a stale sweep at incident_auto_resolve_minutes (90) the whole time. The 37 open incidents were caused by over-creation, not by a missing sweeper, and 90 minutes may be fine now -- worth rechecking on a fully post-fix dump before changing it. No new environment variables: tg_dispatch_thin_idle_minutes is a config.py default and is not templated into any .env, so CI deploys this without an ansible run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Opus 5
|
96625fabd0 |
Stop ambient radio chatter from opening incidents, and refill the map
The 23:46Z correlation dump confirmed the severity gate fixed the problem it
was written for -- orphans fell from 69 to 16, and only three of those are
after the deploy boundary, two of them deliberate skips. Nothing on TG 9048
absorbs the channel any more; the largest post-deploy incident is four calls
over nine minutes and is genuinely one event.
It overcorrected. 37 of 50 incidents were open, most a single routine call.
The cause was the gate's own substance test, which counted `units` and
`location`. Radio protocol puts a unit ID in essentially every transmission
and a place name in most of them, so has_substance was true almost always and
the severity check never actually ran -- "11-Victor, 72 at Holland Station"
became its own permanent incident. Substance is now a vehicle, a geocode or a
tag: things the extractor found beyond who was speaking and where they stood.
Severity still opens an incident on its own, so nothing real is lost.
incident_type is now validated against the enum the prompt offers rather than
trusted. It is written straight through to incident.type and rendered as the
title, so a model that answered the severity question in the type field
produced an incident titled "Routine -- TGID 9563". Unrecognised values become
None and fall to the tag/severity path, which is what "unknown" already did.
The map was empty for a separate reason: geocoding accepted only ROOFTOP and
RANGE_INTERPOLATED. Dispatch names places the way people speak, and Google
returns GEOMETRIC_CENTER for exactly those forms -- intersections ("Lake
Street and Veterans Memorial Drive") and named POIs ("Brewster Station").
Requiring a street address discarded nearly every real dispatch location and
left only numbered addresses plotted, which is why the July incidents have
coordinates and none since do. GEOMETRIC_CENTER is now accepted; APPROXIMATE
is still rejected, since a region centroid is what an ungeocodable string
degrades to. Note this is necessary but may not be sufficient -- if
GOOGLE_MAPS_API_KEY is unset on the host the map stays empty regardless, and
that has not been checked from here.
Two things found and deliberately not fixed, both in DEFERRED.md. One call can
still land in two incidents, because upload.py correlates each extracted scene
independently and the model over-split one conversation; multi-scene is
intentional, so that is prompt tuning rather than a code change. And nothing
closes an incident that merely goes quiet -- signal-resolution and master
auto-resolve both exist, but a one-call incident nobody clears stays active
forever. That wanted the over-creation fixed first so a time-based sweeper
would not just paper over it.
Gate tests updated: units and location alone must now orphan, and the case
that matters most is kept explicit -- units with a real severity still open an
incident. 17 pass. No new environment variables, so CI deploys this without an
ansible run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan Cusano
|
6d5eb4c5f2 |
Let severity, not incident_type, decide what becomes an incident
The 2026-08-16 correlation dump showed two failures that looked unrelated and were the same bug. TG 9048 held one incident of 28 calls spanning 49 minutes -- a prisoner transport, a drone retrieval, a records lookup and a canvass, glued together -- while 32 other calls on that same channel stayed permanently orphaned. Creating an incident required a concrete incident_type. Nothing on a transit police channel produced one: the extraction prompt said to prefer "other" when uncertain, extraction then collapsed "other" to None, and the tag-based fallback had no tags to work with because administrative traffic carries none. So the channel could never open a SECOND incident. Every later call funnelled into whichever incident happened to exist first, and every call too substantial for the thin path had nowhere to go at all. The two symptoms were the same missing value seen from opposite ends. Severity now decides incident-worthiness. It is a better fit for the question being asked -- "is this a real event?" -- than a service label ever was, and unlike incident_type it is always present. The prompt defines four levels with no escape hatch (routine/minor/moderate/major, "unknown" is gone) and calls skipped for a too-short transcript are still recorded as routine, because downstream code reads a missing severity as "not processed yet" rather than "nothing happened". Anything above routine, or carrying any extracted content, opens an incident under the neutral "other" type. "other" is also kept as a real classification now -- rail operations and public works genuinely are not police, fire or EMS. Separately, thin calls no longer refresh updated_at; they write last_thin_at. updated_at drives every recency gate in the fast path, so each "10-4" was resetting the idle clock on whatever it attached to, keeping that incident inside the gate for as long as anyone kept acknowledging. An incident now ages from its last substantive call. This is what made the 49-minute incident possible even once buckets existed, so it is fixed independently rather than being left to the gate change. The re-correlation sweep also now honours skip_reason. /upload has always refused to correlate garbage and too-short transcripts, but the sweep did not apply the same filter, so those fragments came back minutes later through the thin path and attached to whatever was most recent -- a second, quieter route into the same over-merge. Adds tests/test_correlator_gate.py (15 cases), the first tests against incident_correlator.py in its 1,517-line history. tests/conftest.py stubs firebase-admin only when it is genuinely absent, so the container's real SDK is never shadowed; this is what makes the correlator importable in the dev venv. That stub also made test_mqtt_handler and test_node_sweeper collectable for the first time, revealing 10 pre-existing failures in them -- test-vs-code drift, untouched here and catalogued in DEFERRED.md. No new environment variables, so CI deploys this without an ansible run. |
||
|
Logan Cusano
|
97013e1505 |
Stop Whisper hallucinations and dedupe recordings across nodes
Two independent sources of garbage in the AI pipeline, both visible in the 2026-08-16 correlation dump. 1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt text as preceding transcript, so on noisy or silent audio it continued the series, emitting transcripts that count upward from 10-4 to 10-99. The existing no_speech_prob filter could not catch these: the model is highly confident in text it invented by continuing a pattern. The prompt no longer contains a series to extend, and _is_degenerate() rejects the three shapes this failure takes: ascending ten-code runs, one phrase looping, and near-identical segments across a whole recording. Verified against 13 transcripts from production: all four known hallucinations rejected, all nine real ones kept, including terse traffic containing legitimate codes. 2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one dump. Each was transcribed, billed and correlated twice, and the resulting incident listed two units where there was one. Canonical selection is by earliest started_at, tie-broken on call_id, NOT by upload order: upload order varies with encode time and network latency, so it would make the authoritative recording non-deterministic. Call documents are created from MQTT call_start before uploads arrive, so both nodes independently reach the same verdict. The loser keeps its audio (it may be the cleaner capture) but is excluded from STT, correlation, the re-correlation sweep and the orphan debug view. Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is missing, where the caller unpacks two values. A missing key surfaced as a misleading "Transcription failed" instead of the real warning. Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an injected callable so it stays importable without firebase-admin present. |
||
|
Logan
|
2f0597c81b |
Initial commit — DRB server stack
Includes c2-core (FastAPI/MQTT/Firestore), discord-bot (slash commands), frontend (Next.js admin UI), and mosquitto config. |