Author SHA1 Message Date
logan a9197709f8 Merge pull request 'correlator: stop the re-correlation sweep racing an in-flight upload (#131)' (#154) from fix/131-sweep-race-duplicate-link into main
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 1m9s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-14 00:31:14 -04:00
Logan CusanoandClaude Sonnet 5 8dd636af8f correlator: stop the re-correlation sweep racing an in-flight upload (#131)
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.

Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.

15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.

recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.

Sandboxed pytest: 381 -> 387.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:30:24 -04:00
logan e27f8f6636 Merge pull request 'correlator/admin: capture per-scene incident_type + severity (#139)' (#152) from fix/139-scene-incident-type into main
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 1m40s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-14 00:14:53 -04:00
Logan CusanoandClaude Sonnet 5 f23026b9ab correlator: address drb-correlation-review notes on #139
- Document call_severity's routine-coercion asymmetry with incident_type
  (extraction-said-routine vs extraction-said-nothing look identical).
- Test docstring no longer overclaims the ctx-linkage it doesn't cover;
  points to the tests that do (test_consensus_gate.py, test_incident_identity.py).
- scene1 now uses a distinct severity so the test actually exercises both
  fields symmetrically; the 'no flat top-level clobber' claim is now
  asserted, not just commented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:14:32 -04:00
Logan CusanoandClaude Sonnet 5 7717fcccdd correlator/admin: capture per-scene incident_type + severity (#139)
_call_is_substanceless's "type" veto reads ctx["incident_type"] at decision
time, but that value was never persisted per-scene — only the last-scene-wins
flat field, which #138's window-4 dump analysis couldn't tell apart from
cross-scene contamination without re-guessing from a live dump. Adds
incident_type/severity to _apply_and_log's per-scene write and to admin.py's
_scene_summary allowlist (the debug-dump reader has its own field allowlist,
separate from the write side — silently would not have surfaced otherwise).

Sandboxed pytest: 380 -> 381.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:09:26 -04:00
logan 454fe7e81c Merge pull request 'correlator: remove is_dispatch and the tactical fit path entirely (#134)' (#135) from fix/134-drop-tactical-fit-path into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Failing after 1m56s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 14:55:04 -04:00
Logan CusanoandClaude Sonnet 5 422e9a4dc8 correlator: fix two comments left describing the removed tactical path (#134)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:54:54 -04:00
Logan CusanoandClaude Sonnet 5 b1884852d5 correlator: remove is_dispatch and the tactical fit path entirely (#134)
Full removal, not a hardcoded flag: _is_dispatch_channel, _DISPATCH_TG_RE, the is_dispatch parameter, and _call_fits_incident's tactical branch are gone. One evaluation path for every channel.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:49:55 -04:00
Logan CusanoandClaude Sonnet 5 0473e6a583 correlator: always run the dispatch-strict fit test, not name-guessed (#134)
is_dispatch was computed from _is_dispatch_channel(talkgroup_name) and picked between two _call_fits_incident evaluation orders: dispatch (requires a positive signal, runs location-conflict/content-divergence vetoes on unit overlap) vs tactical (skips both vetoes, defaults to True on no signal at all within 20 min). Per #133's reasoning, a name not literally containing dispatch/patched/primary got the unvetoed, default-True path solely because of its label. Hardcoded is_dispatch=True at its one real call site; the tactical branch and its own tests stay in place, unreached.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:38:57 -04:00
logan 4df801c5e0 Merge pull request 'correlator: drop the dispatch/tactical name-guess from the escape hatch (#115)' (#133) from fix/115-drop-dispatch-tactical-split into main
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 1m55s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 14:07:36 -04:00
Logan CusanoandClaude Sonnet 5 c50bfda8db correlator: drop the dispatch/tactical name-guess from the escape hatch, use one window (#115)
Owner correction from direct scanning experience: a talkgroup named tac/tactical only sees materially different traffic during a real incident, and that's rare -- the bulk of traffic on any monitored channel, including high-risk stops and pursuits, runs on the main channel regardless of what it's named. _is_dispatch_channel's string match on the talkgroup name is a naming-convention guess, not a detector of actual channel behavior; trusting it here meant a busy single-channel department not literally named 'dispatch' would silently get the more permissive 15-minute window and could reproduce #115's original bug (the gate never firing on the channels it targets).

Always use tg_dispatch_thin_idle_minutes (5 min) in the escape hatch, regardless of talkgroup name. Does NOT touch incident_correlator.py's own fast/thin idle-window selection, which uses the same dichotomy for a different, decision-changing purpose -- bigger blast radius, left for its own review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:07:00 -04:00
logan 833cfade4e Merge pull request 'correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix' (#132) from fix/96-114-per-scene-call-doc into main
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Failing after 1m28s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 13:34:19 -04:00
Logan CusanoandClaude Sonnet 5 0fe6d3b567 correlator: delete stale scenes on re-extraction instead of leaving them to rot (#96/#114)
Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident.

Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 13:33:52 -04:00
Logan CusanoandClaude Sonnet 5 fae84a45c3 correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix
Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).

Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
  - the existing flat corr_* fields, unchanged (last-scene-wins, the safe
    backward-compatible default for any reader that doesn't know about
    `scenes` yet)
  - a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
    corr_debug}, via doc_set(..., merge=True). Firestore's
    DocumentReference.set(data, merge=True) recursively merges nested map
    fields by key (documented SDK behaviour, not assumed) — a write to
    scenes.1 merges alongside an existing scenes.0 instead of replacing the
    whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.

admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.

summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.

Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.

Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.

Fixes server-26#96, server-26#114

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 13:25:37 -04:00
logan b7701b6d49 Merge pull request 'intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127)' (#128) from feat/115-chatter-classifier-shadow-mode into main
Build & Deploy / Build & push images (push) Successful in 4m2s
Build & Deploy / Deploy to VM (push) Failing after 1m48s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 12:43:18 -04:00
Logan CusanoandClaude Sonnet 5 3ae0bb2d5b intelligence: run the chatter classifier before the too-short skip, not after (#127)
82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 12:42:44 -04:00
logan 76db41adf7 Merge pull request 'ci: prune docker images before AND after deploy, not only on pull failure (#129)' (#130) from fix/129-deploy-disk-prune into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Failing after 1m12s
Build & Deploy / Report a failed deploy (push) Successful in 2s
2026-09-13 12:28:08 -04:00
Logan CusanoandClaude Sonnet 5 e97dab22ce ci: prune docker images before AND after deploy, not only on pull failure (#129)
The 2026-09-12 deploy of #126 failed instantly -- git pull on the VM hit 'No space left on device' before the deploy script could even capture a rollback target. Every deploy leaves 3 freshly SHA-tagged images that only got cleaned up by a prune gated on a failed compose pull; a failure earlier than that (like this one) never reached it. 96 of 100 local images were unreferenced, 23.76GB reclaimable, disk at 100%.

Move an unconditional docker image prune -af to the top of the deploy, before git pull, and upgrade the post-up -d prune from -f (dangling only) to -af (all unused) so stale tagged images stop re-accumulating. Both are safe: prune -a never touches an image a running container references.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 12:27:31 -04:00
Logan CusanoandClaude Sonnet 5 05ddec8284 intelligence: shadow-mode upstream dispatch-vs-chatter classifier (server-26#127)
Three live measurement windows and two consensus-layer fixes (#125, #126)
converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop
iterating the correlator's consensus layer, the actual lever is upstream —
a classifier in scene extraction that recognizes radio housekeeping (roll
call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever
becomes a scene for the correlator to judge.

Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript)
function recognizing two shapes drawn from hand-labeled examples in the
review docs, cross-referenced against the real dumps — not invented regexes.
Deliberately conservative: anything that doesn't cleanly reduce to a known
shape returns (False, None) and the existing pipeline runs unchanged.

SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to
the existing _is_garbage_transcript / transcript_too_short gates and writes
chatter_classifier_verdict / chatter_classifier_reason onto the call doc,
but does not skip extraction. admin.py's correlation-debug _call_summary
surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the
next live window can measure the real-world false-positive rate before
anything is wired to actually skip extraction. TODO(server-26#127) marks
the call site.

Backtest against all three existing dumps (1002 calls): 154 flagged, 0
false positives (no flagged call carries tags, coords, non-routine severity,
or matches any review-doc-named dangerous-to-drop transcript — the major
extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station
subject check, the property-retrieval call, all individually verified).

tests/test_chatter_classifier.py: real transcripts from the dumps/review
docs in both directions. Sandboxed pytest 332 -> 364, green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 23:59:53 -04:00
logan 11c98daed0 Merge pull request 'correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)' (#126) from fix/115-escape-hatch-window into main
Build & Deploy / Build & push images (push) Successful in 4m17s
Build & Deploy / Deploy to VM (push) Failing after 1m24s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-12 04:47:49 -04:00
Logan CusanoandClaude Sonnet 5 83beb2bf35 admin: surface corr_gate_veto on the correlation-debug endpoint (#115)
corr_gate_veto was written to corr_debug but the admin endpoint's whitelist (_call_summary + the summary tally) never surfaced it, so the last commit's whole point -- measuring window #4 instead of guessing -- would have produced nothing to read. Add it to both. Also softened the docstring's remaining overclaim: whether the active-only ctx[recent] limitation explains the 2/24 window-3 misses is unanswered, not confirmed -- read corr_gate_veto next window instead of asserting a guess again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:47:15 -04:00
Logan CusanoandClaude Sonnet 5 598054746a correlator: mirror the dispatch/tactical idle split in the escape hatch, record the gate-veto reason (#115)
Review of #126 found: (1) the escape hatch applied tg_dispatch_thin_idle_minutes (5 min) unconditionally, but incident_correlator's own fast/thin path only uses that on dispatch channels and 15 min on tactical ones via _is_dispatch_channel -- mirrored the same selection here, plus a config.py note flagging the second consumer. (2) the docstring claimed a 'confirmed explanation' for 2 window-3 gate misses that was actually wrong (self-contradictory in its own text); replaced the guess with corr_gate_veto, written into corr_debug on every llm=orphan/rules=new disagreement that escalates, so window #4 can see *why* each one escaped instead of reconstructing it from the raw dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:42:14 -04:00
Logan CusanoandClaude Sonnet 5 400b74b519 correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)
_recent_incident_on_same_talkgroup previously treated ANY same-talkgroup
incident within the 2-hour correlation_window_hours lookback as 'recent',
which disabled the whole LLM-orphan consensus gate on busy dispatch
channels: window #3 (CORRELATION_REVIEW_0912.md) measured 0/24 gate fires
against the exact target shape (rules=new, llm=orphan, tiebreak=new), with
22/24 explained by a same-talkgroup incident existing somewhere in the
prior 2h — nearly guaranteed on channels producing 3-13 incidents/2h.

Now the escape hatch only counts an incident as recent within
settings.tg_dispatch_thin_idle_minutes (5 min), reusing the same recency
bound the fast/thin path already uses for the 'dispatch, thin ack 10-30s
later' case this hatch exists for, instead of inventing a new constant.

Investigated the 2 unexplained misses (no same-tg incident found even by
a naive full-collection timestamp scan): confirmed ctx["recent"] is built
from status=="active" incidents with over-capacity incidents dropped
(_build_context / _drop_capped), not a full collection scan — an incident
that has auto-resolved or hit incident_max_calls/incident_max_duration
within the window is invisible to this check even though it is
chronologically recent. This does not explain the 2 misses (a same-tg
incident was absent by both checks there, so some other
_call_is_substanceless condition must be responsible), but it is a real
gap in the check as written. Documented in the docstring with a
TODO(server-26#115); fixing it needs a new, non-active-filtered Firestore
query, out of scope for this pass.

Tests: added a regression test proving an incident inside the old 2h
window but outside the new 5-minute window now correctly gates (fails on
main, passes here), plus a test proving a truly recent (<5min) same-tg
incident still escapes the gate as intended. Sandboxed pytest: 327 -> 329
passed (2 new tests), all green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:28:07 -04:00
logan 07ff9ba193 Merge pull request 'correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)' (#125) from fix/115-consensus-orphan-gate into main
Build & Deploy / Build & push images (push) Successful in 4m11s
Build & Deploy / Deploy to VM (push) Successful in 2m11s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-11 23:18:12 -04:00
Logan CusanoandClaude Sonnet 5 15a9d10666 correlator: keep the tiebreak for typed / reassignment calls in the orphan gate (#115)
_call_is_substanceless mirrored has_event_substance but not the creation gate's type-resolved short-circuit, so a routine-severity fire/medical call with no coords/tags/vehicles — or a reassignment (unit pulled to a new job) — could be gated to orphan where rules would open an incident. Bail out of the gate on incident_type or reassignment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:57:20 -04:00
Logan CusanoandClaude Sonnet 5 dd426572fc correlator: fix consensus orphan-gate to test call substance, not empty corr_debug (#115)
The gate added in ca1d8fb checked rules_decision["corr_debug"] for a positive
signal, but that dict is empty at preview time for action=="new" (corr_path is
written at apply time). The check was always False, so the gate fired on real
events — replayed against corr_dump_9-7_pm.json it dropped ~36 linked calls
including a major "extinguishing fire", a moderate fire-alarm, geocoded calls
and pursuit updates.

Gate now runs against ctx (fully populated at preview time). It fires ONLY when
the call is substanceless: routine severity, no vehicle/geocode/tag, and no
incident already running on the same talkgroup. Any of those escalates to the
tiebreak instead. The substance predicate (has_event_substance) is factored out
of incident_correlator's creation gate and shared, so the two cannot diverge.

recorrelation_sweep: a call the gate parked gets a longer link-only retry budget
(10 vs 3) — the gate fires before any incident for the job exists, so the
substantive call that justifies linking can land after the standard ~6 min.
Still create_if_new=False.

incident_correlator location path: evaluate every in-radius candidate and link
the nearest that carries corroboration, instead of the first in an unsorted
`recent`. A unit-overlap location link is now tagged "location_unit_overlap" so
it stops merging into the fast path's bucket in the admin fit-signal histogram.

tests/test_consensus_gate.py: replaced the corr_debug-signal cases with ctx
substance cases (severity, coords, tags, vehicles, same-tg incident); added a
nearest-wins location test; the two location guard tests now assert they reach
the new guard. Full drb-c2-core suite 322 -> 325.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:50:41 -04:00
Logan CusanoandClaude Sonnet 5 ca1d8fbdae correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)
CORRELATION_REVIEW_0907b.md measured that radio housekeeping (unit
check-ins, roll call, 10-8/10-98 clearings) is being promoted to
incidents. Every case reads corr_llm_action=orphan, corr_rules_action=new,
corr_consensus=tiebreak -> new: the cheap LLM correctly reads "not an
incident", the rules engine says `new` only because there is no incident to
link to, and the smart tiebreaker then sides with rules ~21/21. Reframing
the tiebreaker prompt (#116) did nothing. The fix is a consensus-logic gate,
not another prompt.

Fix 1 (routers/upload.py) - LLM-orphan gate in _correlate_with_consensus:
when the cheap LLM says `orphan` and the rules engine says `new` with NO
positive event signal, resolve to `orphan` and skip the tiebreak call
entirely. "No positive signal" = the rules corr_debug carries neither a
positive corr_path (unit-continuity / location / fast/disambig / fast/single)
nor a positive corr_fit_signal (unit_overlap / location_proximity). When it
does carry one, the existing escalation-to-tiebreak is kept so a genuine
event the LLM misreads as orphan still gets the second look. The resolved
outcome records corr_consensus="llm_orphan_gate" (greppable, distinct from
"tiebreak") and keeps corr_llm_reasoning / corr_rules_action /
corr_llm_action populated.

Fix 2 (incident_correlator.py) - tighten corr_path=location: the location
path linked on a bare sub-location_proximity_km (0.5 km) distance with no
unit or content check, which stitched a vehicle lockout to a station-restroom
slip and merged two different churches an hour apart. A location link now
requires unit overlap with the candidate OR a distance under a tighter bar
(_LOCATION_TIGHT_PROXIMITY_KM = 0.2 km). Pursuit incidents keep their
movement-speed-validated wide radius. A surviving location link now also
writes corr_fit_signal (unit_overlap | location_proximity), consistent with
Fix 1's positive-signal set.

Tests: new tests/test_consensus_gate.py (13 cases) - the gate resolves to
orphan without calling tiebreak on a no-signal disagreement; a unit_overlap /
location_proximity / unit-continuity / fast-disambig rules signal still
escalates; llm=link vs rules=new still escalates; the location path drops a
shared-area candidate with neither unit overlap nor tight proximity, links on
unit overlap, and links on tight proximity alone. Full c2-core suite
309 -> 322 passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:32:52 -04:00
logan 7f4d684966 Merge pull request 'ci: deploy Firestore rules + indexes on every push to main (#51)' (#124) from fix/51-ci-firestore-deploy into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 3m49s
Build & Deploy / Report a failed deploy (push) Skipped
Reviewed-on: #124
2026-09-07 23:22:38 -04:00
Logan CusanoandClaude Sonnet 5 bd04bdbd69 firestore: declare DESC indexes for the orderBy(desc) queries (#33/#51)
The old //direction note ('ASC serves orderBy desc') was wrong for these query shapes and left useCalls/useIncidents/useAlerts and search_calls throwing FAILED_PRECONDITION. Declare calls/incidents/alert_events (…, DESC) to match the live DB (indexes created via gcloud 2026-09-08). Drop the misleading 'delete these duplicates' drift note.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:20:48 -04:00
Logan CusanoandClaude Sonnet 5 775244bbde ci: deploy Firestore rules + indexes on every push to main (#51)
The deploy job SSHes to the VM (which runs as the project service account) but never touched Firestore, so rules and composite indexes regressed silently after every fix. Add a firebase-tools deploy right after `git pull`, additive for indexes, warn-not-fail on error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 19:44:57 -04:00
logan bc3251e8df Merge pull request 'frontend: #109 punch-list P2 — RulesTab effect, node recent-calls window' (#123) from fix/109-punchlist-p2 into main
Build & Deploy / Build & push images (push) Successful in 5m45s
Build & Deploy / Deploy to VM (push) Successful in 2m23s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 19:06:34 -04:00
logan 7a5bd5dbbb Merge pull request 'map: remove stray clock, unstack incident rail, OSM tile fallback (#118)' (#122) from fix/118-map-overlays into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m31s
2026-09-07 19:06:30 -04:00
logan 629bd1c340 Merge pull request 'firestore: declare the alert_events composite index (#51)' (#121) from fix/51-alert-events-index into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m28s
2026-09-07 19:06:28 -04:00
logan cea094d66b Merge pull request 'c2-core: fix CORS so the browser can call the REST API (#110)' (#120) from fix/110-c2-core-cors into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m31s
2026-09-07 19:06:25 -04:00
logan 01c146e21e Merge pull request 'ci: bake NEXT_PUBLIC_MAP_TILE_URL into the frontend build (#117)' (#119) from fix/117-map-tile-build-arg into main
Build & Deploy / Build & push images (push) Failing after 14s
Build & Deploy / Deploy to VM (push) Skipped
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-07 19:06:21 -04:00
Logan CusanoandClaude Sonnet 5 8a0412b529 firestore: add the alert_events composite index (#51)
collectionGroup alert_events (acknowledged, org_id, triggered_at desc) — the index the /watch Triggered Alerts tab and the site-wide useUnacknowledgedAlerts hook require. Still needs a manual deploy; no automation exists (#51).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:54:10 -04:00
Logan CusanoandClaude Sonnet 5 52edbf105c map: remove stray clock, unstack the incident card from the zoom controls, OSM tile fallback (#118)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:53:14 -04:00
Logan CusanoandClaude Sonnet 5 77f1d2f93f frontend: #109 punch-list P2 — effect-guard RulesTab, pending node card modal, org save, node recent-calls filter
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:52:39 -04:00
Logan CusanoandClaude Sonnet 5 d60fef67ad c2-core: add CORS middleware so the browser can call the REST API (#110)
The Archive page's GET /calls/search failed its CORS preflight (OPTIONS -> 405, no Access-Control-* headers). Allow the app origin(s) explicitly for the standard methods and the authorization/content-type headers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:52:38 -04:00
Logan CusanoandClaude Sonnet 5 fe643924c7 ci: bake NEXT_PUBLIC_MAP_TILE_URL into the frontend build (#117)
The map override var was added to MapView.tsx but never passed as a build-arg, so prod still shipped the dead Carto tile URL. Point it at OSM raster tiles.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:48:42 -04:00
logan bccb3e0316 correlator: give the LLM tier what it needs to link, stop it defaulting to "new" (#116)
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 2m25s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 16:59:46 -04:00
29 changed files with 2176 additions and 292 deletions
+44 -1
View File
@@ -63,6 +63,7 @@ jobs:
NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=${{ secrets.FIREBASE_MESSAGING_SENDER_ID }} NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=${{ secrets.FIREBASE_MESSAGING_SENDER_ID }}
NEXT_PUBLIC_FIREBASE_APP_ID=${{ secrets.FIREBASE_APP_ID }} NEXT_PUBLIC_FIREBASE_APP_ID=${{ secrets.FIREBASE_APP_ID }}
NEXT_PUBLIC_FIRESTORE_DATABASE=${{ secrets.FIRESTORE_DATABASE }} NEXT_PUBLIC_FIRESTORE_DATABASE=${{ secrets.FIRESTORE_DATABASE }}
NEXT_PUBLIC_MAP_TILE_URL=https://tile.openstreetmap.org/{z}/{x}/{y}.png
deploy: deploy:
name: Deploy to VM name: Deploy to VM
@@ -96,9 +97,46 @@ jobs:
set -e set -e
cd /opt/drb cd /opt/drb
# server-26#129: every deploy pushes 3 freshly SHA-tagged images and
# nothing ever removed the old ones except a prune that only ran
# AFTER a successful `compose pull` -- so a run that never got that
# far (this one) left the leak unaddressed forever. That silently
# filled the disk to 100% over ~week of deploys (2026-09-12: 29G/29G
# used, 96 of 100 local images unreferenced, 23.76GB reclaimable) and
# took `git pull` itself down with "No space left on device" before
# the deploy could even determine a rollback target. Prune BEFORE
# doing anything else, not after: `docker image prune -af` only
# removes images with no container referencing them, so it can never
# touch what's currently running -- there is nothing here for a
# mid-flight deploy to lose. Warn-not-fail: a prune failure must not
# block a deploy that doesn't actually need the space this time.
docker image prune -af || echo "WARNING: pre-deploy image prune failed (server-26#129) -- disk pressure may persist"
# Update compose files + mosquitto config # Update compose files + mosquitto config
git pull origin main git pull origin main
# server-26#51: Firestore rules + composite indexes had no deploy
# path and regressed silently after every fix (the alert_events and
# calls(org_id,started_at) indexes among them). The VM runs as the
# project service account, so firebase-tools authenticates via ADC
# with no key file, and infra/firestore/firebase.json pins database
# c2-server. Indexes go on additively -- no --force -- so a stray
# edit to firestore.indexes.json can never delete a live index;
# rules are a full replace, which is the intent. --non-interactive
# means the FIRST run after a drift still needs a one-time manual
# `firebase deploy` on the VM to clear pending deletions (it aborts
# rather than guess). A failure here warns but does NOT fail the
# deploy: a transient Firebase API error must not roll back a good
# app build.
if command -v firebase >/dev/null 2>&1; then
( cd /opt/drb/infra/firestore \
&& firebase deploy --only firestore:rules,firestore:indexes \
--project ${{ secrets.FIREBASE_PROJECT_ID }} --non-interactive ) \
|| echo "WARNING: firestore deploy failed (server-26#51) -- rules/indexes may be stale"
else
echo "WARNING: firebase CLI not on the VM -- skipped firestore deploy (server-26#51); install once with: npm i -g firebase-tools"
fi
# server-26#65: capture what is actually live BEFORE switching, so # server-26#65: capture what is actually live BEFORE switching, so
# a bad deploy has something concrete to fall back to. This reads # a bad deploy has something concrete to fall back to. This reads
# from a state file rather than re-deriving it from git log, # from a state file rather than re-deriving it from git log,
@@ -136,7 +174,12 @@ jobs:
$COMPOSE pull $COMPOSE pull
fi fi
$COMPOSE up -d --remove-orphans $COMPOSE up -d --remove-orphans
docker image prune -f # server-26#129: -f alone only removes dangling (untagged) images --
# the SHA-tagged image from every PAST deploy is not dangling, just
# unreferenced once `up -d` swaps the running container to the new
# tag, so it survived this indefinitely. -a catches those too; see
# the pre-pull prune above for why this can't touch anything live.
docker image prune -af
ENDSSH ENDSSH
) )
echo "$OUTPUT" echo "$OUTPUT"
+1
View File
@@ -47,3 +47,4 @@ Thumbs.db
# Out of scope - not a deployed service (server-26#56) # Out of scope - not a deployed service (server-26#56)
drb-telegram-bot/ drb-telegram-bot/
.claude/worktrees/
+7
View File
@@ -33,6 +33,13 @@ SUMMARY_INTERVAL_MINUTES=15
CORRELATION_WINDOW_HOURS=4 CORRELATION_WINDOW_HOURS=4
EMBEDDING_SIMILARITY_THRESHOLD=0.82 EMBEDDING_SIMILARITY_THRESHOLD=0.82
# Browser origins allowed to call this API cross-origin (JSON list). The only
# browser caller is the frontend's Archive page (GET /calls/search). Set this
# to the exact origin the frontend is served from — scheme + host, no path.
# Defaults to https://drb.cusano.net. A "*" entry works for local dev but is
# logged as a probable misconfiguration and never gets a credentialed response.
CORS_ORIGINS=["https://drb.cusano.net"]
# Fleet-wide token edge nodes present as X-Enrollment-Token on first boot # Fleet-wide token edge nodes present as X-Enrollment-Token on first boot
# (POST /nodes/enroll). Shared across every node — NOT a per-node secret. # (POST /nodes/enroll). Shared across every node — NOT a per-node secret.
# Generate with: openssl rand -hex 32 # Generate with: openssl rand -hex 32
+23 -26
View File
@@ -90,24 +90,19 @@ class Settings(BaseSettings):
unit_continuity_max_idle_minutes: int = 20 # unit-continuity path: skip if incident idle > this unit_continuity_max_idle_minutes: int = 20 # unit-continuity path: skip if incident idle > this
recorrelation_scan_minutes: int = 60 # re-examine orphaned calls ended within this window recorrelation_scan_minutes: int = 60 # re-examine orphaned calls ended within this window
tg_fast_path_idle_minutes: int = 90 # fast path: max minutes since incident last updated tg_fast_path_idle_minutes: int = 90 # fast path: max minutes since incident last updated
# Dispatch channels only: tier-2 thin calls attach to a lone candidate idle < this. # Tier-2 thin calls attach to a lone candidate idle < this, on every
# Was 10, which is long enough for the channel to have moved on to something else: # channel (server-26#133/#134 removed the dispatch/tactical split — a
# on 2026-08-16 a "72 at Holland Station" incident absorbed a Grand Central train # channel's name doesn't change how much scrutiny it gets). Was 10, which
# meet 9.6 min later, and a status check absorbed a records lookup at 9.7 min. # is long enough for the channel to have moved on to something else: on
# Across that dump every correct thin attach was <= 3.4 min idle and every wrong # 2026-08-16 a "72 at Holland Station" incident absorbed a Grand Central
# one was >= 8.2, so 5 separates them with room on both sides. Genuine # train meet 9.6 min later, and a status check absorbed a records lookup
# back-and-forth is handled by the 30-second tier-1 path above this. # at 9.7 min. Every correct thin attach in that dump was <= 3.4 min idle
# and every wrong one was >= 8.2, so 5 separates them with room on both
# sides. Genuine back-and-forth is handled by the 30-second tier-1 path
# above this. Also the escape hatch in routers/upload.py's LLM-orphan gate
# (_recent_incident_on_same_talkgroup, server-26#115) — check both call
# sites before retuning this.
tg_dispatch_thin_idle_minutes: int = 5 tg_dispatch_thin_idle_minutes: int = 5
# Every other channel: tier-2 thin calls attach to a lone candidate idle < this.
# Non-dispatch talkgroups previously had NO tier-2 bound at all — they used the
# whole 90-minute tg_fast_path_idle_minutes window with no single-candidate
# requirement and no fit test, which is the widest version of the 2026-08-20
# over-merge. A tactical channel really is dedicated to one scene, so it earns
# a longer window than a dispatch backbone, but not an unbounded one: 15 sits
# inside the 20-minute tactical-default window in _call_fits_incident, so the
# no-evidence thin path is never more permissive than the fit-tested path on
# the same channel.
tg_thin_idle_minutes: int = 15
# ── Hard caps: an incident past either of these stops accepting calls ────── # ── Hard caps: an incident past either of these stops accepting calls ──────
# Enforced on every correlation path (see _incident_at_capacity). Pairwise fit # Enforced on every correlation path (see _incident_at_capacity). Pairwise fit
@@ -180,16 +175,18 @@ class Settings(BaseSettings):
# between genuinely separate transmissions on a busy dispatch channel. # between genuinely separate transmissions on a busy dispatch channel.
duplicate_window_seconds: int = 10 duplicate_window_seconds: int = 10
# CORS — set to your frontend origin(s) in production, e.g. ["https://app.example.com"] # Browser origins allowed to call this API cross-origin. The only browser
# Defaults to "*" for local development only. # caller is the frontend's Archive page (GET /calls/search) — every other
# page reads Firestore directly. The frontend is served on the BARE domain
# (see infra Caddyfile.j2 — only drb. and api. have DNS records), so the
# default is that origin, not app.<domain>. Override via CORS_ORIGINS (JSON
# list) if the frontend ever moves; keep infra/.../c2-core.env.j2 in sync.
# #
# Leaving this as "*" is not merely permissive: main.py turns OFF # A "*" entry here still works for local dev but is refused a credentialed
# allow_credentials when it sees a wildcard, because Starlette would # response: main.py never enables allow_credentials (auth is a Bearer
# otherwise reflect each caller's origin back WITH # header, not a cookie), and it logs a loud ERROR when it sees a wildcard
# Access-Control-Allow-Credentials. So a production deployment that # in a deployment so a forgotten override is visible.
# forgets to set this gets a loud ERROR at startup and loses credentialed cors_origins: list[str] = ["https://drb.cusano.net"]
# cross-origin requests, rather than silently accepting every origin.
cors_origins: list[str] = ["*"]
# Discord webhook URL that app/internal/ai_health.py posts to when an AI # Discord webhook URL that app/internal/ai_health.py posts to when an AI
# tier (transcription/correlation) transitions into or out of degraded # tier (transcription/correlation) transitions into or out of degraded
@@ -0,0 +1,135 @@
"""
Upstream dispatch-vs-chatter classifier — SHADOW MODE (server-26#115 follow-up).
Three live measurement windows (CORRELATION_REVIEW_0907.md, _0907b.md, _0912.md)
and two consensus-layer fixes (#125, #126) all converged on the same conclusion:
the actual non-event-promotion problem lives upstream of correlation entirely.
Radio housekeeping — unit check-ins, roll call, bare 10-4/10-8/98 acknowledgements
— has no incident content for `intelligence.extract_scenes` to find, but nothing
stops it from being sent to the scene-extraction LLM and coming out the other end
as a thin "scene" for the correlator to then judge. See CORRELATION_REVIEW_0912.md
("Reminder: the real fix is still unscoped") and issue #115.
This module is that classifier. It is a PURE function of the transcript text —
no Firestore, no LLM call, no side effects — so it is cheap to run on every
transcript and cheap to test against real dumps offline.
SHADOW MODE ONLY. As of this module's introduction, nothing skips scene
extraction based on this verdict. `intelligence.extract_scenes` calls
`classify_chatter` purely to record the verdict on the call doc
(`chatter_classifier_verdict` / `chatter_classifier_reason`) so it becomes
observable in the next `/admin` correlation-debug dump, exactly like
`corr_gate_veto` (server-26#115 / PR #126). See the TODO at that call site for
what has to be true before this flips live.
Precision over recall, deliberately. A false positive here — flagging a REAL
event as chatter — would, once live, silently mean that event never gets a
scene, never gets tags/location/severity, and never has a chance to become an
incident. That is a much bigger, harder-to-notice failure than a false
negative (a housekeeping call that still goes through the existing expensive
pipeline and gets judged "not an incident" the same way it is today). When a
transcript doesn't clearly match one of the shapes below, this returns
(False, None) and the existing pipeline runs exactly as it does today.
Patterns are drawn from hand-labeled examples in CORRELATION_REVIEW_0907b.md
and CORRELATION_REVIEW_0912.md, cross-referenced against the real transcripts
in corr_dump_9-7_0437am.json / corr_dump_9-7_pm.json / corr_dump_9-12.json —
not invented regexes. See the backtest script referenced in the PR for the
per-dump catch rate and false-positive count.
"""
import re
from typing import Optional
# Police/law-enforcement phonetic alphabet words (APCO + NATO). Deliberately
# duplicated from intelligence.py's `_PHONETIC_ALPHA_WORDS` rather than
# imported — intelligence.py imports this module (to write the shadow-mode
# verdict onto the call doc), so importing back would be circular. Keep the
# two sets in sync if either changes; they're small and rarely touched.
_PHONETIC_ALPHA_WORDS = frozenset({
# APCO (law enforcement)
"adam", "baker", "charles", "david", "edward", "frank", "george", "henry",
"ida", "john", "king", "lincoln", "mary", "nora", "ocean", "paul", "queen",
"robert", "sam", "tom", "union", "victor", "william", "x-ray", "young", "zebra",
# NATO
"alpha", "bravo", "charlie", "delta", "echo", "foxtrot", "golf", "hotel",
"india", "juliet", "kilo", "lima", "mike", "november", "oscar", "papa",
"quebec", "romeo", "sierra", "tango", "uniform", "whiskey", "yankee", "zulu",
})
_TOKEN_RE = re.compile(r"[a-z0-9][a-z0-9\-]*")
# Bare radio-procedure words that carry zero incident content by themselves.
# Deliberately small and literal — this is not a general stopword list, it's
# the exact vocabulary observed in hand-labeled chatter transcripts. Words
# that are ambiguous outside a pure-procedure context (e.g. "location",
# "call", "phone", "number", "go") are left OUT on purpose: including them
# risks reducing a real, substantive transcript down to nothing.
_FILLER_WORDS = frozenset({
"to", "this", "is", "the", "a", "and", "for", "you", "can", "i", "in",
"on", "of", "that", "just", "from", "out", "ok", "okay", "at", "be",
"show", "me", "mark", "marked", "charge", "standby", "stand", "by",
"clear", "available", "affirm", "affirmative", "negative", "copy",
"copies", "received", "roger",
})
# Agency/procedural designators — who's being addressed, not what happened.
_RADIO_DESIGNATORS = frozenset({
"central", "dispatch", "headquarters", "hq", "post", "unit", "sergeant",
"sgt", "metro", "mta", "division", "county",
})
_ROLL_CALL_RE = re.compile(r"\broll\s*call\b")
def _tokenize(transcript: str) -> list[str]:
return _TOKEN_RE.findall(transcript.lower())
def _is_filler_token(token: str) -> bool:
# Any token starting with a digit is a unit ID, 10-code, badge/post
# number, or call-number fragment ("10-4", "6-8", "72-holland",
# "11-victor", "98", "114") — procedural, not incident content. This is
# deliberately broad: a real event transcript that happens to include a
# digit-led token (an address number, a case number) still has other,
# non-digit descriptive words left over, so this alone never reduces a
# real transcript to nothing. See the backtest for confirmation.
if token[0].isdigit():
return True
return (
token in _FILLER_WORDS
or token in _RADIO_DESIGNATORS
or token in _PHONETIC_ALPHA_WORDS
)
def classify_chatter(transcript: Optional[str]) -> tuple[bool, Optional[str]]:
"""
Pure classification of a transcript as non-event radio housekeeping.
Returns (is_chatter, reason):
(True, "roll_call") — contains a roll-call announcement
(True, "bare_acknowledgement") — every token is a callsign/10-code/
procedural filler word; nothing else
(False, None) — not confidently chatter; let the
existing pipeline run as today
Takes only the transcript. Other call metadata (talkgroup, severity, tags)
doesn't exist yet at the point this needs to run — this classifier is
upstream of the scene-extraction call that produces those fields — so it
deliberately doesn't take them as input.
"""
if not transcript or not transcript.strip():
return False, None
lowered = transcript.lower()
if _ROLL_CALL_RE.search(lowered):
return True, "roll_call"
tokens = _tokenize(transcript)
if not tokens:
return False, None
if any(not _is_filler_token(t) for t in tokens):
return False, None
return True, "bare_acknowledgement"
+9
View File
@@ -7,6 +7,15 @@ from google.cloud.firestore_v1.base_query import FieldFilter
from app.config import settings from app.config import settings
from app.internal.logger import logger from app.internal.logger import logger
# Re-exported so callers never need their own `firebase_admin.firestore` import
# just to delete a field. server-26#96/#114 review: `doc_set(..., merge=True)`
# merges nested maps by key but can never REMOVE one — writing `{"scenes": {}}`
# to clear a map is a no-op, not a delete. Use `doc_update(coll, id, {"field":
# fstore.DELETE_FIELD})` (or doc_set + merge, DELETE_FIELD works under both)
# whenever a re-extraction/reprocess path needs a stale nested field gone
# rather than merged over.
DELETE_FIELD = fs.DELETE_FIELD
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# In-memory TTL cache for rarely-changing documents (systems, nodes config) # In-memory TTL cache for rarely-changing documents (systems, nodes config)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+212 -135
View File
@@ -91,12 +91,12 @@ def _max_severity(current: Optional[str], new: Optional[str]) -> str:
_MAX_PURSUIT_SPEED_KM_PER_MIN = 8.0 # ~300 km/h, intentionally generous _MAX_PURSUIT_SPEED_KM_PER_MIN = 8.0 # ~300 km/h, intentionally generous
_PURSUIT_PROXIMITY_KM = 20.0 # expanded radius for moving incidents _PURSUIT_PROXIMITY_KM = 20.0 # expanded radius for moving incidents
_DISPATCH_TG_RE = re.compile( # server-26#115 — the location path linked on `location_proximity_km` (0.5 km)
r"\bdispatch\b|\bdisp\b" # alone, with no unit or content check. In a dense village two unrelated events
r"|\bpatched\b" # patched channels aggregate multiple call streams # routinely geocode that close (a vehicle lockout stitched to a station-restroom
r"|\bprimary\b", # "Primary" channels serve as shared backbones # slip; two different churches an hour apart). A location link now needs unit
re.IGNORECASE, # overlap with the candidate OR a distance under this tighter bar.
) _LOCATION_TIGHT_PROXIMITY_KM = 0.2
# Matches route/road identifiers in location strings for cross-system parent detection. # Matches route/road identifiers in location strings for cross-system parent detection.
# Groups: numbered routes (Route 202, NY-9, US-6, I-87, CR-35) and named parkways/highways. # Groups: numbered routes (Route 202, NY-9, US-6, I-87, CR-35) and named parkways/highways.
@@ -239,6 +239,22 @@ def _matching_units(call_units: Optional[list[str]], inc_units: Optional[list[st
return [u for u in (call_units or []) if _normalize_unit(u) in inc_keys] return [u for u in (call_units or []) if _normalize_unit(u) in inc_keys]
def has_event_substance(ctx: dict) -> bool:
"""
True when the call carries content beyond who-was-speaking-and-where:
a vehicle, a geocode, or a tag.
This is the substance half of the incident-creation gate (see
`_run_decision`, "Severity, not type, decides..."), factored out so the
consensus LLM-orphan gate in routers/upload.py mirrors it exactly and can
never drop a call the creation gate would have opened. `call_units` and
`location` are deliberately excluded — radio protocol puts a unit ID and a
place name in almost every transmission, so counting them as substance
makes the check trivially true.
"""
return bool(ctx.get("call_vehicles") or ctx.get("coords") or ctx.get("tags"))
def _infer_type_from_tags(tags: list[str]) -> Optional[str]: def _infer_type_from_tags(tags: list[str]) -> Optional[str]:
"""Return an incident type inferred from tags, or None if ambiguous.""" """Return an incident type inferred from tags, or None if ambiguous."""
for tag in tags: for tag in tags:
@@ -471,13 +487,6 @@ def _resolve_incident_title(
return {} return {}
def _is_dispatch_channel(talkgroup_name: Optional[str]) -> bool:
"""True when the talkgroup is a shared dispatch backbone (not a tactical/working channel)."""
if not talkgroup_name:
return False
return bool(_DISPATCH_TG_RE.search(talkgroup_name))
def _incident_idle_minutes(inc: dict, now: datetime) -> float: def _incident_idle_minutes(inc: dict, now: datetime) -> float:
"""Minutes since the incident was last updated (or started). Returns 9999 on parse error.""" """Minutes since the incident was last updated (or started). Returns 9999 on parse error."""
try: try:
@@ -686,6 +695,7 @@ async def correlate_call(
embedding: Optional[list] = None, embedding: Optional[list] = None,
severity: Optional[str] = None, severity: Optional[str] = None,
transcript: Optional[str] = None, transcript: Optional[str] = None,
scene_index: int = 0,
) -> Optional[str]: ) -> Optional[str]:
""" """
Link call_id to an existing incident or create a new one. Link call_id to an existing incident or create a new one.
@@ -695,6 +705,11 @@ async def correlate_call(
Callers that re-correlate a whole call rather than a scene — the Callers that re-correlate a whole call rather than a scene — the
recorrelation sweep — pass the call doc's stored values explicitly; they are recorrelation sweep — pass the call doc's stored values explicitly; they are
no longer read from the doc inside _build_context. no longer read from the doc inside _build_context.
``scene_index`` (server-26#96) identifies which scene of the call this
decision belongs to for the per-scene ``scenes`` map written by
_apply_and_log. Defaults to 0 — correct for every caller here, since this
entry point always re-correlates a call as a single unit, not a scene loop.
""" """
ctx = await _build_context( ctx = await _build_context(
call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units, call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units,
@@ -703,6 +718,7 @@ async def correlate_call(
tags=tags, incident_type=incident_type, location=location, tags=tags, incident_type=incident_type, location=location,
reassignment=reassignment, create_if_new=create_if_new, reassignment=reassignment, create_if_new=create_if_new,
embedding=embedding, severity=severity, transcript=transcript, embedding=embedding, severity=severity, transcript=transcript,
scene_index=scene_index,
) )
decision = _run_decision(ctx) decision = _run_decision(ctx)
return await _apply_and_log(decision, ctx) return await _apply_and_log(decision, ctx)
@@ -727,6 +743,7 @@ async def preview_correlation(
embedding: Optional[list] = None, embedding: Optional[list] = None,
severity: Optional[str] = None, severity: Optional[str] = None,
transcript: Optional[str] = None, transcript: Optional[str] = None,
scene_index: int = 0,
) -> dict: ) -> dict:
""" """
Run the rules engine and return the decision WITHOUT committing to Firestore. Run the rules engine and return the decision WITHOUT committing to Firestore.
@@ -740,6 +757,15 @@ async def preview_correlation(
matched_incident the candidate incident doc (action == "link") matched_incident the candidate incident doc (action == "link")
incident_type resolved type after tag inference (action == "new") incident_type resolved type after tag inference (action == "new")
corr_debug fields to persist on the call doc corr_debug fields to persist on the call doc
``scene_index`` (server-26#96) — which scene of the call (upload.py's
``for scene_index, scene in enumerate(scenes):`` loop) this call is. It
rides through ctx to _apply_and_log, which uses it as the key under the
call doc's ``scenes`` map so each scene's corr_debug/transcript lands in
its own map entry instead of colliding on the shared flat fields. Defaults
to 0 for callers with no scene concept (a single-scene call, or the
no-scenes-extracted correlation attempt) — equivalent to today's
behaviour for those calls.
""" """
ctx = await _build_context( ctx = await _build_context(
call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units, call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units,
@@ -748,6 +774,7 @@ async def preview_correlation(
tags=tags, incident_type=incident_type, location=location, tags=tags, incident_type=incident_type, location=location,
reassignment=reassignment, create_if_new=create_if_new, reassignment=reassignment, create_if_new=create_if_new,
embedding=embedding, severity=severity, transcript=transcript, embedding=embedding, severity=severity, transcript=transcript,
scene_index=scene_index,
) )
decision = _run_decision(ctx) decision = _run_decision(ctx)
return {"decision": decision, "ctx": ctx} return {"decision": decision, "ctx": ctx}
@@ -783,6 +810,7 @@ async def _build_context(
embedding: Optional[list] = None, embedding: Optional[list] = None,
severity: Optional[str] = None, severity: Optional[str] = None,
transcript: Optional[str] = None, transcript: Optional[str] = None,
scene_index: int = 0,
) -> dict: ) -> dict:
now = reference_time or datetime.now(timezone.utc) now = reference_time or datetime.now(timezone.utc)
window = timedelta(hours=settings.correlation_window_hours) window = timedelta(hours=settings.correlation_window_hours)
@@ -860,6 +888,9 @@ async def _build_context(
"incident_type": incident_type, "location": location, "incident_type": incident_type, "location": location,
"location_coords": location_coords, "reassignment": reassignment, "location_coords": location_coords, "reassignment": reassignment,
"create_if_new": create_if_new, "create_if_new": create_if_new,
# server-26#96 — which scene of the call this decision is for. Carried
# through so _apply_and_log can key the per-scene write correctly.
"scene_index": scene_index,
} }
@@ -928,14 +959,13 @@ def _run_decision(ctx: dict) -> dict:
if talkgroup_id is not None and system_id: if talkgroup_id is not None and system_id:
tg_str = str(talkgroup_id) tg_str = str(talkgroup_id)
# talkgroup_name may be None when the upload form omits it (node sets it # talkgroup_name may be None when the upload form omits it (node sets it
# directly on the Firestore call doc). Fall back to the call doc so that # directly on the Firestore call doc). Fall back to the call doc so the
# dispatch-channel strictness works regardless of how the call arrived. # log lines below still name the channel.
effective_talkgroup_name = talkgroup_name or call_doc.get("talkgroup_name") effective_talkgroup_name = talkgroup_name or call_doc.get("talkgroup_name")
is_dispatch = _is_dispatch_channel(effective_talkgroup_name)
if effective_talkgroup_name != talkgroup_name: if effective_talkgroup_name != talkgroup_name:
logger.info( logger.info(
f"Correlator: talkgroup_name missing from request for call {call_id}, " f"Correlator: talkgroup_name missing from request for call {call_id}, "
f"resolved from call doc: {effective_talkgroup_name!r} → is_dispatch={is_dispatch}" f"resolved from call doc: {effective_talkgroup_name!r}"
) )
tg_matches = [ tg_matches = [
@@ -979,10 +1009,7 @@ def _run_decision(ctx: dict) -> dict:
# single-candidate requirement and no fit test of any kind. Four # single-candidate requirement and no fit test of any kind. Four
# hours is not a bound, and neither is ninety minutes. # hours is not a bound, and neither is ninety minutes.
THIN_CONVERSATIONAL_SECS = 30 THIN_CONVERSATIONAL_SECS = 30
thin_window_min = ( thin_window_min = settings.tg_dispatch_thin_idle_minutes
settings.tg_dispatch_thin_idle_minutes if is_dispatch
else settings.tg_thin_idle_minutes
)
very_recent = [ very_recent = [
inc for inc in tg_recent inc for inc in tg_recent
if _idle_gate_minutes(inc, now) * 60 <= THIN_CONVERSATIONAL_SECS if _idle_gate_minutes(inc, now) * 60 <= THIN_CONVERSATIONAL_SECS
@@ -1004,8 +1031,7 @@ def _run_decision(ctx: dict) -> dict:
if len(thin_pool) > 1: if len(thin_pool) > 1:
logger.info( logger.info(
f"Correlator fast-path thin (tier-2): {len(thin_pool)} active incidents " f"Correlator fast-path thin (tier-2): {len(thin_pool)} active incidents "
f"on {'dispatch' if is_dispatch else 'tactical'} channel — " f"— ambiguous, skipping thin call {call_id}"
f"ambiguous, skipping thin call {call_id}"
) )
thin_pool = [] thin_pool = []
@@ -1035,14 +1061,14 @@ def _run_decision(ctx: dict) -> dict:
candidate = tg_recent[0] candidate = tg_recent[0]
logger.info( logger.info(
f"Correlator fast/single: call {call_id} vs incident {candidate['incident_id']} " f"Correlator fast/single: call {call_id} vs incident {candidate['incident_id']} "
f"tg_name={effective_talkgroup_name!r} is_dispatch={is_dispatch} " f"tg_name={effective_talkgroup_name!r} "
f"idle={round(_incident_idle_minutes(candidate, now), 1)}min " f"idle={round(_incident_idle_minutes(candidate, now), 1)}min "
f"call_units={call_units} inc_units={candidate.get('units')} " f"call_units={call_units} inc_units={candidate.get('units')} "
f"call_coords={'yes' if coords else 'no'} inc_coords={'yes' if candidate.get('location_coords') else 'no'}" f"call_coords={'yes' if coords else 'no'} inc_coords={'yes' if candidate.get('location_coords') else 'no'}"
) )
fit, fit_signal = _call_fits_incident( fit, fit_signal = _call_fits_incident(
candidate, call_units, call_vehicles, coords, candidate, call_units, call_vehicles, coords,
settings.location_proximity_km, is_dispatch=is_dispatch, settings.location_proximity_km,
call_embedding=call_embedding, now=now, call_embedding=call_embedding, now=now,
reassignment=reassignment, reassignment=reassignment,
) )
@@ -1052,13 +1078,12 @@ def _run_decision(ctx: dict) -> dict:
"corr_path": "fast/single", "corr_path": "fast/single",
"corr_incident_idle_min": round(_incident_idle_minutes(candidate, now), 1), "corr_incident_idle_min": round(_incident_idle_minutes(candidate, now), 1),
"corr_fit_signal": fit_signal, "corr_fit_signal": fit_signal,
"corr_is_dispatch": is_dispatch,
} }
if fit_signal == "unit_overlap" and call_units: if fit_signal == "unit_overlap" and call_units:
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units")) corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
logger.info( logger.info(
f"Correlator fast-path: call {call_id} → {candidate['incident_id']} " f"Correlator fast-path: call {call_id} → {candidate['incident_id']} "
f"(signal={fit_signal}, is_dispatch={is_dispatch})" f"(signal={fit_signal})"
) )
else: else:
logger.info( logger.info(
@@ -1074,14 +1099,14 @@ def _run_decision(ctx: dict) -> dict:
# dispatch channel should create its own incident, not be force-merged. # dispatch channel should create its own incident, not be force-merged.
logger.info( logger.info(
f"Correlator fast/disambig: call {call_id} vs incident {candidate['incident_id']} " f"Correlator fast/disambig: call {call_id} vs incident {candidate['incident_id']} "
f"tg_name={effective_talkgroup_name!r} is_dispatch={is_dispatch} " f"tg_name={effective_talkgroup_name!r} "
f"idle={round(_incident_idle_minutes(candidate, now), 1)}min " f"idle={round(_incident_idle_minutes(candidate, now), 1)}min "
f"call_units={call_units} inc_units={candidate.get('units')} " f"call_units={call_units} inc_units={candidate.get('units')} "
f"call_coords={'yes' if coords else 'no'} inc_coords={'yes' if candidate.get('location_coords') else 'no'}" f"call_coords={'yes' if coords else 'no'} inc_coords={'yes' if candidate.get('location_coords') else 'no'}"
) )
fit, fit_signal = _call_fits_incident( fit, fit_signal = _call_fits_incident(
candidate, call_units, call_vehicles, coords, candidate, call_units, call_vehicles, coords,
settings.location_proximity_km, is_dispatch=is_dispatch, settings.location_proximity_km,
call_embedding=call_embedding, now=now, call_embedding=call_embedding, now=now,
reassignment=reassignment, reassignment=reassignment,
) )
@@ -1092,7 +1117,6 @@ def _run_decision(ctx: dict) -> dict:
"corr_incident_idle_min": round(_incident_idle_minutes(candidate, now), 1), "corr_incident_idle_min": round(_incident_idle_minutes(candidate, now), 1),
"corr_candidates": len(tg_recent), "corr_candidates": len(tg_recent),
"corr_fit_signal": fit_signal, "corr_fit_signal": fit_signal,
"corr_is_dispatch": is_dispatch,
} }
if fit_signal == "unit_overlap" and call_units: if fit_signal == "unit_overlap" and call_units:
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units")) corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
@@ -1173,6 +1197,10 @@ def _run_decision(ctx: dict) -> dict:
# ── 2. Location path: proximity match (time-limited, cross-type) ───────── # ── 2. Location path: proximity match (time-limited, cross-type) ─────────
if not matched_incident and coords: if not matched_incident and coords:
# server-26#115 — score every in-radius candidate and link the NEAREST
# that carries corroboration, rather than whichever incident happened to
# come first in an unsorted `recent`.
loc_candidates: list[tuple] = []
for inc in recent: for inc in recent:
inc_coords = inc.get("location_coords") inc_coords = inc.get("location_coords")
if not inc_coords: if not inc_coords:
@@ -1189,18 +1217,49 @@ def _run_decision(ctx: dict) -> dict:
elapsed_min = max(_incident_idle_minutes(inc, now), 0.1) elapsed_min = max(_incident_idle_minutes(inc, now), 0.1)
if (dist_km / elapsed_min) > _MAX_PURSUIT_SPEED_KM_PER_MIN: if (dist_km / elapsed_min) > _MAX_PURSUIT_SPEED_KM_PER_MIN:
continue # implausible speed — skip this candidate continue # implausible speed — skip this candidate
if dist_km <= radius: if dist_km > radius:
matched_incident = inc continue
corr_debug = { # server-26#115 — a bare sub-radius distance is not enough on its
"corr_path": "location", # own. Require corroboration: unit overlap with the candidate, OR
"corr_distance_km": round(dist_km, 3), # a much tighter proximity. Pursuit incidents keep their
"corr_pursuit_mode": is_pursuit_inc, # movement-speed-validated wide radius (they passed the speed
} # check above), so they are exempt.
unit_overlap = bool(
_unit_keys(call_units) & _unit_keys(inc.get("units"))
)
tight_proximity = dist_km <= _LOCATION_TIGHT_PROXIMITY_KM
if not (is_pursuit_inc or unit_overlap or tight_proximity):
logger.info( logger.info(
f"Correlator location-path: call {call_id} → {inc['incident_id']} " f"Correlator location-path skipped: call {call_id} vs "
f"(dist={dist_km:.2f}km, pursuit={is_pursuit_inc})" f"{inc['incident_id']} — dist={dist_km:.2f}km within radius "
f"but no unit overlap and not tight-proximity "
f"(<= {_LOCATION_TIGHT_PROXIMITY_KM}km)"
) )
break continue
loc_candidates.append((dist_km, unit_overlap, is_pursuit_inc, inc))
if loc_candidates:
loc_candidates.sort(key=lambda c: c[0])
dist_km, unit_overlap, is_pursuit_inc, inc = loc_candidates[0]
matched_incident = inc
# Distinct from the fast path's "unit_overlap" so the admin
# corr_fit_signal histogram (routers/admin.py) does not merge a
# location-path link into the fast-path bucket (#35).
fit_signal = "location_unit_overlap" if unit_overlap else "location_proximity"
corr_debug = {
"corr_path": "location",
"corr_distance_km": round(dist_km, 3),
"corr_pursuit_mode": is_pursuit_inc,
"corr_fit_signal": fit_signal,
}
if unit_overlap and call_units:
corr_debug["corr_matched_units"] = _matching_units(
call_units, inc.get("units")
)
logger.info(
f"Correlator location-path: call {call_id} → {inc['incident_id']} "
f"(dist={dist_km:.2f}km, pursuit={is_pursuit_inc}, signal={fit_signal})"
)
# ── 2.5. Cross-TG path: same department, overlapping units, moderate similarity ── # ── 2.5. Cross-TG path: same department, overlapping units, moderate similarity ──
# #
@@ -1337,7 +1396,7 @@ def _run_decision(ctx: dict) -> dict:
# each. A vehicle, a geocode, or a tag means the extractor found something # each. A vehicle, a geocode, or a tag means the extractor found something
# beyond who was speaking and where they stood. # beyond who was speaking and where they stood.
if not resolved_type: if not resolved_type:
has_substance = bool(call_vehicles or coords or tags) has_substance = has_event_substance(ctx)
if call_severity in ("minor", "moderate", "major") or has_substance: if call_severity in ("minor", "moderate", "major") or has_substance:
resolved_type = "other" resolved_type = "other"
logger.info( logger.info(
@@ -1361,12 +1420,66 @@ def _run_decision(ctx: dict) -> dict:
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]: async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
"""Commit a rules decision and persist the corr_debug fields to the call doc.""" """
Commit a rules decision and persist the corr_debug fields to the call doc.
server-26#96: every scene of a multi-scene call reaches this function
independently (upload.py's ``for scene_index, scene in enumerate(scenes):``
loop → _correlate_with_consensus → preview_correlation/apply_correlation),
and every scene writes to the SAME call doc. Writing corr_debug only at
the flat top level meant scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc — the call doc ended up describing a splice
of decisions, not any one of them.
Fix: write the flat fields exactly as before (kept for any reader that
doesn't know about `scenes` yet — last-scene-wins, same as pre-#96
behaviour, a safe backward-compatible default) AND additionally nest the
same corr_debug — plus this scene's own transcript and the incident_id it
resolved to — under scenes.<scene_index>. Firestore's
`DocumentReference.set(data, merge=True)` recursively merges nested map
fields by key. Verified against `internal/firestore.py`'s `doc_set`
wrapper (a straight `ref.set(data, merge=merge)` pass-through — no
`update()`, no read-modify-write, nothing that would change this) and
against the documented set-with-merge semantics; NOT exercised against a
live Firestore instance (no SDK available in the sandboxes this landed
from — review flagged this distinction explicitly). A write of
{"scenes": {"1": {...}}} merges into an existing
{"scenes": {"0": {...}}} to produce {"scenes": {"0": {...}, "1": {...}}}
rather than replacing the whole `scenes` map, so scene 0's and scene 1's
entries land side by side instead of colliding like the flat fields do.
`scene_index` defaults to 0 (see preview_correlation/correlate_call), so a
plain single-scene call still gets a `scenes` map — just with one entry,
equivalent to reading the flat fields today.
"""
incident_id = await _apply_decision(decision, ctx) incident_id = await _apply_decision(decision, ctx)
corr_debug = decision.get("corr_debug") or {} corr_debug = decision.get("corr_debug") or {}
if corr_debug: if corr_debug:
scene_index = ctx.get("scene_index", 0)
updates = dict(corr_debug)
updates["scenes"] = {
str(scene_index): {
"transcript": ctx.get("scene_transcript"),
"incident_id": incident_id,
"corr_debug": corr_debug,
# server-26#139: this scene's OWN extracted incident_type/
# severity, as read by _call_is_substanceless's ctx at
# decision time — not the call doc's flat top-level field,
# which is last-scene-wins (server-26#96) and was the reason
# #138's "type" veto couldn't be told apart from cross-scene
# contamination without re-guessing from a live dump.
# NOTE: unlike incident_type, call_severity is already
# coerced to "routine" when extraction emitted nothing
# (ctx build: `severity or "routine"`) — a scene reading
# "routine" here doesn't distinguish "extraction said
# routine" from "extraction said nothing". Don't split a
# severity veto the way #138 splits the type veto without
# accounting for that.
"incident_type": ctx.get("incident_type"),
"severity": ctx.get("call_severity"),
}
}
try: try:
await fstore.doc_set("calls", ctx["call_id"], corr_debug) await fstore.doc_set("calls", ctx["call_id"], updates)
except Exception as e: except Exception as e:
logger.warning(f"Could not write corr_debug for call {ctx['call_id']}: {e}") logger.warning(f"Could not write corr_debug for call {ctx['call_id']}: {e}")
return incident_id return incident_id
@@ -1597,7 +1710,6 @@ def _call_fits_incident(
call_vehicles: list[str], call_vehicles: list[str],
call_coords: Optional[dict], call_coords: Optional[dict],
proximity_km: float, proximity_km: float,
is_dispatch: bool = False,
call_embedding: Optional[list] = None, call_embedding: Optional[list] = None,
now: Optional[datetime] = None, now: Optional[datetime] = None,
reassignment: bool = False, reassignment: bool = False,
@@ -1607,48 +1719,24 @@ def _call_fits_incident(
the incident; signal names the specific evidence that drove the decision. the incident; signal names the specific evidence that drove the decision.
fits=True signals: "unit_overlap" | "vehicle_overlap" | "location_proximity" fits=True signals: "unit_overlap" | "vehicle_overlap" | "location_proximity"
| "time_fallback" | "tactical_default"
fits=False signals: "unit_loc_conflict" | "content_divergence" fits=False signals: "unit_loc_conflict" | "content_divergence"
| "location_conflict" | "no_signal" | "tactical_idle" | "location_conflict" | "no_signal"
Original docstring (logic unchanged): Evaluation order:
Return True if this call plausibly belongs to the given incident. 1. Unit overlap. Same officer = same call. Also runs a location-conflict
guard: geocoded on both sides and clearly different → the officer has
Evaluation order for dispatch channels (is_dispatch=True): moved to a new scene, false positive. No geocode on the call (or on
───────────────────────────────────────────────────────── the incident, with a call geocode) AND the incident is old (≥ 15 min)
1. Unit overlap → content divergence as a location proxy (embedding similarity < 0.82
Same officer = same call. On dispatch channels, also run a location → different scene). Skipped for recent incidents (< 15 min) — an
conflict guard: if both sides carry geocoded coords and they differ update without re-stating the address is normal.
significantly, the officer has moved to a new scene and the unit match 2. Vehicle overlap → True.
is a false positive. 3. Location proximity. Both geocoded and close → True; far apart with no
When the call has NO geocoded coordinates AND the incident is old other positive signal → False.
(≥ 15 min), use content divergence as a location proxy: an officer at 4. No positive signal at all → False. A shared channel must not absorb
a genuinely different scene will be talking about clearly different calls by default (server-26#134 — this used to default True within
things. For recent incidents (< 15 min) we skip this proxy — the 20 min on any channel not name-matched as "dispatch"; a talkgroup
officer may simply be giving an update without mentioning the address. named tac/tactical is no less scrutinized in practice than any other).
2. Vehicle overlap → True
3. Location proximity
Both sides geocoded and close → True.
Both sides geocoded and far apart (no other positive signal) → False.
4. No positive signals fired → dispatch fallback
a. Conversational continuity: idle < 2 min → True.
A call arriving within 2 minutes of the last incident activity almost
certainly belongs to the same dispatch thread. "Baker, head over
there too" or "copy that" carries no incident-specific vocabulary but
is unambiguously a response to what was just said. We do not require
embedding similarity here — embeddings capture word meaning, not
conversational context, and short operational messages will always
have lower similarity than the incident's accumulated content.
b. Older incident, no positive signals → False.
A shared dispatch channel must not absorb calls by default.
Tactical / working channel (is_dispatch=False):
────────────────────────────────────────────────
Channel is dedicated to one scene. No evidence of separation ≈ same call.
Default → True.
Thin calls (no units/vehicles/coords) never reach this function — Thin calls (no units/vehicles/coords) never reach this function —
they are intercepted before it in correlate_call. they are intercepted before it in correlate_call.
@@ -1657,7 +1745,7 @@ def _call_fits_incident(
# signed value: the re-correlation sweep anchors `now` to the call's own # signed value: the re-correlation sweep anchors `now` to the call's own
# started_at, which can be earlier than the incident's last activity and # started_at, which can be earlier than the incident's last activity and
# send the signed value negative — silently defeating every `idle_min` # send the signed value negative — silently defeating every `idle_min`
# gate below (content-divergence veto and the tactical default alike). # gate below, the content-divergence veto included.
# See `_idle_gate_minutes` docstring. The signed value is reported to # See `_idle_gate_minutes` docstring. The signed value is reported to
# callers separately as `corr_incident_idle_min` (they compute it via # callers separately as `corr_incident_idle_min` (they compute it via
# `_incident_idle_minutes` themselves) — nothing here needs it. # `_incident_idle_minutes` themselves) — nothing here needs it.
@@ -1668,44 +1756,43 @@ def _call_fits_incident(
inc_units = _unit_keys(inc.get("units")) inc_units = _unit_keys(inc.get("units"))
matched_units = _matching_units(call_units, inc.get("units")) matched_units = _matching_units(call_units, inc.get("units"))
if matched_units: if matched_units:
if is_dispatch: if call_coords:
if call_coords: # Hard location conflict: geocoded on both sides and clearly different.
# Hard location conflict: geocoded on both sides and clearly different. inc_coords_u = inc.get("location_coords")
inc_coords_u = inc.get("location_coords") if inc_coords_u:
if inc_coords_u: dist_km = _haversine_km(
dist_km = _haversine_km( call_coords["lat"], call_coords["lng"],
call_coords["lat"], call_coords["lng"], inc_coords_u["lat"], inc_coords_u["lng"],
inc_coords_u["lat"], inc_coords_u["lng"], )
) if dist_km > proximity_km:
if dist_km > proximity_km: logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but location_conflict dist={dist_km:.2f}km → unit_loc_conflict")
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but location_conflict dist={dist_km:.2f}km → unit_loc_conflict") return False, "unit_loc_conflict"
return False, "unit_loc_conflict"
elif call_embedding and idle_min >= 15:
# Call has geocode but incident doesn't — fall back to content
# divergence as a location proxy. Without this, stale incidents
# that never geocoded absorb unrelated calls purely on unit
# overlap (e.g. a patrol officer working a second scene 70 min
# after the original call).
inc_emb_u = inc.get("embedding")
if inc_emb_u:
sim = _cosine_similarity(call_embedding, inc_emb_u)
if sim < 0.82:
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence (has_call_coords/no_inc_coords) sim={sim:.3f} → content_divergence")
return False, "content_divergence"
elif call_embedding and idle_min >= 15: elif call_embedding and idle_min >= 15:
# No geocode available AND old incident: use content divergence as a # Call has geocode but incident doesn't — fall back to content
# location-proxy veto. After 15+ minutes an officer at a completely # divergence as a location proxy. Without this, stale incidents
# different scene will be discussing clearly different content. # that never geocoded absorb unrelated calls purely on unit
# Skip this for recent incidents — an officer updating on the same # overlap (e.g. a patrol officer working a second scene 70 min
# scene without re-stating the address is normal and their update # after the original call).
# won't share much vocabulary with the original dispatch.
inc_emb_u = inc.get("embedding") inc_emb_u = inc.get("embedding")
if inc_emb_u: if inc_emb_u:
sim = _cosine_similarity(call_embedding, inc_emb_u) sim = _cosine_similarity(call_embedding, inc_emb_u)
if sim < 0.82: if sim < 0.82:
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence sim={sim:.3f} → content_divergence") logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence (has_call_coords/no_inc_coords) sim={sim:.3f} → content_divergence")
return False, "content_divergence" return False, "content_divergence"
logger.info(f" fits[{inc_id}]: unit_overlap matched={matched_units} is_dispatch={is_dispatch} → unit_overlap") elif call_embedding and idle_min >= 15:
# No geocode available AND old incident: use content divergence as a
# location-proxy veto. After 15+ minutes an officer at a completely
# different scene will be discussing clearly different content.
# Skip this for recent incidents — an officer updating on the same
# scene without re-stating the address is normal and their update
# won't share much vocabulary with the original dispatch.
inc_emb_u = inc.get("embedding")
if inc_emb_u:
sim = _cosine_similarity(call_embedding, inc_emb_u)
if sim < 0.82:
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence sim={sim:.3f} → content_divergence")
return False, "content_divergence"
logger.info(f" fits[{inc_id}]: unit_overlap matched={matched_units} → unit_overlap")
return True, "unit_overlap" return True, "unit_overlap"
# ── 2. Vehicle overlap ──────────────────────────────────────────────────── # ── 2. Vehicle overlap ────────────────────────────────────────────────────
@@ -1759,29 +1846,19 @@ def _call_fits_incident(
return False, "location_conflict" return False, "location_conflict"
# ── 4. No positive signals ──────────────────────────────────────────────── # ── 4. No positive signals ────────────────────────────────────────────────
# Requires at least one positive signal (unit, vehicle, or location match).
# A substantive call with no matching signals is more likely a separate
# incident than a follow-up — two dispatches can arrive within seconds of
# each other on a busy channel. Content-free thin calls are handled before
# this function via the thin path in correlate_call, with a tighter
# 30-second recency window.
logger.info( logger.info(
f" fits[{inc_id}]: no positive signal — is_dispatch={is_dispatch} idle={idle_min:.1f}min " f" fits[{inc_id}]: no positive signal — idle={idle_min:.1f}min "
f"inc_units={list(inc_units)} call_units={call_units} " f"inc_units={list(inc_units)} call_units={call_units} "
f"inc_vehicles={list(inc_vehicles)} call_vehicles={call_vehicles} " f"inc_vehicles={list(inc_vehicles)} call_vehicles={call_vehicles} "
f"call_coords={call_coords is not None} inc_coords={inc_coords is not None}" f"call_coords={call_coords is not None} inc_coords={inc_coords is not None}"
) )
if is_dispatch: return False, "no_signal"
# Dispatch channels require at least one positive signal (unit, vehicle,
# or location match). A substantive call with no matching signals is more
# likely a separate incident than a follow-up to the current one — two
# dispatches can arrive within seconds of each other on a busy channel.
# Content-free thin calls are handled before this function via the thin
# path in correlate_call, with a tighter 30-second recency window.
return False, "no_signal"
# Tactical channel: one scene per channel.
# Within 20 min of the last incident activity, link by default — same
# working channel almost certainly means same scene.
# After 20 min of silence, require at least one positive signal; the same
# frequency can be reused for a new unrelated incident later in the shift.
if idle_min < 20.0:
return True, "tactical_default"
return False, "tactical_idle"
async def _update_incident( async def _update_incident(
+33
View File
@@ -16,6 +16,7 @@ from typing import Optional
from app.internal.logger import logger from app.internal.logger import logger
from app.internal import firestore as fstore from app.internal import firestore as fstore
from app.internal import area_context from app.internal import area_context
from app.internal.chatter_classifier import classify_chatter
# Location validity is defined once, by the module that owns the incident's # Location validity is defined once, by the module that owns the incident's
# location/pin invariant. incident_correlator does not import this module, so # location/pin invariant. incident_correlator does not import this module, so
# this is not a cycle. # this is not a cycle.
@@ -199,6 +200,28 @@ async def extract_scenes(
pass pass
return [] return []
# server-26#127 — SHADOW MODE ONLY. Computes whether this transcript looks
# like non-event radio housekeeping (roll call, bare 10-4/10-8/98
# acknowledgements, unit check-ins) and records the verdict on the call
# doc, but does NOT skip extraction anywhere below — every path runs
# exactly as it did before this landed. Deliberately ahead of the ≤5-word
# skip: most bare acknowledgements ARE ≤5 words, and the first pass of
# this feature put the classifier after that return, so it never saw the
# bulk of its own target population — a review backtest against three
# live dumps found 82% of what it would have flagged already exits above
# as transcript_too_short, meaning a shadow-mode window would have shown
# roughly a fifth of the real catch rate. Computing it once, here, and
# folding the result into whichever skip/continue path runs below fixes
# that without adding a second Firestore write.
# TODO(server-26#127): flip this from shadow to live (skip extraction and
# write skip_reason="non_event_chatter" instead of just recording the
# verdict) once a live shadow-mode window confirms 0 false positives on
# real production traffic — pay particular attention to whole-transcript
# vs contains-anywhere matching for "roll call" and to digit-hyphen street
# addresses (e.g. "72-Holland"), both flagged as classifier risks that the
# dump backtest could not surface on its own.
chatter_is_chatter, chatter_reason = classify_chatter(transcript)
# Transcripts with ≤5 words carry no extractable intelligence — GPT hallucinates # Transcripts with ≤5 words carry no extractable intelligence — GPT hallucinates
# units and tags from thin context (e.g. "Main Lot", "10-4", "David"). # units and tags from thin context (e.g. "Main Lot", "10-4", "David").
if len(transcript.split()) <= 5: if len(transcript.split()) <= 5:
@@ -213,11 +236,21 @@ async def extract_scenes(
await fstore.doc_set("calls", call_id, { await fstore.doc_set("calls", call_id, {
"skip_reason": "transcript_too_short", "skip_reason": "transcript_too_short",
"severity": "routine", "severity": "routine",
"chatter_classifier_verdict": chatter_is_chatter,
"chatter_classifier_reason": chatter_reason,
}) })
except Exception: except Exception:
pass pass
return [] return []
try:
await fstore.doc_set("calls", call_id, {
"chatter_classifier_verdict": chatter_is_chatter,
"chatter_classifier_reason": chatter_reason,
})
except Exception:
pass
raw_scenes: list[dict] = await asyncio.to_thread( raw_scenes: list[dict] = await asyncio.to_thread(
_sync_extract, _sync_extract,
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes, transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
@@ -20,6 +20,46 @@ from app.internal.logger import logger
from app.internal import firestore as fstore from app.internal import firestore as fstore
from app.config import settings from app.config import settings
# server-26#131: minimum time since the real-time pipeline (routers/upload.py
# _run_intelligence_pipeline) marked intelligence_started_at before the sweep
# will touch a call, even though it already looks orphaned. STT + scene
# extraction + correlation is a multi-second-to-low-minutes chain (Whisper,
# then a Gemini call per scene); without this buffer the sweep could pick up
# a call mid-pipeline — no incident_id/corr_path written yet — and correlate
# it a second time, independently, sometimes onto a different incident than
# the real-time path lands on. That race is what #131 found (same call in
# two incidents' call_ids, ~2% of linked calls). A call with no
# intelligence_started_at at all (pre-#131 call doc, or the marker write
# itself failed) is NOT held back by this — absence isn't evidence of an
# in-flight pipeline, and #131's own bug predates this field existing.
#
# 15, not 5: neither OpenAI's Whisper client nor Gemini's call in
# llm_correlator.py sets a request timeout (server-26#153), so a hung call can
# run well past a few minutes on SDK-default retries, and this constant is a
# guess against that unbounded tail, not a measured bound. Raising it costs
# nothing on the recovery side: a call that finished processing (linked OR
# genuinely orphaned) always has corr_path set (_apply_and_log writes it even
# on the orphan action), so it's already excluded by the
# `not c.get("corr_path")` filter below and never reaches this check at all —
# this constant only ever delays calls that are still actually running.
MIN_MINUTES_SINCE_PIPELINE_START = 15
# Standard link-only retry budget before a call is tombstoned corr_path="unlinked".
MAX_SWEEP_ATTEMPTS = 3
# server-26#115 — a call the consensus LLM-orphan gate parked (llm=orphan vs
# rules=new, no substance) gets a longer budget. The gate fires before any
# incident for the job may exist, so the substantive call that would justify
# linking can land well after the standard ~6 min. Still link-only: a genuinely
# thin call must not mint an incident, and the rules creation gate would re-orphan
# it anyway.
GATED_ORPHAN_SWEEP_ATTEMPTS = 10
def _max_sweep_attempts(call: dict) -> int:
if call.get("corr_consensus") == "llm_orphan_gate":
return GATED_ORPHAN_SWEEP_ATTEMPTS
return MAX_SWEEP_ATTEMPTS
async def recorrelation_loop() -> None: async def recorrelation_loop() -> None:
interval = settings.summary_interval_minutes * 60 interval = settings.summary_interval_minutes * 60
@@ -36,8 +76,22 @@ async def recorrelation_loop() -> None:
logger.error(f"Re-correlation sweep failed: {e}") logger.error(f"Re-correlation sweep failed: {e}")
def _pipeline_likely_still_running(call: dict, now: datetime) -> bool:
"""server-26#131 — True when the real-time pipeline marked
intelligence_started_at recently enough that it's probably still mid-flight
(STT / scene extraction / correlation), so the sweep should not race it.
No marker at all (older call doc, or the marker write itself failed)
returns False — absence isn't evidence of an in-flight pipeline."""
started = _parse_dt(call.get("intelligence_started_at"))
if not started:
return False
age_minutes = (now - started).total_seconds() / 60
return age_minutes < MIN_MINUTES_SINCE_PIPELINE_START
async def _run_sweep_pass() -> None: async def _run_sweep_pass() -> None:
cutoff = datetime.now(timezone.utc) - timedelta(minutes=settings.recorrelation_scan_minutes) now = datetime.now(timezone.utc)
cutoff = now - timedelta(minutes=settings.recorrelation_scan_minutes)
# Server-side range query: only calls that ended within the scan window. # Server-side range query: only calls that ended within the scan window.
# Filter incident_id=null client-side (Firestore can't query for missing fields). # Filter incident_id=null client-side (Firestore can't query for missing fields).
@@ -46,10 +100,9 @@ async def _run_sweep_pass() -> None:
("status", "==", "ended"), ("status", "==", "ended"),
("ended_at", ">=", cutoff), ("ended_at", ">=", cutoff),
]) ])
# corr_path="unlinked" is written after MAX_SWEEP_ATTEMPTS failures. # corr_path="unlinked" is written after the attempt budget is exhausted.
# Allows a few retries so a welfare-check call can link to an escalation # Allows a few retries so a welfare-check call can link to an escalation
# incident that is created a few minutes later, without sweeping 30× forever. # incident that is created a few minutes later, without sweeping 30× forever.
MAX_SWEEP_ATTEMPTS = 3
orphans = [ orphans = [
c for c in recent_ended c for c in recent_ended
if not c.get("incident_ids") and not c.get("incident_id") if not c.get("incident_ids") and not c.get("incident_id")
@@ -61,7 +114,8 @@ async def _run_sweep_pass() -> None:
# the thin path minutes later and attached to whatever was most recent — # the thin path minutes later and attached to whatever was most recent —
# a second route into the over-merge the thin fix above addresses. # a second route into the over-merge the thin fix above addresses.
and not c.get("skip_reason") and not c.get("skip_reason")
and c.get("corr_sweep_count", 0) < MAX_SWEEP_ATTEMPTS and c.get("corr_sweep_count", 0) < _max_sweep_attempts(c)
and not _pipeline_likely_still_running(c, now)
] ]
if not orphans: if not orphans:
@@ -120,12 +174,12 @@ async def _recorrelate_orphan(call: dict) -> bool:
) )
return True return True
# Increment the attempt counter. Once MAX_SWEEP_ATTEMPTS is reached the # Increment the attempt counter. Once the budget is reached the orphan filter
# orphan filter above will stop picking this call up, and we write # above will stop picking this call up, and we write corr_path="unlinked" as
# corr_path="unlinked" as a permanent tombstone. # a permanent tombstone.
attempts = call.get("corr_sweep_count", 0) + 1 attempts = call.get("corr_sweep_count", 0) + 1
update: dict = {"corr_sweep_count": attempts} update: dict = {"corr_sweep_count": attempts}
if attempts >= 3: if attempts >= _max_sweep_attempts(call):
update["corr_path"] = "unlinked" update["corr_path"] = "unlinked"
await fstore.doc_set("calls", call_id, update) await fstore.doc_set("calls", call_id, update)
return False return False
+54 -3
View File
@@ -15,6 +15,39 @@ from app.internal import firestore as fstore
from app.config import settings from app.config import settings
def _scene_sort_key(scene_index: str):
"""Numeric-first sort so a >=10-scene call's entries still read in order."""
return (0, int(scene_index)) if scene_index.isdigit() else (1, scene_index)
def _scene_text_for_incident(doc: dict, incident_id: str) -> Optional[str]:
"""
The text of `doc` (a call doc) that actually belongs to `incident_id`.
server-26#96 records, per scene, which incident_id that scene's
correlation decision resolved to (incident_correlator._apply_and_log's
`scenes.<index>.incident_id`). Use that to pick only the scene(s) of this
call that are genuinely part of this incident, joining more than one if
several scenes happened to link into the same incident.
Falls back to transcript_corrected-or-transcript when the call doc has no
`scenes` field (predates server-26#96) or — defensively — when it has one
but nothing in it names this incident_id (should not happen for a call_id
that's actually in this incident's call_ids, but silently dropping a
call's contribution to its own summary would be a worse failure mode than
falling back to the whole-call text).
"""
scenes = doc.get("scenes") or {}
matched = [
scene.get("transcript")
for _, scene in sorted(scenes.items(), key=lambda kv: _scene_sort_key(kv[0]))
if scene.get("incident_id") == incident_id and scene.get("transcript")
]
if matched:
return "\n".join(matched)
return doc.get("transcript_corrected") or doc.get("transcript")
async def summarizer_loop() -> None: async def summarizer_loop() -> None:
from app.internal.feature_flags import get_flags from app.internal.feature_flags import get_flags
interval = settings.summary_interval_minutes * 60 interval = settings.summary_interval_minutes * 60
@@ -63,12 +96,30 @@ async def _summarize_incident(inc: dict) -> None:
if not call_ids: if not call_ids:
return return
# Fetch transcripts for all calls in this incident # Fetch transcripts for all calls in this incident.
#
# server-26#114: a call links into an incident one SCENE at a time (see
# incident_correlator._apply_decision / server-26#96's `scenes` map on the
# call doc), and the same call_id can appear in more than one incident's
# call_ids — once per scene, each scene possibly landing in a different
# incident. Reading doc["transcript"] (the whole call, raw) meant an
# incident's summary was built partly on text from a DIFFERENT scene of
# that call that this incident has nothing to do with, and ignored
# transcript_corrected entirely.
#
# _scene_text_for_incident reads the specific scene(s) whose corr_debug
# recorded a link into THIS incident_id. For a call doc that predates
# this fix (no `scenes` field) it falls back to
# transcript_corrected-or-transcript — the one-liner half of #114, worth
# doing even for old-schema docs since it stops raw-transcript summaries.
transcripts: list[str] = [] transcripts: list[str] = []
for cid in call_ids: for cid in call_ids:
doc = await fstore.doc_get("calls", cid) doc = await fstore.doc_get("calls", cid)
if doc and doc.get("transcript"): if not doc:
transcripts.append(doc["transcript"]) continue
text = _scene_text_for_incident(doc, incident_id)
if text:
transcripts.append(text)
if not transcripts: if not transcripts:
# No transcripts yet — clear stale flag and wait for next pass # No transcripts yet — clear stale flag and wait for next pass
+24 -17
View File
@@ -78,33 +78,40 @@ async def lifespan(app: FastAPI):
app = FastAPI(title="DRB C2 Core", lifespan=lifespan) app = FastAPI(title="DRB C2 Core", lifespan=lifespan)
# "*" plus allow_credentials=True is not the permissive-but-harmless setting it # The browser needs CORS to reach this API at all: the frontend's Archive page
# looks like. Starlette does not refuse the combination -- it reflects the # calls GET /calls/search with Authorization + Content-Type headers, which
# caller's Origin back and still sends Access-Control-Allow-Credentials: true, # forces a preflight. Without this middleware the OPTIONS gets a bare 405 and
# so the effective policy becomes "any origin, with credentials", the opposite # the fetch fails (#110). allow_origins is an explicit list -- never "*" in a
# of what a wildcard normally means. Rather than trust every deployment to # deployment -- so name every host the frontend is served from in CORS_ORIGINS.
# remember to override CORS_ORIGINS, make the dangerous pair unrepresentable. #
# allow_credentials stays False on purpose: auth here is a Bearer header, not a
# cookie, so credentialed CORS is never needed, and keeping it False is what
# lets an explicit-origin allowlist work without Starlette's "*"-only
# restriction. "*" + credentials is the dangerous pair (Starlette reflects the
# caller's Origin back WITH Access-Control-Allow-Credentials: true); this code
# cannot produce it because credentials are hard-off.
def cors_allows_credentials(origins: list[str]) -> bool: def cors_allows_credentials(origins: list[str]) -> bool:
"""False when any entry is a wildcard. Extracted so it can be tested """Always False -- credentialed CORS is never enabled here (Bearer auth,
without re-importing this module, which drags in every router.""" not cookies). Kept as a named predicate so a future edit that wants to
return "*" not in origins turn credentials on has to go through here and confront the "*" case.
A wildcard entry would additionally be refused a credentialed response."""
return False
_cors_is_wildcard = not cors_allows_credentials(settings.cors_origins) _cors_is_wildcard = "*" in settings.cors_origins
if _cors_is_wildcard: if _cors_is_wildcard:
logger.error( logger.error(
"CORS_ORIGINS is '*', so credentialed cross-origin requests are being " "CORS_ORIGINS contains '*'. That is fine for local dev but is almost "
"DISABLED to avoid reflecting every caller's origin back with " "certainly a misconfigured deployment -- set CORS_ORIGINS to your "
"Access-Control-Allow-Credentials. Set CORS_ORIGINS to your frontend " "frontend origin(s), e.g. [\"https://drb.cusano.net\"]."
"origin(s) in production, e.g. [\"https://app.example.com\"]."
) )
app.add_middleware( app.add_middleware(
CORSMiddleware, CORSMiddleware,
allow_origins=settings.cors_origins, allow_origins=settings.cors_origins,
allow_methods=["*"], allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"],
allow_headers=["*"], allow_headers=["authorization", "content-type"],
allow_credentials=not _cors_is_wildcard, allow_credentials=False,
) )
app.include_router(nodes.router, dependencies=[Depends(require_service_or_firebase_token)]) app.include_router(nodes.router, dependencies=[Depends(require_service_or_firebase_token)])
+100 -5
View File
@@ -97,8 +97,54 @@ async def debug_correlation(
def _strip(doc: dict) -> dict: def _strip(doc: dict) -> dict:
return {k: v for k, v in doc.items() if k != "embedding"} return {k: v for k, v in doc.items() if k != "embedding"}
def _call_summary(call: dict) -> dict: def _scene_summary(scene_index: str, scene: dict) -> dict:
"""
One scene's own correlation record, from the call doc's `scenes` map
(server-26#96). Same corr_* field names as _call_summary's flat
fields below, deliberately — a scene entry and a scene-less call
summary are interchangeable data points to the tally functions.
"""
corr_debug = scene.get("corr_debug") or {}
return { return {
"scene_index": scene_index,
"transcript": scene.get("transcript"),
"incident_id": scene.get("incident_id"),
# server-26#139: this scene's OWN incident_type/severity, as seen
# by _call_is_substanceless at decision time — not the call doc's
# flat top-level field, which is last-scene-wins (server-26#96).
"incident_type": scene.get("incident_type"),
"severity": scene.get("severity"),
"corr_path": corr_debug.get("corr_path"),
"corr_incident_idle_min": corr_debug.get("corr_incident_idle_min"),
"corr_distance_km": corr_debug.get("corr_distance_km"),
"corr_score": corr_debug.get("corr_score"),
"corr_candidates": corr_debug.get("corr_candidates"),
"corr_shared_units": corr_debug.get("corr_shared_units"),
"corr_fit_signal": corr_debug.get("corr_fit_signal"),
"corr_matched_units": corr_debug.get("corr_matched_units"),
"corr_consensus": corr_debug.get("corr_consensus"),
"corr_llm_reasoning": corr_debug.get("corr_llm_reasoning"),
"corr_llm_action": corr_debug.get("corr_llm_action"),
"corr_rules_action": corr_debug.get("corr_rules_action"),
"corr_gate_veto": corr_debug.get("corr_gate_veto"),
}
def _call_summary(call: dict) -> dict:
# server-26#96 — per-scene records, keyed by scene index as written by
# incident_correlator._apply_and_log. Present only on calls that went
# through correlation after this fix landed; absent (None) on older
# call docs, which the tally below falls back for. Sorted numerically
# so a >=10-scene call still reads in scene order.
scenes_map = call.get("scenes") or {}
scenes = [
_scene_summary(idx, s)
for idx, s in sorted(
scenes_map.items(),
key=lambda kv: (0, int(kv[0])) if kv[0].isdigit() else (1, kv[0]),
)
] or None
return {
"scenes": scenes,
"call_id": call.get("call_id"), "call_id": call.get("call_id"),
"started_at": call.get("started_at"), "started_at": call.get("started_at"),
"ended_at": call.get("ended_at"), "ended_at": call.get("ended_at"),
@@ -135,6 +181,19 @@ async def debug_correlation(
"corr_llm_reasoning": call.get("corr_llm_reasoning"), "corr_llm_reasoning": call.get("corr_llm_reasoning"),
"corr_llm_action": call.get("corr_llm_action"), "corr_llm_action": call.get("corr_llm_action"),
"corr_rules_action": call.get("corr_rules_action"), "corr_rules_action": call.get("corr_rules_action"),
# server-26#115 — why an llm=orphan/rules=new disagreement escalated
# to tiebreak instead of being gated (see upload.py's
# _call_is_substanceless). Present only on that disagreement shape;
# written here specifically so a live measurement window can read
# the reason instead of reconstructing it by hand from the dump.
"corr_gate_veto": call.get("corr_gate_veto"),
# server-26#127 — shadow-mode upstream chatter classifier verdict.
# Written by intelligence.extract_scenes on every transcript that
# reaches real scene extraction (not on garbage/too-short skips).
# Nothing skips extraction on this yet — it's here purely so a
# live measurement window can read the false-positive rate.
"chatter_classifier_verdict": call.get("chatter_classifier_verdict"),
"chatter_classifier_reason": call.get("chatter_classifier_reason"),
} }
# ── Determine which systems have AI active ──────────────────────────────── # ── Determine which systems have AI active ────────────────────────────────
@@ -258,6 +317,21 @@ async def debug_correlation(
linked = [c for inc in incident_records for c in (inc.get("calls_detail") or [])] linked = [c for inc in incident_records for c in (inc.get("calls_detail") or [])]
call_counts = [len(inc.get("call_ids") or []) for inc in incident_records] call_counts = [len(inc.get("call_ids") or []) for inc in incident_records]
def _tally_entries(call_summary: dict) -> list:
"""
server-26#96 — the unit correlation actually decided over is the
scene, not the call. A call summary carrying a `scenes` list (every
call correlated after this fix) contributes one entry per scene, each
with its own corr_path/corr_consensus/etc, instead of the single flat
record that used to blend every scene's last write together. A call
summary with no `scenes` (a call doc from before this fix) falls back
to contributing itself as one entry — identical to pre-#96 behaviour.
"""
scenes = call_summary.get("scenes")
return scenes if scenes else [call_summary]
scene_entries = [entry for c in linked for entry in _tally_entries(c)]
def _span_minutes(inc: dict) -> float: def _span_minutes(inc: dict) -> float:
stamps = sorted( stamps = sorted(
s for s in ((c.get("started_at") or "") for c in (inc.get("calls_detail") or [])) if s s for s in ((c.get("started_at") or "") for c in (inc.get("calls_detail") or [])) if s
@@ -289,10 +363,31 @@ async def debug_correlation(
"ai_systems_only": ai_systems_only, "ai_systems_only": ai_systems_only,
"ai_enabled_system_ids": sorted(ai_systems), "ai_enabled_system_ids": sorted(ai_systems),
"linked_call_count": len(linked), "linked_call_count": len(linked),
"corr_path": _tally(c.get("corr_path") for c in linked), # server-26#96 — tallied over scene_entries (one entry per scene of a
"corr_fit_signal": _tally(c.get("corr_fit_signal") for c in linked), # multi-scene call, from its `scenes` map; one entry per call when it
"corr_consensus": _tally(c.get("corr_consensus") for c in linked), # has none) rather than over `linked` directly, so a 2-scene call
"corr_llm_action": _tally(c.get("corr_llm_action") for c in linked), # with two different corr_path values counts as two data points
# instead of one blended flat record. scene_decision_count makes that
# distinction visible next to linked_call_count.
"scene_decision_count": len(scene_entries),
"corr_path": _tally(e.get("corr_path") for e in scene_entries),
"corr_fit_signal": _tally(e.get("corr_fit_signal") for e in scene_entries),
"corr_consensus": _tally(e.get("corr_consensus") for e in scene_entries),
"corr_llm_action": _tally(e.get("corr_llm_action") for e in scene_entries),
# server-26#115 — this IS the number the escape-hatch fix exists to
# produce: why each llm=orphan/rules=new call escaped the gate.
"corr_gate_veto": _tally(e.get("corr_gate_veto") for e in scene_entries),
# server-26#127 — shadow-mode chatter classifier. The target
# population is non-events, which land as orphans or single-call
# incidents, NOT as a slice of every linked call -- tally `orphans`
# too or this undercounts the exact thing the feature measures.
"chatter_classifier_flagged": sum(
1 for c in (linked + orphans) if c.get("chatter_classifier_verdict")
),
"chatter_classifier_reason": _tally(
c.get("chatter_classifier_reason") for c in (linked + orphans)
if c.get("chatter_classifier_verdict")
),
# STT coverage: correlation quality is capped by this, so it belongs in # STT coverage: correlation quality is capped by this, so it belongs in
# the same view rather than a separate investigation. # the same view rather than a separate investigation.
"linked_calls_with_transcript": with_transcript, "linked_calls_with_transcript": with_transcript,
+9
View File
@@ -260,6 +260,15 @@ async def patch_transcript(
"vehicles": [], "vehicles": [],
"embedding": None, "embedding": None,
}) })
# server-26#96/#114 review: doc_set(merge=True) can only ADD/overwrite keys
# in a nested map, never remove one, so the fields above get cleared but a
# prior `scenes` map would survive re-extraction forever. A call corrected
# from 3 scenes down to 1 would keep scenes.1/scenes.2 with pre-correction
# transcripts and incident_ids -- corrupting the exact per-scene tally #96
# exists to make trustworthy, and re-feeding stale text into #114's
# summarizer fix if a stale scene's incident_id still names a real
# incident. Must be a real delete, not a merge over an empty map.
await fstore.doc_update("calls", call_id, {"scenes": fstore.DELETE_FIELD})
# Unlink from ALL current incidents so re-correlation starts clean. # Unlink from ALL current incidents so re-correlation starts clean.
# Handles both old single incident_id and new incident_ids list. # Handles both old single incident_id and new incident_ids list.
+165 -2
View File
@@ -100,6 +100,98 @@ async def upload_call_audio(
return {"url": gcs_uri} return {"url": gcs_uri}
# server-26#115 — the consensus LLM-orphan gate only fires when the call is
# genuinely substanceless. The earlier version tested `rules_decision["corr_debug"]`
# for a "positive signal", but corr_debug is EMPTY at preview time for
# action=="new" (corr_path:"new" is written at APPLY time), so that test was
# always False and the gate dropped real events — a major "extinguishing fire",
# geocoded calls, pursuit updates. The substance test now runs against `ctx`,
# which is fully populated at preview time.
def _recent_incident_on_same_talkgroup(ctx: dict) -> bool:
"""
True when a recent incident is running on this call's own system +
talkgroup, within `settings.tg_dispatch_thin_idle_minutes` (5 min) —
applied uniformly regardless of the talkgroup's name (server-26#134).
Covers "unit dispatched, thin ack 10-30s later": the ack has no
substance of its own but plainly belongs to the job just opened.
Reads ctx["recent"] (the rules engine's own candidate list — no extra
Firestore read). That list is status=="active" incidents only, so an
already-resolved or capacity-capped same-talkgroup incident won't be
seen here even if chronologically recent (server-26#115, unresolved —
would need a dedicated non-status-filtered query).
Whether this limitation explains the 2/24 unexplained gate misses in the
window #3 measurement is UNANSWERED, not confirmed either way — a prior
pass here claimed a "confirmed explanation" for both that turned out to
be self-contradictory. Read `corr_gate_veto` (written to corr_debug on
every escalation of this exact disagreement shape — see the caller) in
the next measurement window instead of guessing from the raw dump again.
# TODO(server-26#115): add a talkgroup-scoped incident lookup (any
# status, no capacity filter) if a future measurement window pins a real
# gate miss on a resolved/capped same-talkgroup incident.
"""
from app.internal.incident_correlator import _idle_gate_minutes
tg_id = ctx.get("talkgroup_id")
system_id = ctx.get("system_id")
if tg_id is None or not system_id:
return False
tg_str = str(tg_id)
now = ctx.get("now") or datetime.now(timezone.utc)
idle_limit = settings.tg_dispatch_thin_idle_minutes
for inc in ctx.get("recent") or []:
if system_id not in (inc.get("system_ids") or []):
continue
if tg_str not in (inc.get("talkgroup_ids") or []):
continue
if _idle_gate_minutes(inc, now) <= idle_limit:
return True
return False
def _call_is_substanceless(ctx: dict) -> tuple[bool, Optional[str]]:
"""
True when the call carries nothing that marks it as a real event:
• no resolved incident_type and not a reassignment, AND
• severity is not moderate/major, AND
• no vehicle, geocode or tag (incident_correlator.has_event_substance —
the same predicate the incident-creation gate uses), AND
• no recent incident already running on the same talkgroup.
Only then may the LLM-orphan gate drop the call without a tiebreak.
Returns (substanceless, veto_reason). veto_reason names whichever
condition kept the tiebreak alive ("type" | "reassignment" | "severity" |
"substance" | "recent_tg"), or None when the call is substanceless. The
caller writes this into corr_debug on the escalation path so a live
measurement window can see *why* each llm=orphan/rules=new call escaped
the gate instead of inferring it after the fact from the raw dump —
exactly the guesswork that produced a wrong "confirmed explanation" for
2 window-#3 misses on the first pass of this fix.
"""
from app.internal import incident_correlator
# The incident-creation gate skips the has_event_substance check entirely
# when a type resolved (incident_correlator._run_decision ~:1397), so a
# typed call — fire/medical/etc. — opens an incident on substance we do not
# re-check here. reassignment=True is dispatch pulling a unit onto a NEW
# job (units are blanked at :296 for exactly that reason): the strongest
# new-incident signal in the pipeline. Either one means "keep the tiebreak".
if ctx.get("incident_type"):
return False, "type"
if ctx.get("reassignment"):
return False, "reassignment"
if (ctx.get("call_severity") or "routine") in ("moderate", "major"):
return False, "severity"
if incident_correlator.has_event_substance(ctx):
return False, "substance"
if _recent_incident_on_same_talkgroup(ctx):
return False, "recent_tg"
return True, None
async def _correlate_with_consensus( async def _correlate_with_consensus(
call_id: str, call_id: str,
node_id: str, node_id: str,
@@ -117,6 +209,7 @@ async def _correlate_with_consensus(
embedding: Optional[list] = None, embedding: Optional[list] = None,
severity: Optional[str] = None, severity: Optional[str] = None,
transcript: Optional[str] = None, transcript: Optional[str] = None,
scene_index: int = 0,
) -> Optional[str]: ) -> Optional[str]:
""" """
Consensus correlator: runs the rules engine and the cheap LLM in sequence. Consensus correlator: runs the rules engine and the cheap LLM in sequence.
@@ -125,6 +218,11 @@ async def _correlate_with_consensus(
Falls back to rules-only when GEMINI_API_KEY is absent, the call is Falls back to rules-only when GEMINI_API_KEY is absent, the call is
content-free (thin), or any LLM call fails. content-free (thin), or any LLM call fails.
``scene_index`` (server-26#96) — which scene of the call this is, from the
caller's ``enumerate(scenes)`` loop. Threaded through so the call doc's
per-scene ``scenes`` map records this scene's own corr_debug/transcript
instead of colliding with every other scene's write on the flat fields.
""" """
from app.internal import incident_correlator, llm_correlator from app.internal import incident_correlator, llm_correlator
@@ -135,6 +233,7 @@ async def _correlate_with_consensus(
location_coords=location_coords, units=units, vehicles=vehicles, location_coords=location_coords, units=units, vehicles=vehicles,
cleared_units=cleared_units, reassignment=reassignment, cleared_units=cleared_units, reassignment=reassignment,
embedding=embedding, severity=severity, transcript=transcript, embedding=embedding, severity=severity, transcript=transcript,
scene_index=scene_index,
) )
ctx = preview["ctx"] ctx = preview["ctx"]
rules_decision = preview["decision"] rules_decision = preview["decision"]
@@ -151,6 +250,37 @@ async def _correlate_with_consensus(
rules_decision["corr_debug"]["corr_llm_reasoning"] = llm_decision.get("reasoning", "") rules_decision["corr_debug"]["corr_llm_reasoning"] = llm_decision.get("reasoning", "")
return await incident_correlator.apply_correlation(preview) return await incident_correlator.apply_correlation(preview)
# server-26#115 — LLM-orphan gate.
# When the cheap LLM says `orphan`, the rules engine says `new`, and the call
# is genuinely substanceless (routine severity, no vehicle/geocode/tag, and
# no incident already running on this talkgroup), resolve to `orphan` and DO
# NOT pay for the smart tiebreaker. A bare rules `new` there means only
# "nothing to link to" — trivially true for radio housekeeping (check-ins,
# roll call, 10-8/10-98) — and the tiebreaker rubber-stamped it ~21/21 of the
# time on exactly this disagreement (CORRELATION_REVIEW_0907b.md). Any real
# signal (severity, coords, tags, a live same-talkgroup incident) still
# escalates, so an event the LLM misreads as orphan is not lost.
is_orphan_vs_new = llm_decision["action"] == "orphan" and rules_decision["action"] == "new"
substanceless, gate_veto_reason = _call_is_substanceless(ctx) if is_orphan_vs_new else (False, None)
if is_orphan_vs_new and substanceless:
logger.info(
f"Consensus gate for call {call_id}: llm=orphan vs rules=new and call "
f"is substanceless — resolving orphan, skipping tiebreak"
)
gated = {
"action": "orphan",
"matched_incident": None,
"incident_type": None,
"corr_debug": dict(rules_decision.get("corr_debug") or {}),
}
gated["corr_debug"].update({
"corr_consensus": "llm_orphan_gate",
"corr_rules_action": rules_decision["action"],
"corr_llm_action": llm_decision["action"],
"corr_llm_reasoning": llm_decision.get("reasoning", ""),
})
return await incident_correlator.apply_correlation({"decision": gated, "ctx": ctx})
# Disagree — escalate to the smarter tiebreaker. # Disagree — escalate to the smarter tiebreaker.
logger.info( logger.info(
f"Consensus disagreement for call {call_id}: " f"Consensus disagreement for call {call_id}: "
@@ -160,6 +290,12 @@ async def _correlate_with_consensus(
final["corr_debug"]["corr_consensus"] = "tiebreak" final["corr_debug"]["corr_consensus"] = "tiebreak"
final["corr_debug"]["corr_rules_action"] = rules_decision["action"] final["corr_debug"]["corr_rules_action"] = rules_decision["action"]
final["corr_debug"]["corr_llm_action"] = llm_decision["action"] final["corr_debug"]["corr_llm_action"] = llm_decision["action"]
if is_orphan_vs_new:
# server-26#115 — record *why* the llm=orphan/rules=new gate stood
# down instead of leaving a future measurement window to guess it
# from the raw dump (which produced a wrong "confirmed explanation"
# for 2/24 misses the first time around).
final["corr_debug"]["corr_gate_veto"] = gate_veto_reason
return await incident_correlator.apply_correlation({"decision": final, "ctx": ctx}) return await incident_correlator.apply_correlation({"decision": final, "ctx": ctx})
@@ -205,7 +341,10 @@ async def _run_extraction_pipeline(
) )
# Step 3: Correlate each scene to an incident independently. # Step 3: Correlate each scene to an incident independently.
for scene in scenes: # server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"]) all_tags.extend(scene["tags"])
# When dispatch is pulling a unit to a NEW call (reassignment), suppress unit # When dispatch is pulling a unit to a NEW call (reassignment), suppress unit
# overlap so the new scene doesn't chain into the unit's previous incident. # overlap so the new scene doesn't chain into the unit's previous incident.
@@ -228,6 +367,7 @@ async def _run_extraction_pipeline(
embedding=scene.get("embedding"), embedding=scene.get("embedding"),
severity=scene.get("severity"), severity=scene.get("severity"),
transcript=scene.get("transcript"), transcript=scene.get("transcript"),
scene_index=scene_index,
) )
if incident_id and incident_id not in incident_ids: if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id) incident_ids.append(incident_id)
@@ -273,6 +413,25 @@ async def _run_intelligence_pipeline(
""" """
from app.internal import transcription, intelligence, incident_correlator, alerter, talkgroups from app.internal import transcription, intelligence, incident_correlator, alerter, talkgroups
# server-26#131: mark that real-time processing has started for this call
# BEFORE any of the slow steps below (STT, scene extraction, correlation).
# The re-correlation sweep (internal/recorrelation_sweep.py) scans for
# calls that still look orphaned within a wide window (recorrelation_scan_
# minutes, default 60) — with no guard here, a call whose real-time
# pipeline is still mid-flight (still transcribing, still waiting on a
# Gemini call) has no incident_id/corr_path written yet, so the sweep's
# orphan filter can't tell "never processed" from "processing right now"
# and correlates it a second time, independently, sometimes landing on a
# different incident than the real-time path — the exact duplicate-link
# bug #131 found (same call in two incidents' call_ids, ~2% of linked
# calls). Best-effort: a write failure here must not abort the pipeline.
try:
await fstore.doc_set("calls", call_id, {
"intelligence_started_at": datetime.now(timezone.utc).isoformat()
})
except Exception as e:
logger.warning(f"Could not mark intelligence_started_at for call {call_id}: {e}")
# The node only sends talkgroup_name when OP25 had it in the loaded tags # The node only sends talkgroup_name when OP25 had it in the loaded tags
# file, so it arrives empty for exactly the talkgroups C2 can name from the # file, so it arrives empty for exactly the talkgroups C2 can name from the
# system config. Resolve it once, here, at the single funnel both /upload # system config. Resolve it once, here, at the single funnel both /upload
@@ -325,7 +484,10 @@ async def _run_intelligence_pipeline(
incident_ids: list[str] = [] incident_ids: list[str] = []
all_tags: list[str] = [] all_tags: list[str] = []
if _flag("correlation_enabled"): if _flag("correlation_enabled"):
for scene in scenes: # server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"]) all_tags.extend(scene["tags"])
is_reassignment = bool(scene.get("reassignment")) is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units") corr_units = [] if is_reassignment else scene.get("units")
@@ -346,6 +508,7 @@ async def _run_intelligence_pipeline(
embedding=scene.get("embedding"), embedding=scene.get("embedding"),
severity=scene.get("severity"), severity=scene.get("severity"),
transcript=scene.get("transcript"), transcript=scene.get("transcript"),
scene_index=scene_index,
) )
if incident_id and incident_id not in incident_ids: if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id) incident_ids.append(incident_id)
+4
View File
@@ -34,6 +34,10 @@ except ModuleNotFoundError:
# into dicts that tests compare against, and a MagicMock compares unequal # into dicts that tests compare against, and a MagicMock compares unequal
# to itself across attribute accesses. # to itself across attribute accesses.
_fs.SERVER_TIMESTAMP = "__SERVER_TIMESTAMP__" _fs.SERVER_TIMESTAMP = "__SERVER_TIMESTAMP__"
# Same reasoning as SERVER_TIMESTAMP above: a distinct sentinel, not a
# MagicMock, so `fstore.DELETE_FIELD is fs.DELETE_FIELD` and dict/`is`
# comparisons against it in tests (server-26#96/#114, PR #132) behave.
_fs.DELETE_FIELD = "__DELETE_FIELD__"
_auth = ModuleType("firebase_admin.auth") _auth = ModuleType("firebase_admin.auth")
_auth.verify_id_token = MagicMock() _auth.verify_id_token = MagicMock()
@@ -89,3 +89,75 @@ async def test_debug_correlation_llm_fields_absent_when_rules_only():
assert detail["corr_consensus"] == "rules_only" assert detail["corr_consensus"] == "rules_only"
assert detail["corr_llm_reasoning"] is None assert detail["corr_llm_reasoning"] is None
assert detail["corr_llm_action"] is None assert detail["corr_llm_action"] is None
# ---------------------------------------------------------------------------
# server-26#96 — the summary tally must count per-scene decisions, not the
# one blended flat record a multi-scene call used to leave behind.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_debug_correlation_exposes_scenes_and_tallies_each_as_its_own_datapoint():
"""A 2-scene call: the flat fields still show last-scene-wins (unchanged
behaviour for old readers), but the summary tally must see two distinct
corr_path/corr_consensus data points, not one blend."""
call = {
"call_id": "call-1",
# Flat fields — last scene wins, kept as-is for backward compat.
"corr_path": "slow",
"corr_consensus": "tiebreak",
"scenes": {
"0": {
"transcript": "scene zero",
"incident_id": "inc-1",
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
},
"1": {
"transcript": "scene one",
"incident_id": "inc-1",
"corr_debug": {"corr_path": "slow", "corr_consensus": "tiebreak"},
},
},
}
result = await _run([_incident(["call-1"])], {"call-1": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["corr_path"] == "slow" # flat field: last scene wins
assert len(detail["scenes"]) == 2
assert detail["scenes"][0]["corr_path"] == "new"
assert detail["scenes"][1]["corr_path"] == "slow"
summary = result["summary"]
assert summary["linked_call_count"] == 1 # still one CALL
assert summary["scene_decision_count"] == 2 # but two DECISIONS
assert summary["corr_path"] == {"new": 1, "slow": 1}
assert summary["corr_consensus"] == {"agreed": 1, "tiebreak": 1}
@pytest.mark.asyncio
async def test_debug_correlation_tally_falls_back_for_single_scene_call():
"""A plain single-scene call has no `scenes` field at all — the tally
must fall back to its flat fields as one data point, same as pre-#96."""
call = {"call_id": "call-2", "corr_path": "fast/single", "corr_consensus": "rules_only"}
result = await _run([_incident(["call-2"])], {"call-2": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["scenes"] is None
summary = result["summary"]
assert summary["linked_call_count"] == 1
assert summary["scene_decision_count"] == 1
assert summary["corr_path"] == {"fast/single": 1}
assert summary["corr_consensus"] == {"rules_only": 1}
@pytest.mark.asyncio
async def test_debug_correlation_tally_handles_old_schema_call_with_no_scenes_field():
"""A call doc written before server-26#96 has never heard of `scenes` —
must behave identically to the single-scene case, not error."""
old_call = {"call_id": "call-3", "corr_path": "cross-tg", "corr_consensus": "agreed"}
result = await _run([_incident(["call-3"])], {"call-3": old_call})
summary = result["summary"]
assert summary["scene_decision_count"] == 1
assert summary["corr_path"] == {"cross-tg": 1}
@@ -0,0 +1,144 @@
"""
server-26#127 — upstream dispatch-vs-chatter classifier, shadow mode.
Fixtures are real transcripts, not invented ones: pulled from
`corr_dump_9-7_0437am.json`, `corr_dump_9-7_pm.json`, `corr_dump_9-12.json`
and the hand-labeled examples in `CORRELATION_REVIEW_0907b.md` /
`CORRELATION_REVIEW_0912.md`. The "must classify False" set specifically
includes every transcript those review docs flagged as dangerous to drop —
a false positive here is a real event silently losing its scene once this
classifier ever goes live, which is a much worse failure than a missed
chatter call staying in the existing (already-working) pipeline.
"""
import pytest
from app.internal.chatter_classifier import classify_chatter
# ─────────────────────────────────────────────────────────────────────────────
# Must classify as chatter
# ─────────────────────────────────────────────────────────────────────────────
CHATTER_EXAMPLES = [
# Bare acknowledgements / unit check-ins (CORRELATION_REVIEW_0907b.md)
("114 Paul.\n114 Paul, Metro Central.\n10-4.", "bare_acknowledgement"),
("Affirmative, in charge of 10-8. 10-8, 10-4.", "bare_acknowledgement"),
("6-8, you can show me 98. 10-4.", "bare_acknowledgement"),
("10-4, 10-4 Central, 98. 10-4, 98.", "bare_acknowledgement"),
("7 for Post 1 and 2, 98. Affirm.", "bare_acknowledgement"),
("11-Victor to Central. 11-Victor. 72-Holland, 1-5. Central.", "bare_acknowledgement"),
# Roll call (CORRELATION_REVIEW_0907b.md / _0912.md)
("Post 4, Ossining. And to volunteer patrol, stand by for roll call.", "roll_call"),
("Headquarters to all cars, stand by for roll call.", "roll_call"),
("All Troop NYC Patrols, stand by for roll call.", "roll_call"),
(
"Car 100, roll call.\nHenry 1.\nHenry 1.\nSam 1.\nSam 1.\n45 Baker.\n"
"45 Baker.\n11 Adam.\nAdam.\n11 Baker.\nBaker.\nStaff 1.\n1.\nStaff 2.",
"roll_call",
),
(
"Headquarters, all cars on a roll call. Baker 1? Baker 1. Henry 1? "
"Henry 1. Sam 2? Sam 2. 11 Adam? 11. 11 Baker? 11 Baker.",
"roll_call",
),
("Because all cars came out for roll call.", "roll_call"),
("10-1. KL Cars, that concludes roll call, time is 3-31.", "roll_call"),
# Minimal single-word / bare-code transmissions (orphan pool, all 3 dumps)
("10-4.", "bare_acknowledgement"),
("Roger.", "bare_acknowledgement"),
("Clear.", "bare_acknowledgement"),
("Affirmative.", "bare_acknowledgement"),
("Received.", "bare_acknowledgement"),
("10-8, clear. 10-4.", "bare_acknowledgement"),
("Post 4, 10-8. 10-4.", "bare_acknowledgement"),
("Central to 6 Henry.", "bare_acknowledgement"),
]
@pytest.mark.parametrize("transcript,expected_reason", CHATTER_EXAMPLES)
def test_classifies_chatter(transcript, expected_reason):
is_chatter, reason = classify_chatter(transcript)
assert is_chatter is True
assert reason == expected_reason
# ─────────────────────────────────────────────────────────────────────────────
# Must NOT classify as chatter — real events, including every transcript the
# review docs specifically named as dangerous to drop.
# ─────────────────────────────────────────────────────────────────────────────
REAL_EVENT_EXAMPLES = [
# The major "extinguishing fire" call (severity=major, tags=[extinguishing-fire])
("Dispatch, this is 7-4, extinguishing fire.", "extinguishing_fire"),
# Geocoded 911-hangup call (has location_coords)
(
"7, Charlie. Charlie, check and advise, we've got a call for service "
"coming over, it's going to be a 9-1-1 hangout, no voice contact. "
"Looks like it was an automated message saying it's the Doral Hat Company.",
"geocoded_911_hangup",
),
# Pursuit updates (severity=major, tags include pursuit / low-speed-pursuit)
("I'm aware of that one. It's a low-speed pursuit. It's refusing to pull over.", "low_speed_pursuit"),
(
"1. Headquarters to 5-charlie. I'm going to say the last thing to anyone.\n"
"2. Info, Sgt. Repeat.\n"
"3. The SP is on a pursuit southbound on I-684. It's approaching the airport.\n"
"4. Okay, thank you.\n5. 23-59.",
"pursuit_i684",
),
# "6 Alpha ... Pelham Station" subject check (CORRELATION_REVIEW_0907b.md's
# own "genuinely distinct events" list) — looks like a bare check-in but
# dispatches a unit to a specific location.
(
"6 Alpha, this is Central. 7 Alpha here.\n"
"6 Alpha, can you show me on scene at Pelham Station? Stand by.",
"pelham_station_subject_check",
),
# Property-retrieval call (tags=[property-retrieval])
(
"Property was retrieved with a 911. Can I get a phone number? 10-4. "
"Phone number is 214792. 214792.",
"property_retrieval",
),
# Subject check south of Maronex Station (tags=[subject-check])
(
"Proceed. Show me on a subject south of Maronex Station. Can I get a "
"15 check by New York client ID?",
"maronex_subject_check",
),
# Trespassing at milepost 13.7 (tags=[trespassing])
(
"Can you just 10-5 that job? You came over real muffled.\n"
"10-4, there's going to be a trespass on the tracks.\n"
"Train 8755 reports two juveniles, one male, one female, both wearing "
"white shirts, track three side, at milepost 13.7.",
"trespass_milepost_13_7",
),
# MVA (severity=moderate, tags=[traffic-accident])
("1. Train patrol 9.\n2. MVA 4, how close is it?\n3. 10-4.", "mva"),
]
@pytest.mark.parametrize("transcript,label", REAL_EVENT_EXAMPLES, ids=[l for _, l in REAL_EVENT_EXAMPLES])
def test_does_not_classify_real_events_as_chatter(transcript, label):
is_chatter, reason = classify_chatter(transcript)
assert is_chatter is False, f"{label}: false positive, reason={reason!r}"
assert reason is None
# ─────────────────────────────────────────────────────────────────────────────
# Edge cases
# ─────────────────────────────────────────────────────────────────────────────
def test_empty_transcript_not_chatter():
assert classify_chatter("") == (False, None)
assert classify_chatter(None) == (False, None)
assert classify_chatter(" ") == (False, None)
def test_unrecognized_content_defaults_to_not_chatter():
# Anything containing real descriptive words the classifier doesn't
# recognize must fall through to (False, None), not guess.
is_chatter, reason = classify_chatter("Shots fired, officer down, requesting backup immediately.")
assert is_chatter is False
assert reason is None
+430
View File
@@ -0,0 +1,430 @@
"""
server-26#115 — two consensus-quality fixes.
Fix 1 (routers/upload.py): when the cheap LLM says `orphan`, the rules engine
says `new`, and the call is genuinely SUBSTANCELESS (routine severity, no
vehicle/geocode/tag, and no incident already running on the same talkgroup),
resolve to `orphan` and DO NOT pay for the smart tiebreaker. Radio housekeeping
(unit check-ins, roll call, 10-8/10-98) was being promoted to incidents because
the tiebreaker rubber-stamped the rules `new` ~21/21 of the time
(CORRELATION_REVIEW_0907b.md).
The substance test runs against `ctx` (fully populated at preview time), NOT
against `rules_decision["corr_debug"]` — that dict is EMPTY at preview time for
action=="new" (corr_path:"new" is written at APPLY time), so the first version of
this gate fired on real events (a `major` "extinguishing fire", geocoded calls,
pursuit updates).
Fix 2 (incident_correlator.py): the `location` correlation path linked on a bare
sub-`location_proximity_km` (0.5 km) distance alone, taking whichever incident
came first in an unsorted `recent`. In a dense village two unrelated events
routinely geocode that close. A `location` link now needs unit overlap with the
candidate OR a distance under a tighter bar, and picks the NEAREST qualifying
candidate. A unit-overlap location link is tagged `location_unit_overlap` so it
does not merge into the fast path's bucket in the admin fit-signal histogram.
"""
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
import pytest
from app.routers import upload
from app.internal.incident_correlator import _run_decision, has_event_substance
NOW = datetime(2026, 9, 7, 21, 30, 0, tzinfo=timezone.utc)
# ─────────────────────────────────────────────────────────────────────────────
# Fix 1 — the LLM-orphan gate in _correlate_with_consensus
# ─────────────────────────────────────────────────────────────────────────────
def _preview(action, corr_debug=None, ctx=None):
base_ctx = {"call_id": "call-1"}
if ctx:
base_ctx.update(ctx)
return {
"decision": {
"action": action,
"matched_incident": None,
"incident_type": "other" if action == "new" else None,
"corr_debug": {} if corr_debug is None else dict(corr_debug),
},
"ctx": base_ctx,
}
def _llm(action, reasoning="—"):
md = {"incident_id": "inc-1"} if action == "link" else None
return {"action": action, "matched_incident": md, "reasoning": reasoning}
async def _run_consensus(preview, llm_decision):
tiebreak_result = {
"action": "new", "matched_incident": None, "incident_type": "other",
"corr_debug": {}, "reasoning": "tb",
}
with patch("app.internal.incident_correlator.preview_correlation",
new=AsyncMock(return_value=preview)), \
patch("app.internal.incident_correlator.apply_correlation",
new=AsyncMock(return_value="incident-x")) as m_apply, \
patch("app.internal.llm_correlator.decide",
new=AsyncMock(return_value=llm_decision)), \
patch("app.internal.llm_correlator.tiebreak",
new=AsyncMock(return_value=tiebreak_result)) as m_tiebreak:
await upload._correlate_with_consensus(
call_id="call-1", node_id="n1", system_id="sys-1",
talkgroup_id=9048, talkgroup_name="Dispatch", tags=[],
incident_type=None, location=None, location_coords=None,
)
return m_apply, m_tiebreak
async def test_substanceless_no_recent_same_tg_incident_gates_without_tiebreak():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}), _llm("orphan", "unit check-in, not an incident"),
)
m_tiebreak.assert_not_called()
m_apply.assert_called_once()
gated = m_apply.call_args[0][0]["decision"]
assert gated["action"] == "orphan"
dbg = gated["corr_debug"]
assert dbg["corr_consensus"] == "llm_orphan_gate"
assert dbg["corr_consensus"] != "tiebreak"
assert dbg["corr_rules_action"] == "new"
assert dbg["corr_llm_action"] == "orphan"
assert dbg["corr_llm_reasoning"] == "unit check-in, not an incident"
@pytest.mark.parametrize("severity", ["moderate", "major"])
async def test_moderate_or_major_severity_is_not_gated(severity):
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_severity": severity}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_routine_severity_alone_still_gates():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_severity": "routine"}), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_call_with_coords_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"coords": {"lat": 41.15, "lng": -73.86}}),
_llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_tags_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"tags": ["structure-fire"]}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_vehicles_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_vehicles": ["red sedan"]}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_resolved_incident_type_is_not_gated():
# The creation gate skips has_event_substance when a type resolved, so a
# typed call (fire/medical/…) opens an incident on substance the gate does
# not re-check — it must keep the tiebreak, not be dropped.
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"incident_type": "fire"}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_reassignment_call_is_not_gated():
# reassignment=True is dispatch pulling a unit onto a NEW job (units are
# blanked for exactly that reason) — the strongest new-incident signal.
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"reassignment": True}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_recent_incident_on_same_talkgroup_is_not_gated():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-live",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
"updated_at": (NOW - timedelta(minutes=1)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
# server-26#115 window #3 (CORRELATION_REVIEW_0912.md): the escape hatch used
# to treat ANY same-talkgroup incident inside the 2h correlation_window_hours
# as "recent", which on a busy dispatch channel (3-13 incidents/2h) was
# satisfied almost unconditionally — the gate fired 0/24 times against its own
# target shape. It now only counts an incident as recent within
# settings.tg_dispatch_thin_idle_minutes (5 min), applied uniformly regardless
# of the talkgroup's name (owner correction, 2026-09-13 — see
# test_channel_name_does_not_affect_the_window below for why the dichotomy
# this originally had with incident_correlator's fast/thin idle selection was
# removed here).
async def test_recent_same_tg_incident_inside_new_short_window_still_escapes_gate():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-live",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
# 3 min ago — inside tg_dispatch_thin_idle_minutes (5).
"updated_at": (NOW - timedelta(minutes=3)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_recent_same_tg_incident_older_than_short_window_now_gates():
# Regression test for the fix: 8 minutes is past the 5-minute bound but
# still inside the OLD 2-hour correlation_window_hours lookback. Before
# the fix this escaped the gate on any channel; after the fix it gates.
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-stale",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
"updated_at": (NOW - timedelta(minutes=8)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_channel_name_does_not_affect_the_window():
# Owner correction, 2026-09-13 (direct scanning experience): a talkgroup
# named "tac"/"tactical" only sees materially different traffic during a
# real incident, and that's rare -- the bulk of traffic on any monitored
# channel, including high-risk stops and pursuits, runs on the main
# channel regardless of what it's named. An earlier version of this used
# a longer 15-minute window on anything not literally named "dispatch"/
# "patched"/"primary" (mirroring incident_correlator's fast/thin idle
# selection); that meant a busy single-channel department not literally
# named "dispatch" silently got the more permissive window and could
# reproduce #115's original bug. Same 8-minute age as the dispatch-named
# test above, but on a channel named "Tac 3" -- must gate identically,
# not escape into a longer window just because of the name.
ctx = {
"system_id": "sys-1",
"talkgroup_id": 383,
"talkgroup_name": "Tac 3",
"now": NOW,
"recent": [{
"incident_id": "inc-tac",
"system_ids": ["sys-1"],
"talkgroup_ids": ["383"],
"updated_at": (NOW - timedelta(minutes=8)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_gate_veto_reason_is_recorded_on_the_escalation_path():
# server-26#115: a live measurement window must be able to see *why* an
# llm=orphan/rules=new call escaped the gate without guessing from the raw
# dump (which produced a wrong "confirmed explanation" for 2 window-#3
# misses the first time). corr_gate_veto names the surviving condition.
ctx = {"call_severity": "major"}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
final = m_apply.call_args[0][0]["decision"]
assert final["corr_debug"]["corr_gate_veto"] == "severity"
async def test_gate_veto_reason_is_absent_when_the_disagreement_is_not_orphan_vs_new():
# corr_gate_veto is only meaningful for the llm=orphan/rules=new shape the
# gate targets — it must not appear (or be misleadingly None-vs-absent) on
# an unrelated disagreement shape.
m_apply, m_tiebreak = await _run_consensus(
_preview("link", {}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
final = m_apply.call_args[0][0]["decision"]
assert "corr_gate_veto" not in final["corr_debug"]
async def test_recent_incident_on_a_different_talkgroup_still_gates():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"recent": [{
"incident_id": "inc-other",
"system_ids": ["sys-1"],
"talkgroup_ids": ["1200"],
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_llm_link_vs_rules_new_still_escalates():
m_apply, m_tiebreak = await _run_consensus(_preview("new", {}), _llm("link", "same job"))
m_tiebreak.assert_called_once()
async def test_llm_orphan_vs_rules_link_still_escalates():
# Not the gate condition (gate needs rules=="new"); must fall through.
m_apply, m_tiebreak = await _run_consensus(_preview("link", {}), _llm("orphan"))
m_tiebreak.assert_called_once()
def test_has_event_substance_predicate():
assert has_event_substance({"coords": {"lat": 1, "lng": 2}})
assert has_event_substance({"tags": ["fire"]})
assert has_event_substance({"call_vehicles": ["sedan"]})
assert not has_event_substance({})
assert not has_event_substance({"coords": None, "tags": [], "call_vehicles": []})
# units and location are NOT substance — nearly every transmission has them.
assert not has_event_substance({"call_units": ["7-Adam"], "location": "Main St"})
# ─────────────────────────────────────────────────────────────────────────────
# Fix 2 — tighten corr_path=location
# ─────────────────────────────────────────────────────────────────────────────
CALL_COORDS = {"lat": 41.150000, "lng": -73.860000}
# ~0.39 km north of the call — inside location_proximity_km (0.5) but well
# outside the tight bar (_LOCATION_TIGHT_PROXIMITY_KM, 0.2).
FAR_INC_COORDS = {"lat": 41.153500, "lng": -73.860000}
# ~0.13 km north of the call — inside the tight bar.
NEAR_INC_COORDS = {"lat": 41.151200, "lng": -73.860000}
# ~0.28 km north — inside the 0.5 radius, outside the 0.2 tight bar; used as a
# second candidate that must lose the nearest-wins sort to NEAR_INC_COORDS.
MID_INC_COORDS = {"lat": 41.152500, "lng": -73.860000}
def _inc(incident_id, coords, units):
return {
"incident_id": incident_id,
"system_ids": ["sys-1"],
"talkgroup_ids": ["100"], # different TGID → fast path is a no-op
"location_coords": coords,
"units": units,
"tags": [],
"type": "police",
"updated_at": (NOW - timedelta(minutes=6)).isoformat(),
"started_at": (NOW - timedelta(minutes=20)).isoformat(),
"status": "active",
"call_ids": ["c0"],
}
def _loc_ctx(*, incidents, call_units):
return {
"call_id": "call-loc",
"all_active": list(incidents),
"recent": list(incidents),
"call_doc": {},
"call_embedding": None,
"call_units": call_units,
"call_vehicles": [],
"call_cleared": [],
"call_severity": "routine",
"coords": CALL_COORDS,
"is_thin_call": False,
"now": NOW,
"system_id": "sys-1",
"talkgroup_id": 999, # not in inc.talkgroup_ids
"talkgroup_name": "Tactical",
"tags": [],
"incident_type": "police",
"location": "Main St",
"location_coords": CALL_COORDS,
"reassignment": True, # suppress the unit-continuity path
"create_if_new": True,
}
def test_location_path_in_radius_but_no_unit_overlap_no_tight_proximity_does_not_link(caplog):
ctx = _loc_ctx(
incidents=[_inc("inc-loc", FAR_INC_COORDS, ["7-Adam"])],
call_units=["3-Boy"],
)
with caplog.at_level("INFO", logger="drb-c2-core"):
decision = _run_decision(ctx)
# Reaches, and is rejected by, the new guard (not an earlier path).
assert "location-path skipped" in caplog.text
assert decision["action"] != "link"
assert (decision.get("corr_debug") or {}).get("corr_path") != "location"
def test_location_path_links_on_unit_overlap_with_distinct_fit_signal():
ctx = _loc_ctx(
incidents=[_inc("inc-loc", FAR_INC_COORDS, ["5-Adam"])],
call_units=["5-Adam"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "location"
# NOT "unit_overlap" — that value belongs to the fast path's histogram bucket.
assert decision["corr_debug"]["corr_fit_signal"] == "location_unit_overlap"
def test_location_path_links_on_tight_proximity_without_unit_overlap():
ctx = _loc_ctx(
incidents=[_inc("inc-loc", NEAR_INC_COORDS, ["7-Adam"])],
call_units=["3-Boy"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "location"
assert decision["corr_debug"]["corr_fit_signal"] == "location_proximity"
def test_location_path_picks_nearest_in_radius_candidate():
# `recent` order puts the farther tight-proximity incident first; the guard
# must still select the nearest one.
ctx = _loc_ctx(
incidents=[
_inc("inc-mid", MID_INC_COORDS, ["3-Boy"]), # ~0.28 km, tight-fail
_inc("inc-near", NEAR_INC_COORDS, ["3-Boy"]), # ~0.13 km, tight-pass
],
call_units=["3-Boy"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-near"
assert decision["corr_debug"]["corr_path"] == "location"
+29 -41
View File
@@ -31,8 +31,10 @@ from app.internal.incident_correlator import (
NOW = datetime(2026, 8, 20, 7, 0, 0, tzinfo=timezone.utc) NOW = datetime(2026, 8, 20, 7, 0, 0, tzinfo=timezone.utc)
# TG 383 from the dump: "Ch 1 (Patched with 155.310)". _DISPATCH_TG_RE matches # TG 383 from the dump: "Ch 1 (Patched with 155.310)", a shared dispatch
# "patched", so this is a shared dispatch backbone carrying the whole department. # backbone carrying the whole department. Kept as two distinct fixture names
# for readability even though the channel's name no longer affects behavior
# (server-26#134).
DISPATCH_TG = "Ch 1 (Patched with 155.310)" DISPATCH_TG = "Ch 1 (Patched with 155.310)"
TACTICAL_TG = "Fireground 2" TACTICAL_TG = "Fireground 2"
@@ -140,12 +142,9 @@ def test_thin_call_with_no_overlap_does_not_attach_on_a_dispatch_channel():
assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan" assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan"
def test_thin_call_with_no_overlap_does_not_attach_on_a_tactical_channel(): def test_thin_call_with_no_overlap_does_not_attach_on_a_tactical_named_channel():
""" """A channel's name no longer changes anything (server-26#134) — same
The widest version of the bug: non-dispatch talkgroups skipped the tiering assertion as the dispatch-named case above, different fixture name."""
entirely and used the whole 90-minute fast-path window with no
single-candidate requirement, so ANY thin call joined whatever was newest.
"""
inc = _incident(idle_minutes=40) inc = _incident(idle_minutes=40)
decision = _run_decision(_ctx( decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG, all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
@@ -153,9 +152,18 @@ def test_thin_call_with_no_overlap_does_not_attach_on_a_tactical_channel():
assert decision["action"] == "orphan" assert decision["action"] == "orphan"
def test_tactical_thin_call_still_attaches_inside_its_own_window(): def test_tactical_named_channel_uses_the_dispatch_window_now():
"""Bounded, not removed — a "10-4" on a working channel is still context.""" """server-26#134: 14 min was inside the old 15-min tactical window; now
inc = _incident(idle_minutes=settings.tg_thin_idle_minutes - 1) every channel uses the 5-min window regardless of name."""
inc = _incident(idle_minutes=14)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
def test_tactical_named_channel_still_attaches_inside_the_dispatch_window():
inc = _incident(idle_minutes=settings.tg_dispatch_thin_idle_minutes - 1)
decision = _run_decision(_ctx( decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG, all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
)) ))
@@ -224,47 +232,27 @@ def test_back_dated_thin_call_does_not_sail_through_the_recency_gate():
def test_back_dated_call_does_not_bypass_the_content_divergence_veto(monkeypatch): def test_back_dated_call_does_not_bypass_the_content_divergence_veto(monkeypatch):
""" """
Same `9d376ffe` failure mode, but exercised directly against Same `9d376ffe` failure mode, exercised directly against
`_call_fits_incident` on a dispatch channel: unit overlap plus a `_call_fits_incident`: unit overlap plus a back-dated call (incident
back-dated call (incident updated 45 minutes AFTER the call's own updated 45 minutes AFTER the call's own `started_at`, which the sweep
`started_at`, which the sweep passes as `now`) used to make the signed passes as `now`) used to make the signed idle -45, so `idle_min >= 15`
idle -45, so `idle_min >= 15` read False and the content-divergence read False and the content-divergence veto never ran — unit overlap
veto never ran — unit overlap alone forced the merge regardless of alone forced the merge regardless of what the call was actually about.
what the call was actually about. With the gate fixed to compare With the gate fixed to compare distance, idle_min is 45 (>= 15), the
distance, idle_min is 45 (>= 15), the veto runs, and a divergent veto runs, and a divergent embedding (patched below so the assertion
embedding (patched below so the assertion doesn't depend on numpy doesn't depend on numpy being installed in this environment) fails it.
being installed in this environment) fails it.
""" """
monkeypatch.setattr(correlator_mod, "_cosine_similarity", lambda a, b: 0.0) monkeypatch.setattr(correlator_mod, "_cosine_similarity", lambda a, b: 0.0)
inc = _incident(idle_minutes=-45, units=["6-Adam"]) inc = _incident(idle_minutes=-45, units=["6-Adam"])
inc["embedding"] = [1.0, 0.0] inc["embedding"] = [1.0, 0.0]
fits, signal = _call_fits_incident( fits, signal = _call_fits_incident(
inc, call_units=["6-Adam"], call_vehicles=[], call_coords=None, inc, call_units=["6-Adam"], call_vehicles=[], call_coords=None,
proximity_km=settings.location_proximity_km, is_dispatch=True, proximity_km=settings.location_proximity_km,
call_embedding=[0.0, 1.0], now=NOW, call_embedding=[0.0, 1.0], now=NOW,
) )
assert (fits, signal) == (False, "content_divergence") assert (fits, signal) == (False, "content_divergence")
def test_back_dated_call_on_tactical_channel_does_not_get_the_default():
"""
Tactical-channel counterpart: no unit/vehicle/location signal, so the
function falls through to step 4's `idle_min < 20.0` default. A
back-dated call (incident updated 45 minutes after the call's own
started_at) used to read idle_min as -45, which is always < 20.0, so
`tactical_default` fired unconditionally no matter how stale the
incident actually was relative to this call. Fixed, idle_min is the
45-minute distance, which is not < 20.0.
"""
inc = _incident(idle_minutes=-45)
fits, signal = _call_fits_incident(
inc, call_units=[], call_vehicles=[], call_coords=None,
proximity_km=settings.location_proximity_km, is_dispatch=False,
call_embedding=None, now=NOW,
)
assert (fits, signal) == (False, "tactical_idle")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 4. Hard caps — path-independent, because pairwise fit tests can't see shape # 4. Hard caps — path-independent, because pairwise fit tests can't see shape
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+66
View File
@@ -0,0 +1,66 @@
"""
End-to-end CORS wiring for the one browser-facing REST surface.
The frontend's Archive page calls GET /calls/search with Authorization +
Content-Type headers, which forces the browser to send a CORS preflight
first. Before #110 that OPTIONS got a bare 405 with no Access-Control-*
headers and the fetch failed with "TypeError: Failed to fetch". These
tests drive the real app through TestClient so a regression in the
middleware wiring (not just the helper) is caught.
TestClient is NOT used as a context manager on purpose: that would run the
lifespan (mqtt_handler.connect(), the sweeper loops, dynsec bootstrap),
none of which is needed here -- CORSMiddleware answers a preflight before
routing or dependencies run.
"""
from fastapi.testclient import TestClient
from app.config import settings
from app.main import app
client = TestClient(app)
ALLOWED_ORIGIN = "https://drb.cusano.net"
DISALLOWED_ORIGIN = "https://evil.example.com"
def test_default_allowed_origin_matches_the_deployed_frontend():
# The frontend is served on the bare domain (infra Caddyfile.j2), so the
# default must allow exactly that origin without any env override.
assert ALLOWED_ORIGIN in settings.cors_origins
def test_preflight_for_calls_search_is_allowed():
resp = client.options(
"/calls/search",
headers={
"Origin": ALLOWED_ORIGIN,
"Access-Control-Request-Method": "GET",
"Access-Control-Request-Headers": "authorization,content-type",
},
)
assert resp.status_code == 200
assert resp.headers.get("access-control-allow-origin") == ALLOWED_ORIGIN
allow_methods = resp.headers.get("access-control-allow-methods", "").upper()
assert "GET" in allow_methods
# Bearer auth, not cookies -- credentials must never be advertised.
assert "access-control-allow-credentials" not in resp.headers
def test_preflight_from_disallowed_origin_gets_no_allow_origin():
resp = client.options(
"/calls/search",
headers={
"Origin": DISALLOWED_ORIGIN,
"Access-Control-Request-Method": "GET",
},
)
assert resp.headers.get("access-control-allow-origin") is None
def test_simple_get_from_allowed_origin_is_annotated():
# Even a non-preflight GET must carry Access-Control-Allow-Origin or the
# browser hides the response body from the page.
resp = client.get("/health", headers={"Origin": ALLOWED_ORIGIN})
assert resp.status_code == 200
assert resp.headers.get("access-control-allow-origin") == ALLOWED_ORIGIN
+9 -7
View File
@@ -5,8 +5,9 @@ Starlette does not reject `allow_origins=["*"]` combined with
`allow_credentials=True`. It reflects the caller's Origin back in `allow_credentials=True`. It reflects the caller's Origin back in
Access-Control-Allow-Origin and still sends Access-Control-Allow-Origin and still sends
Access-Control-Allow-Credentials: true, so the effective policy is the Access-Control-Allow-Credentials: true, so the effective policy is the
opposite of what a wildcard usually means. main.py defuses that by turning opposite of what a wildcard usually means. main.py never enables
credentials off whenever it sees a wildcard; these tests hold it to that. credentials at all (auth is a Bearer header, not a cookie), which makes
that pair unrepresentable; these tests hold it to that.
The policy lives in a pure function so it can be exercised directly -- The policy lives in a pure function so it can be exercised directly --
reloading app.main to vary settings drags every router back through import reloading app.main to vary settings drags every router back through import
@@ -28,11 +29,11 @@ def test_wildcard_among_real_origins_still_disables_credentials():
assert cors_allows_credentials(["https://app.example.com", "*"]) is False assert cors_allows_credentials(["https://app.example.com", "*"]) is False
def test_named_origins_keep_credentials(): def test_credentials_never_enabled_even_for_named_origins():
# Naming your origins is how you ask for credentialed requests, so a # Auth here is a Bearer header, not a cookie, so credentialed CORS is
# correctly configured deployment must not be penalised. # never needed. The predicate is hard-off regardless of the origin list.
assert cors_allows_credentials(["https://app.example.com"]) is True assert cors_allows_credentials(["https://app.example.com"]) is False
assert cors_allows_credentials([]) is True assert cors_allows_credentials([]) is False
def test_the_app_actually_mounted_that_policy(): def test_the_app_actually_mounted_that_policy():
@@ -42,6 +43,7 @@ def test_the_app_actually_mounted_that_policy():
(mw.kwargs for mw in app.user_middleware if mw.cls is CORSMiddleware), None (mw.kwargs for mw in app.user_middleware if mw.cls is CORSMiddleware), None
) )
assert opts is not None, "CORSMiddleware is not mounted at all" assert opts is not None, "CORSMiddleware is not mounted at all"
assert opts["allow_credentials"] is False
assert opts["allow_credentials"] is cors_allows_credentials(settings.cors_origins) assert opts["allow_credentials"] is cors_allows_credentials(settings.cors_origins)
@@ -0,0 +1,197 @@
"""
server-26#96 — every scene of a multi-scene call writes corr_debug onto the
SAME call doc via incident_correlator._apply_and_log, last-scene-wins. The
fix additionally nests each scene's corr_debug/transcript/incident_id under
scenes.<scene_index> on the call doc, keyed so Firestore's
`set(merge=True)` (a recursive merge of nested map fields — this is the
behaviour these tests assume and pin) lands each scene in its own map entry
instead of colliding.
Firestore itself isn't available in this sandbox (see tests/conftest.py), so
`_fake_doc_set` below implements that documented recursive-merge semantics by
hand and is used as the fstore stand-in — these tests both exercise
_apply_and_log's write shape AND pin the merge behaviour it depends on.
"""
import pytest
from unittest.mock import patch
from app.internal import incident_correlator
def _merge(dst: dict, src: dict) -> None:
"""Firestore DocumentReference.set(data, merge=True) semantics: nested
map fields are merged recursively by key, not replaced wholesale."""
for k, v in src.items():
if isinstance(v, dict) and isinstance(dst.get(k), dict):
_merge(dst[k], v)
else:
dst[k] = v
@pytest.mark.asyncio
async def test_multiscene_call_lands_each_scene_distinctly_and_flat_fields_last_write_wins():
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision0 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
}
ctx0 = {"call_id": "call-1", "scene_index": 0, "scene_transcript": "scene zero text"}
decision1 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "slow", "corr_consensus": "tiebreak"},
}
ctx1 = {"call_id": "call-1", "scene_index": 1, "scene_transcript": "scene one text"}
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision0, ctx0)
await incident_correlator._apply_and_log(decision1, ctx1)
doc = docs[("calls", "call-1")]
# Flat top-level fields: unchanged behaviour, last scene's write wins —
# the safe backward-compatible default for any reader that doesn't yet
# know about `scenes`.
assert doc["corr_path"] == "slow"
assert doc["corr_consensus"] == "tiebreak"
# New `scenes` map: both scenes present, distinct, uncorrupted by the
# second write.
assert set(doc["scenes"].keys()) == {"0", "1"}
assert doc["scenes"]["0"]["corr_debug"]["corr_path"] == "new"
assert doc["scenes"]["0"]["corr_debug"]["corr_consensus"] == "agreed"
assert doc["scenes"]["0"]["transcript"] == "scene zero text"
assert doc["scenes"]["1"]["corr_debug"]["corr_path"] == "slow"
assert doc["scenes"]["1"]["corr_debug"]["corr_consensus"] == "tiebreak"
assert doc["scenes"]["1"]["transcript"] == "scene one text"
@pytest.mark.asyncio
async def test_scene_entry_records_which_incident_it_resolved_to():
"""summarizer.py (#114) needs this to pick the right scene per incident."""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
with patch.object(incident_correlator, "fstore") as mock_fstore, \
patch.object(incident_correlator, "_apply_decision", return_value="inc-42"):
mock_fstore.doc_set = fake_doc_set
decision = {
"action": "new", "matched_incident": None, "incident_type": "fire",
"corr_debug": {"corr_path": "new"},
}
ctx = {"call_id": "call-2", "scene_index": 0, "scene_transcript": "structure fire"}
incident_id = await incident_correlator._apply_and_log(decision, ctx)
assert incident_id == "inc-42"
assert docs[("calls", "call-2")]["scenes"]["0"]["incident_id"] == "inc-42"
@pytest.mark.asyncio
async def test_single_scene_call_still_gets_a_scenes_map_equivalent_to_flat_fields():
"""scene_index defaults to 0 for every caller with no scene concept, so a
plain single-scene call is one entry in `scenes` — equivalent to reading
the flat fields, not a behaviour change for that population."""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "fast/thin", "corr_consensus": "rules_only"},
}
ctx = {"call_id": "call-3", "scene_transcript": "10-4"} # no scene_index key at all
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision, ctx)
doc = docs[("calls", "call-3")]
assert doc["corr_path"] == "fast/thin"
assert doc["scenes"] == {
"0": {
"transcript": "10-4",
"incident_id": None,
"corr_debug": {"corr_path": "fast/thin", "corr_consensus": "rules_only"},
"incident_type": None,
"severity": None,
}
}
@pytest.mark.asyncio
async def test_scene_entry_captures_its_own_incident_type_not_a_sibling_scenes():
"""
server-26#139: _call_is_substanceless's "type" veto reads ctx["incident_type"]
at decision time, but that value was never persisted per-scene — only the
last-scene-wins flat field, which #138's dump analysis couldn't
distinguish from cross-scene contamination. Pins _apply_and_log's write
side: each scene's own scenes.<n> entry carries its own incident_type/
severity, distinct from any other scene on the same call. Does NOT cover
whether the ctx handed to _call_is_substanceless is the same object that
reaches here — that linkage is pinned by test_consensus_gate.py and
test_incident_identity.py, not this file.
"""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision0 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "tiebreak", "corr_gate_veto": "type"},
}
ctx0 = {
"call_id": "call-5", "scene_index": 0, "scene_transcript": "10-4, clear",
"incident_type": "traffic-stop", "call_severity": "routine",
}
decision1 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
}
ctx1 = {
"call_id": "call-5", "scene_index": 1, "scene_transcript": "roll call",
"incident_type": None, "call_severity": "moderate",
}
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision0, ctx0)
await incident_correlator._apply_and_log(decision1, ctx1)
doc = docs[("calls", "call-5")]
scenes = doc["scenes"]
assert scenes["0"]["incident_type"] == "traffic-stop"
assert scenes["0"]["severity"] == "routine"
assert scenes["1"]["incident_type"] is None
assert scenes["1"]["severity"] == "moderate"
# _apply_and_log only ever flat-merges corr_debug's own keys (:1460) — a
# future corr_debug["incident_type"] would silently clobber
# intelligence.py's flat field, so this is asserted, not just commented.
assert "incident_type" not in doc
@pytest.mark.asyncio
async def test_empty_corr_debug_writes_nothing_same_as_before():
"""Preserve the pre-#96 short-circuit: no corr_debug means no write at
all, flat or nested."""
with patch.object(incident_correlator, "fstore") as mock_fstore, \
patch.object(incident_correlator, "_apply_decision", return_value=None):
mock_fstore.doc_set = None # would raise TypeError if ever called
decision = {"action": "orphan", "matched_incident": None, "incident_type": None, "corr_debug": {}}
ctx = {"call_id": "call-4", "scene_index": 0, "scene_transcript": "x"}
result = await incident_correlator._apply_and_log(decision, ctx)
assert result is None
@@ -0,0 +1,97 @@
"""
server-26#131 — the re-correlation sweep's orphan filter checked incident_id/
incident_ids/corr_path but had no way to tell "never processed" apart from
"real-time pipeline (routers/upload.py _run_intelligence_pipeline) is still
mid-flight". Racing the sweep against an in-flight real-time correlation could
land the same call on two different incidents — the exact duplicate-link bug
#131 found in 3 live dumps (~2% of linked calls). This pins the fix: a call
whose intelligence_started_at marker is recent is held back from the sweep
regardless of how orphaned it otherwise looks.
"""
from datetime import datetime, timezone, timedelta
from unittest.mock import patch
import pytest
from app.internal import recorrelation_sweep
def _iso(dt: datetime) -> str:
return dt.isoformat()
class TestPipelineLikelyStillRunning:
def test_recent_marker_is_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=1))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is True
def test_old_marker_is_not_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=30))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_marker_exactly_at_the_threshold_is_not_held_back(self):
# age_minutes < MIN_MINUTES_SINCE_PIPELINE_START (strict), so exactly
# at the threshold is old enough to release — pins the boundary so it
# can't drift to <= by accident and silently double the hold time.
now = datetime.now(timezone.utc)
threshold = recorrelation_sweep.MIN_MINUTES_SINCE_PIPELINE_START
call = {"intelligence_started_at": _iso(now - timedelta(minutes=threshold))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_no_marker_at_all_is_not_held_back(self):
"""A pre-#131 call doc, or the marker write itself failed — absence
isn't evidence of an in-flight pipeline, so the sweep must still be
able to pick these up (that's its whole job)."""
now = datetime.now(timezone.utc)
assert recorrelation_sweep._pipeline_likely_still_running({}, now) is False
def test_unparseable_marker_is_not_held_back(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": "not-a-timestamp"}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
@pytest.mark.asyncio
async def test_sweep_pass_skips_a_call_whose_pipeline_just_started():
"""Integration-shaped: a call that looks orphaned by every OTHER filter
(no incident_ids, no corr_path, no skip_reason, under the attempt budget)
but has a fresh intelligence_started_at must not reach correlate_call —
that's the race #131 found."""
now = datetime.now(timezone.utc)
racing_call = {
"call_id": "call-racing",
"started_at": _iso(now - timedelta(minutes=2)),
"ended_at": _iso(now - timedelta(minutes=1)),
"intelligence_started_at": _iso(now - timedelta(seconds=30)),
}
genuinely_orphaned_call = {
"call_id": "call-genuine-orphan",
"started_at": _iso(now - timedelta(minutes=20)),
"ended_at": _iso(now - timedelta(minutes=19)),
"intelligence_started_at": _iso(now - timedelta(minutes=19)),
}
async def fake_collection_where(collection, clauses):
assert collection == "calls"
return [racing_call, genuinely_orphaned_call]
correlate_calls: list[str] = []
async def fake_correlate_call(**kwargs):
correlate_calls.append(kwargs["call_id"])
return None # no match — exercises the "not linked" branch too
doc_sets: list[tuple] = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch.object(recorrelation_sweep, "fstore") as mock_fstore, \
patch("app.internal.incident_correlator.correlate_call", fake_correlate_call):
mock_fstore.collection_where = fake_collection_where
mock_fstore.doc_set = fake_doc_set
await recorrelation_sweep._run_sweep_pass()
assert correlate_calls == ["call-genuine-orphan"]
@@ -0,0 +1,89 @@
"""
server-26#96/#114 review (PR #132): `PATCH /calls/{id}/transcript` clears
stale intelligence fields before re-extraction runs, but `doc_set(...,
merge=True)` can only add/overwrite keys in a nested map, never remove one.
A call corrected from 3 scenes down to 1 would keep `scenes.1`/`scenes.2`
with pre-correction transcripts and incident_ids forever -- corrupting the
per-scene tally #96 exists to make trustworthy, and re-feeding stale text
into #114's summarizer fix if a stale scene's incident_id still names a real
incident. The fix deletes the field with `fstore.DELETE_FIELD` instead of
merging over it with an empty map (which is a no-op).
"""
from unittest.mock import AsyncMock, patch
import pytest
from fastapi import BackgroundTasks
from app.internal import firestore as fstore
from app.routers.calls import TranscriptUpdate, patch_transcript
@pytest.mark.asyncio
async def test_transcript_correction_deletes_the_scenes_field_not_merges_over_it():
call = {
"call_id": "call-1",
"system_id": "sys-1",
"node_id": "node-1",
"transcript": "old raw text",
# Simulates a prior 3-scene call, per #96's schema.
"scenes": {
"0": {"transcript": "scene zero", "incident_id": "inc-a", "corr_debug": {}},
"1": {"transcript": "scene one", "incident_id": "inc-b", "corr_debug": {}},
},
}
doc_set_calls: list[tuple] = []
doc_update_calls: list[tuple] = []
async def fake_doc_get(collection, doc_id):
if collection == "calls" and doc_id == "call-1":
return call
return None
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_set_calls.append((collection, doc_id, data))
async def fake_doc_update(collection, doc_id, data):
doc_update_calls.append((collection, doc_id, data))
fake_flags = (None, lambda name: name == "correlation_enabled")
with patch("app.routers.calls.fstore.doc_get", new=fake_doc_get), \
patch("app.routers.calls.fstore.doc_set", new=fake_doc_set), \
patch("app.routers.calls.fstore.doc_update", new=fake_doc_update), \
patch("app.internal.feature_flags.resolve_flags", new=AsyncMock(return_value=fake_flags)):
result = await patch_transcript(
call_id="call-1",
body=TranscriptUpdate(transcript="corrected text"),
background_tasks=BackgroundTasks(),
_={},
)
assert result == {"ok": True, "call_id": "call-1"}
# The stale scenes map must be DELETED, not merged over with {} (a no-op
# under Firestore's set(merge=True) semantics) and not left untouched by
# a doc_set call that never mentions it.
scenes_deletions = [
(coll, doc_id, data) for (coll, doc_id, data) in doc_update_calls
if coll == "calls" and doc_id == "call-1" and "scenes" in data
]
assert len(scenes_deletions) == 1, (
f"expected exactly one doc_update clearing 'scenes', got {doc_update_calls}"
)
assert scenes_deletions[0][2]["scenes"] is fstore.DELETE_FIELD
# And no doc_set call should paper over the same field with an empty map
# instead -- that would silently do nothing and leave stale scenes intact.
for (coll, doc_id, data) in doc_set_calls:
if coll == "calls" and doc_id == "call-1":
assert "scenes" not in data, (
"a doc_set (merge=True) write must never carry 'scenes' -- "
"merging {} over an existing map is a no-op, not a delete"
)
def test_delete_field_is_the_real_firestore_sentinel():
"""Catches an import-path typo turning this into a silent no-op sentinel."""
from firebase_admin import firestore as fs
assert fstore.DELETE_FIELD is fs.DELETE_FIELD
@@ -0,0 +1,124 @@
"""
server-26#114 — the incident summarizer used to read doc["transcript"] (the
WHOLE call, raw) for every linked call, so a multi-scene call contributed
text from scenes it wasn't part of into an incident's summary, and
transcript_corrected was never consulted at all.
Fix: _scene_text_for_incident reads the server-26#96 `scenes` map to find the
scene(s) that actually resolved into a given incident_id, and falls back to
transcript_corrected-or-transcript for a call doc with no `scenes` field
(predates #96).
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import summarizer
from app.internal.summarizer import _scene_text_for_incident
# ---------------------------------------------------------------------------
# _scene_text_for_incident — pure function, no Firestore
# ---------------------------------------------------------------------------
def test_picks_the_scene_that_linked_to_this_incident():
doc = {
"transcript": "whole raw transcript blend",
"transcript_corrected": "whole corrected transcript blend",
"scenes": {
"0": {"transcript": "scene zero text", "incident_id": "inc-A", "corr_debug": {}},
"1": {"transcript": "scene one text", "incident_id": "inc-B", "corr_debug": {}},
},
}
assert _scene_text_for_incident(doc, "inc-A") == "scene zero text"
assert _scene_text_for_incident(doc, "inc-B") == "scene one text"
def test_joins_multiple_scenes_linked_to_the_same_incident_in_scene_order():
doc = {
"scenes": {
"1": {"transcript": "second", "incident_id": "inc-A"},
"0": {"transcript": "first", "incident_id": "inc-A"},
},
}
assert _scene_text_for_incident(doc, "inc-A") == "first\nsecond"
def test_old_schema_doc_falls_back_to_transcript_corrected_over_transcript():
doc = {"transcript": "raw", "transcript_corrected": "corrected"}
assert _scene_text_for_incident(doc, "inc-A") == "corrected"
def test_old_schema_doc_with_only_raw_transcript_still_returns_it():
doc = {"transcript": "raw only"}
assert _scene_text_for_incident(doc, "inc-A") == "raw only"
def test_scenes_present_but_none_match_falls_back_defensively():
"""Should not happen for a call_id genuinely in this incident's call_ids,
but silently dropping the call's contribution would be worse than a
whole-call fallback."""
doc = {
"transcript": "raw",
"transcript_corrected": "corrected",
"scenes": {"0": {"transcript": "x", "incident_id": "inc-OTHER"}},
}
assert _scene_text_for_incident(doc, "inc-A") == "corrected"
# ---------------------------------------------------------------------------
# _summarize_incident — end to end with fstore/Gemini mocked
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_summarize_incident_uses_scene_specific_text_for_a_multiscene_call():
"""
call-1 is a 2-scene call: scene 0 linked into inc-OTHER, scene 1 linked
into inc-1 (the incident being summarized). Only scene 1's text may reach
the model.
"""
call_1 = {
"call_id": "call-1",
"transcript": "scene zero text scene one text", # the old, wrong, whole-call blend
"scenes": {
"0": {"transcript": "scene zero text", "incident_id": "inc-OTHER"},
"1": {"transcript": "scene one text", "incident_id": "inc-1"},
},
}
async def fake_doc_get(collection, doc_id):
assert collection == "calls"
return call_1 if doc_id == "call-1" else None
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value={"summaries_enabled": True})), \
patch.object(summarizer, "fstore") as fs, \
patch.object(summarizer, "_sync_summarize", return_value="a summary") as sync:
fs.doc_get = AsyncMock(side_effect=fake_doc_get)
fs.doc_set = AsyncMock()
await summarizer._summarize_incident({"incident_id": "inc-1", "call_ids": ["call-1"]})
sync.assert_called_once()
_inc_arg, transcripts_arg = sync.call_args.args
assert transcripts_arg == ["scene one text"]
assert "scene zero text scene one text" not in transcripts_arg
@pytest.mark.asyncio
async def test_summarize_incident_falls_back_for_old_schema_call_doc():
"""A call doc with no `scenes` field at all — summarizer must still work,
using transcript_corrected over raw transcript."""
call_1 = {"call_id": "call-1", "transcript": "raw", "transcript_corrected": "corrected"}
async def fake_doc_get(collection, doc_id):
return call_1 if doc_id == "call-1" else None
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value={"summaries_enabled": True})), \
patch.object(summarizer, "fstore") as fs, \
patch.object(summarizer, "_sync_summarize", return_value="a summary") as sync:
fs.doc_get = AsyncMock(side_effect=fake_doc_get)
fs.doc_set = AsyncMock()
await summarizer._summarize_incident({"incident_id": "inc-1", "call_ids": ["call-1"]})
_inc_arg, transcripts_arg = sync.call_args.args
assert transcripts_arg == ["corrected"]
+3 -3
View File
@@ -1,6 +1,6 @@
"use client"; "use client";
import { useState } from "react"; import { useEffect, useState } from "react";
import { useAuth } from "@/components/AuthProvider"; import { useAuth } from "@/components/AuthProvider";
import { useAlerts } from "@/lib/useAlerts"; import { useAlerts } from "@/lib/useAlerts";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice"; import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
@@ -32,8 +32,8 @@ function RulesTab({ isAdmin }: { isAdmin: boolean }) {
} }
} }
// Load on first render of this tab // Load once when this tab mounts (load() self-guards on `loaded`).
if (!loaded) { load(); } useEffect(() => { load(); }, []);
async function handleCreate(e: React.FormEvent) { async function handleCreate(e: React.FormEvent) {
e.preventDefault(); e.preventDefault();
+4 -1
View File
@@ -120,7 +120,10 @@ export default function NodeDetailPage() {
const [approving, setApproving] = useState(false); const [approving, setApproving] = useState(false);
const [deleting, setDeleting] = useState(false); const [deleting, setDeleting] = useState(false);
const { systems } = useSystems(); const { systems } = useSystems();
const { calls } = useCalls(20); // TODO(server-26#109 item5): server-side node_id filter. A where("node_id","==",id)
// alongside the existing org_id equality + started_at orderBy needs a brand-new
// composite index, so for now pull a wider window and filter client-side.
const { calls } = useCalls(200);
const { isAdmin } = useAuth(); const { isAdmin } = useAuth();
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s])); const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
+16 -29
View File
@@ -25,15 +25,15 @@ L.Icon.Default.mergeOptions({
}); });
// ── Basemap tiles ───────────────────────────────────────────────────────────── // ── Basemap tiles ─────────────────────────────────────────────────────────────
// Default is CARTO's keyless dark raster basemap — no token, fits the dark UI. // Prod sets NEXT_PUBLIC_MAP_TILE_URL to a keyed style (a CARTO account style,
// Overridable via NEXT_PUBLIC_MAP_TILE_URL so a keyed style (a CARTO account // MapTiler, Mapbox, …). The in-code fallback is plain OpenStreetMap so the map
// style, MapTiler, Mapbox, …) can be dropped in for prod without a code change. // still renders if that var is missing — CARTO's keyless CDN has proven flaky.
// Whatever is supplied must use Leaflet's {s}/{z}/{x}/{y}{r} placeholder scheme. // Whatever is supplied must use Leaflet's {s}/{z}/{x}/{y}{r} placeholder scheme;
// the {z}/{x}/{y} tokens below are substituted by Leaflet at runtime.
const MAP_TILE_URL = const MAP_TILE_URL =
process.env.NEXT_PUBLIC_MAP_TILE_URL || process.env.NEXT_PUBLIC_MAP_TILE_URL ||
"https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png"; "https://tile.openstreetmap.org/{z}/{x}/{y}.png";
const MAP_TILE_ATTRIBUTION = const MAP_TILE_ATTRIBUTION = "&copy; OpenStreetMap contributors";
'&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors &copy; <a href="https://carto.com/">CARTO</a>';
// ── Colour ──────────────────────────────────────────────────────────────────── // ── Colour ────────────────────────────────────────────────────────────────────
// Severity is the only hue on this map — see UI_REDESIGN.md §2.3. Incident // Severity is the only hue on this map — see UI_REDESIGN.md §2.3. Incident
@@ -459,9 +459,6 @@ export default function MapView({ nodes, activeCalls, incidents = [], calls = []
const [drawerOpen, setDrawerOpen] = useState(false); const [drawerOpen, setDrawerOpen] = useState(false);
const [agoClock, setAgoClock] = useState(0); const [agoClock, setAgoClock] = useState(0);
const [radarEpoch, setRadarEpoch] = useState(() => Date.now()); const [radarEpoch, setRadarEpoch] = useState(() => Date.now());
const [clockStr, setClockStr] = useState(() =>
new Date().toLocaleTimeString([], { hour12: false, hour: "2-digit", minute: "2-digit", second: "2-digit" })
);
useEffect(() => { useEffect(() => {
const id = setInterval(() => setAgoClock((t: number) => t + 1), 10_000); const id = setInterval(() => setAgoClock((t: number) => t + 1), 10_000);
@@ -474,15 +471,6 @@ export default function MapView({ nodes, activeCalls, incidents = [], calls = []
return () => clearInterval(id); return () => clearInterval(id);
}, []); }, []);
// Live clock for TOC situational awareness
useEffect(() => {
const id = setInterval(() =>
setClockStr(new Date().toLocaleTimeString([], { hour12: false, hour: "2-digit", minute: "2-digit", second: "2-digit" })),
1000
);
return () => clearInterval(id);
}, []);
// eslint-disable-next-line react-hooks/exhaustive-deps // eslint-disable-next-line react-hooks/exhaustive-deps
const ago = useMemo(() => (lastUpdated ? timeAgo(lastUpdated) : null), [lastUpdated, agoClock]); const ago = useMemo(() => (lastUpdated ? timeAgo(lastUpdated) : null), [lastUpdated, agoClock]);
@@ -623,13 +611,8 @@ export default function MapView({ nodes, activeCalls, incidents = [], calls = []
)} )}
</div> </div>
{/* ── Clock — bottom-left for TOC situational awareness ───────────────── */}
<div className="absolute bottom-8 left-3 z-[1001] bg-surface/90 border border-line rounded-lg px-3 py-2 pointer-events-none">
<span className="text-ink text-sm font-mono tabular-nums">{clockStr}</span>
</div>
{/* ── Legend — shape-first, both themes. Never a bare colour swatch. ──── */} {/* ── Legend — shape-first, both themes. Never a bare colour swatch. ──── */}
<div className="absolute bottom-8 right-3 z-[1001] bg-surface/90 border border-line rounded-lg px-3 py-2.5 text-xs pointer-events-none space-y-2"> <div className="absolute bottom-8 right-3 z-[1001] bg-surface/90 border border-line rounded-lg px-3 py-2.5 text-xs pointer-events-none space-y-2 max-h-[calc(100%-4rem)] overflow-y-auto">
<div className="space-y-1"> <div className="space-y-1">
<p className="text-ink-muted font-medium text-[10px] uppercase tracking-wide">Severity</p> <p className="text-ink-muted font-medium text-[10px] uppercase tracking-wide">Severity</p>
{(["major", "moderate", "minor", "routine"] as Severity[]).map((sev) => ( {(["major", "moderate", "minor", "routine"] as Severity[]).map((sev) => (
@@ -673,15 +656,19 @@ export default function MapView({ nodes, activeCalls, incidents = [], calls = []
{/* ── Incident overlay panel ───────────────────────────────────────────── */} {/* ── Incident overlay panel ───────────────────────────────────────────── */}
{incidents.length > 0 && ( {incidents.length > 0 && (
<> <>
{/* Desktop: left sidebar — starts below zoom controls + fit-all button */} {/* Desktop: left sidebar — offset below the zoom stack + fit-all button
<div className="absolute top-[8rem] left-3 bottom-[4.5rem] z-[1001] hidden md:flex flex-col w-56 gap-1.5"> so it never overlaps the Leaflet +/- controls (#118). Height is
capped and the list scrolls on its own, so the rail never reaches
the bottom-right legend. pointer-events are off on the wrapper and
back on for the cards, so the map still pans in the gaps. */}
<div className="absolute top-[9.5rem] left-3 z-[1001] hidden md:flex flex-col w-56 gap-1.5 max-h-[calc(100%-12rem)] pointer-events-none">
{/* Gate A / A2 (server-26#46) — the rail's titles, locations and {/* Gate A / A2 (server-26#46) — the rail's titles, locations and
unit counts are pipeline output. Pinned above the scroll area unit counts are pipeline output. Pinned above the scroll area
so it cannot be scrolled off the screen it qualifies. */} so it cannot be scrolled off the screen it qualifies. */}
<div className="bg-surface/90 backdrop-blur-sm border border-line rounded-lg px-2 py-1.5 shrink-0"> <div className="bg-surface/90 backdrop-blur-sm border border-line rounded-lg px-2 py-1.5 shrink-0 pointer-events-auto">
<MachineOutputNotice variant="inline" className="text-[10px] leading-snug items-start" /> <MachineOutputNotice variant="inline" className="text-[10px] leading-snug items-start" />
</div> </div>
<div className="flex flex-col gap-1.5 overflow-y-auto"> <div className="flex flex-col gap-1.5 overflow-y-auto min-h-0 pointer-events-auto">
{incidents.map((inc) => { {incidents.map((inc) => {
const color = severityColor(inc.severity); const color = severityColor(inc.severity);
const age = inc.started_at ? timeAgo(new Date(inc.started_at)) : null; const age = inc.started_at ? timeAgo(new Date(inc.started_at)) : null;
+14 -14
View File
@@ -1,15 +1,15 @@
{ {
"//": "Composite indexes for the c2-server database. Firestore auto-indexes single-field lookups and equality-only compound queries; an equality filter combined with an inequality, an orderBy on a different field, or array-contains needs an explicit composite index or the query fails at runtime with FAILED_PRECONDITION. Deploy with: firebase deploy --only firestore:indexes --project <project-id> (firebase.json pins database c2-server — without that key the CLI targets (default) and changes nothing the app can see).", "//": "Composite indexes for the c2-server database. Firestore auto-indexes single-field lookups and equality-only compound queries; an equality filter combined with an inequality, an orderBy on a different field, or array-contains needs an explicit composite index or the query fails at runtime with FAILED_PRECONDITION. Deploy with: firebase deploy --only firestore:indexes --project <project-id> (firebase.json pins database c2-server — without that key the CLI targets (default) and changes nothing the app can see).",
"//direction": "Every index here is declared ASCENDING. Firestore scans an index in either direction, so org_id+started_at ASC serves orderBy(started_at, 'desc') as well — which is what every frontend hook actually asks for. Declaring only the ASC form keeps one index per query shape instead of a matched pair.", "//direction": "The sort field's ORDER here must match the query's orderBy direction. The old note claimed 'Firestore scans either direction so ASC serves orderBy(desc)' — that is WRONG for these query shapes and cost us three broken pages (server-26 #33/#51/#110-followup, 2026-09-08): useCalls/useIncidents/useAlerts and c2-core search_calls all orderBy(x,'desc') and each got FAILED_PRECONDITION until an explicit DESCENDING index existed. A range/inequality filter with no orderBy (the backend status/ended_at, system_id/started_at, system_id/ended_at entries) is genuinely direction-agnostic and stays ASCENDING.",
"//drift-2026-08-23": "Reconciled against `gcloud firestore indexes composite list --database=c2-server` (server-26#33). The file had drifted four indexes behind the live database, and a deploy against the stale file then added ASC copies of indexes that already existed as DESC. The next deploy will offer to delete three live indexes that are deliberately not declared here — answer YES to all three: calls(org_id ASC, started_at DESC) and incidents(org_id ASC, started_at DESC) are duplicates of the ASC entries below, and alert_events(acknowledged ASC, triggered_at DESC) predates tenancy and is superseded by the org-scoped entry below. Nothing else may be deleted.", "//drift-2026-09-08": "Reconciled against the live c2-server via `gcloud firestore indexes composite list` (server-26#33). Live already carries the three DESC indexes below (calls(org_id,started_at DESC), incidents(org_id,started_at DESC), alert_events(org_id,triggered_at DESC)) plus alert_events(acknowledged,org_id,triggered_at DESC) — created directly with gcloud on 2026-09-08 to unbreak Archive + Watch. This file now declares them so a `firebase deploy --only firestore:indexes` is a no-op, NOT a set of deletions. Do NOT delete calls(org_id,started_at DESC) or incidents(org_id,started_at DESC) — the pre-2026-09-08 note calling them deletable 'duplicates of the ASC entries' was the bug. The only genuinely dead index is the pre-tenancy alert_events(acknowledged,triggered_at) with no org_id, which may be deleted.",
"indexes": [ "indexes": [
{ {
"//": "drb-frontend lib/useCalls.ts — org-scoped call list, orderBy started_at desc.", "//": "drb-frontend lib/useCalls.ts + c2-core routers/calls.py search_calls — org-scoped call list, orderBy started_at DESC.",
"collectionGroup": "calls", "collectionGroup": "calls",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
{ "fieldPath": "org_id", "order": "ASCENDING" }, { "fieldPath": "org_id", "order": "ASCENDING" },
{ "fieldPath": "started_at", "order": "ASCENDING" } { "fieldPath": "started_at", "order": "DESCENDING" }
] ]
}, },
{ {
@@ -22,7 +22,7 @@
] ]
}, },
{ {
"//": "c2-core internal/recorrelation_sweep.py:45 — status == 'ended' AND ended_at >= cutoff. Backend only; was live but undeclared until 2026-08-23.", "//": "c2-core internal/recorrelation_sweep.py:45 — status == 'ended' AND ended_at >= cutoff. Range filter, no orderBy: direction-agnostic. Backend only.",
"collectionGroup": "calls", "collectionGroup": "calls",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
@@ -31,7 +31,7 @@
] ]
}, },
{ {
"//": "c2-core internal/dedup.py:84 — system_id == X AND started_at within a +/- window. Was live-failing on essentially every inbound call (server-26#84): dedup caught the FAILED_PRECONDITION and degraded to \"not a duplicate\", so double-heard transmissions were stored twice and would have been transcribed and correlated twice the moment an AI window opened. Created directly on c2-server 2026-08-28.", "//": "c2-core internal/dedup.py:84 — system_id == X AND started_at within a +/- window. Range filter, direction-agnostic. Was live-failing on essentially every inbound call (server-26#84): dedup caught the FAILED_PRECONDITION and degraded to \"not a duplicate\", so double-heard transmissions were stored twice. Created directly on c2-server 2026-08-28.",
"collectionGroup": "calls", "collectionGroup": "calls",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
@@ -40,7 +40,7 @@
] ]
}, },
{ {
"//": "c2-core internal/vocabulary_learner.py:290 — system_id == X AND ended_at >= cutoff. Backend only; was live but undeclared until 2026-08-23.", "//": "c2-core internal/vocabulary_learner.py:290 — system_id == X AND ended_at >= cutoff. Range filter, direction-agnostic. Backend only.",
"collectionGroup": "calls", "collectionGroup": "calls",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
@@ -49,31 +49,31 @@
] ]
}, },
{ {
"//": "drb-frontend lib/useIncidents.ts — org-scoped incident browse, orderBy started_at desc.", "//": "drb-frontend lib/useIncidents.ts — org-scoped incident browse, orderBy started_at DESC.",
"collectionGroup": "incidents", "collectionGroup": "incidents",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
{ "fieldPath": "org_id", "order": "ASCENDING" }, { "fieldPath": "org_id", "order": "ASCENDING" },
{ "fieldPath": "started_at", "order": "ASCENDING" } { "fieldPath": "started_at", "order": "DESCENDING" }
] ]
}, },
{ {
"//": "drb-frontend lib/useAlerts.ts — org-scoped alert feed, orderBy triggered_at desc.", "//": "drb-frontend lib/useAlerts.ts — org-scoped alert feed, where(org_id ==) orderBy(triggered_at DESC).",
"collectionGroup": "alert_events", "collectionGroup": "alert_events",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
{ "fieldPath": "org_id", "order": "ASCENDING" }, { "fieldPath": "org_id", "order": "ASCENDING" },
{ "fieldPath": "triggered_at", "order": "ASCENDING" } { "fieldPath": "triggered_at", "order": "DESCENDING" }
] ]
}, },
{ {
"//": "drb-frontend lib/useAlerts.ts useUnacknowledgedAlerts — the nav badge.", "//": "drb-frontend lib/useAlerts.ts useUnacknowledgedAlerts (nav badge) and the /watch \"Triggered Alerts\" tab — where(org_id ==) where(acknowledged == false) orderBy(triggered_at DESC). Field tuple + triggered_at DESCENDING copy the console create_composite link verbatim (server-26#51). Distinct from the (org_id, triggered_at) feed index above (no acknowledged filter).",
"collectionGroup": "alert_events", "collectionGroup": "alert_events",
"queryScope": "COLLECTION", "queryScope": "COLLECTION",
"fields": [ "fields": [
{ "fieldPath": "org_id", "order": "ASCENDING" },
{ "fieldPath": "acknowledged", "order": "ASCENDING" }, { "fieldPath": "acknowledged", "order": "ASCENDING" },
{ "fieldPath": "triggered_at", "order": "ASCENDING" } { "fieldPath": "org_id", "order": "ASCENDING" },
{ "fieldPath": "triggered_at", "order": "DESCENDING" }
] ]
} }
], ],