Author SHA1 Message Date
Logan CusanoandClaude Opus 5.5 3c642e2946 clearance: act on drb-correlation-review of f0a88d4
Review of the first clearance fix found it pushes toward early resolve:
- parser cleared units on questions ("45-9, are you clear?"), negations
  ("not clear yet"), orders ("clear the scene", "clear to transport"),
  places/times ("Room 2 clear", "1400 hours, clear") and split "45 9" into
  unit 45. Now rejects "?", not/is/are/you, anything after the status word
  but sign-offs, place/time words; joins "45 9" -> "45-9".
- a clear from a unit never active on an incident was recorded in
  units_cleared and could pass the all-clear gate. Only units actually
  active there can clear there now.
- clearance-only calls skip the LLM tier (same as thin calls): only the
  rules engine's unit match can say which incident a 10-8 belongs to.
- replay incident view carries srcaddr/srcaddrs for the radio-ID clearance
  investigation.

Replay 09-22 10:00-12:00 ET with f0a88d4: real clears 0 -> 2 (both LLM
closure), unit clears still 0 — the parsed clears are right but those
units were never recorded as assigned (Whisper mangles unit IDs at
dispatch), which this commit does not fix.

c2-core: 465 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 17:26:38 -04:00
logan f0a88d401c Merge pull request 'correlator/intelligence: let 10-8s actually close incidents' (#174) from fix/clearance-signal into main
Build & Deploy / Build & push images (push) Successful in 4m11s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m36s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 16:49:33 -04:00
Logan CusanoandClaude Opus 5.5 8eac32caf5 correlator/intelligence: let 10-8s actually close incidents
Replay of 09-22 10:00-12:00 ET (server-26#170): 0 of 19 incidents resolved
on a clear, 19 on the idle timer, although 25 transmissions said 10-8/clear.
Three independent breaks:

1. Short clears never reached extraction. "45-9, I'm clear." is <=5 words,
   so extract_scenes skipped it before GPT and cleared_units stayed empty.
   A rule parser now names the unit when it precedes the status word
   (never guesses: "10-8, 10-8." / "CMT clear." clear nobody) and returns a
   minimal scene that links by unit overlap but cannot open an incident.
2. Clearance compared unit strings exactly, so "11-Adam" clearing never
   removed "11 Adam". Now by _normalize_unit key.
3. units_active collected "Desk", "Central", "Division", "unknown", plate
   phonetics — none of which ever clear, so all-clear could never pass.
   Only units carrying a number (and not a ten-code) are tracked now; the
   rest stay in `units` for matching.

c2-core: 463 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 16:49:30 -04:00
logan 6e82ee8579 Merge pull request 'correlation: smart tiebreak model gemini-2.5-pro is gone; use gemini-3.8-flash' (#173) from fix/smart-model into main
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m50s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 16:07:56 -04:00
Logan CusanoandClaude Opus 5.5 cdc61dcc9d correlation: smart tiebreak model gemini-2.5-pro is gone; use gemini-3.8-flash
The first replay (server-26#170) aborted on 5x 'model is unavailable'
from the tiebreak tier. Google's model list shows 2.5 closed to new
projects and no stable Pro model; 3.8-flash is the newest stable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 16:07:53 -04:00
logan 032e9bd653 Merge pull request 'Replay: fail fast on a dead AI account; extraction reports to ai_health' (#172) from fix/replay-visibility into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m35s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 15:39:04 -04:00
Logan CusanoandClaude Opus 5.5 ec91a9175f Replay: fail fast on a dead AI account; extraction reports to ai_health
First replay (290 calls, 09-22 10:00-12:00 ET) produced 0 incidents and
no errors: every gpt-4o-mini extraction failed and _sync_extract
swallowed it as "no scenes". Same shape as #169 — and the live extraction
tier in /health/ai had no reporter at all, so this has been invisible in
production too.

- intelligence: API failures propagate out of _sync_extract; extract_scenes
  reports them to ai_health ("extraction" tier, billing/dead-model
  classified) and still returns [] so the pipeline degrades as before.
- ai_health: inside a replay sandbox, failures go to the run's own sink
  instead of being dropped.
- replay: aborts after 5 permanent failures on a tier, naming the cause;
  run metrics carry ai_failures; UI shows them.
- replay estimate: audio minutes from started_at/ended_at (no duration
  field exists on call docs).
- ReplayTab exposes the loaded run on window.__drbReplay for in-page
  analysis.

c2-core: 458 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 15:39:01 -04:00
logan 8dadbdd977 Merge pull request 'Admin Replay: re-run the pipeline over past calls in a sandbox' (#171) from feat/replay into main
Build & Deploy / Build & push images (push) Successful in 4m15s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m31s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 15:24:37 -04:00
Logan CusanoandClaude Opus 5.5 aff3f16d32 Admin Replay: re-run the pipeline over past calls in a sandbox (#170)
Correlation has only ever been measured through live AI windows: days of
wall time per change, and the 09-20→22 window was invalidated outright by
unfunded AI accounts (#169). Recordings are kept regardless of AI, so the
traffic to measure against already exists.

- internal/replay.py: runs a time range of real calls through the live
  pipeline code in original order, clock pinned per call, into
  replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe),
  transcripts (re-extract), reuse (correlation only from a prior run's
  scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on
  virtual time. No alerts, summaries, vocab, AI-health alerts or pending
  terms. One run at a time, <=5000 calls, <=7 days.
- firestore.py: ContextVar sandbox redirect for calls/incidents.
- clock.py: ContextVar-pinnable now(), used on the correlation path.
- feature_flags.py: ContextVar flag override so replay runs with live AI off.
- upload.py: scene loop extracted to _extract_and_correlate, shared by the
  live pipeline and replay so replay measures the code that runs live.
- resolved_via on every incident resolve, so a real clear can be told
  from the idle timeout — live and in replay.
- routers/replay.py + /admin Replay tab: estimate, start, compare runs,
  drill into incidents with audio.

Reviewed by drb-correlation-review; its leak and fidelity findings are
fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run
(no Node on the authoring box).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 15:23:57 -04:00
logan e79b8bc37d Merge pull request 'Incident date filter matched nothing' (#168) from fix/incident-date-filter into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m27s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:15:31 -04:00
Logan CusanoandClaude Opus 5.5 c72c28f5dc frontend: incident date filter matched nothing
Incident started_at is an isoformat() string (incident_correlator.py,
routers/incidents.py), not a Firestore timestamp like calls. The range
bounds were Dates, which Firestore compares by type, so any date range
returned zero incidents. Bounds are now UTC ISO strings in the same
"+00:00" shape, which order lexicographically by time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:15:23 -04:00
logan 02b5b7b5a5 Merge pull request 'Archive Load more skipped 150 of every 200 calls' (#166) from fix/archive-paging-skip into main
Build & Deploy / Build & push images (push) Successful in 4m17s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 3m45s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:05:37 -04:00
Logan CusanoandClaude Opus 5.5 40014a47a3 c2-core: Archive "Load more" skipped 150 of every 200 calls
/calls/search scans a 200-row window and returns 50, but the next cursor
was always the last SCANNED row — so with an unfiltered list each page
jumped past the 150 matches it had already read and not shown. Resume
after the last RETURNED row when matches overflow the page; keep the
last-scanned cursor only when the page holds every match (the sparse-
filter case that cursor exists for). Same fix for /calls/eval-queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:05:31 -04:00
logan 6c0e7a4f8e Merge pull request 'Date range picker on Incidents and Archive; fix Archive Load more' (#165) from feat/date-range into main
Build & Deploy / Build & push images (push) Successful in 4m22s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m35s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:04:06 -04:00
Logan CusanoandClaude Opus 5.5 6479174022 frontend: date range picker on Incidents and Archive; fix Archive paging
Incidents and Archive get a from/to date range (native date inputs,
local-day bounds). Incidents filters in the Firestore query; Archive
passes date_from/date_to to GET /calls/search, which applies them as a
started_at range — both ride the existing org_id/started_at index.

Also fixes /calls/search and /calls/eval-queue paging: the cursor went to
Firestore as a raw ISO string against a timestamp field, which compares
by type rather than time, so "Load more" re-read the first page. Cursor
and range bounds are now parsed to datetimes (400 on garbage).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:03:40 -04:00
logan c043298902 Merge pull request 'Incidents search/filter/load-more; Archive viewable by viewers' (#164) from feat/archive-search-viewers into main
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 4s
Build & Deploy / Deploy to VM (push) Successful in 2m26s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-23 23:54:35 -04:00
Logan CusanoandClaude Opus 5.5 fa194e0f0a frontend: search, filters and load-more on Incidents; open Archive to viewers
Incidents page: text search (title, location, summary, units, vehicles,
tags, location mentions), status and type filters, and a Load more button
that pages the Firestore query 100 at a time. Filtering runs over the
loaded window, and the page says so when older incidents exist.

Archive (/calls): readable by every org member, not just admins.
GET /calls/search now takes any Firebase token scoped to the caller's org
— the Firestore rules already let members read every call in their org,
so this widens nothing. Attach/detach stays admin-only (UI and routes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 23:53:46 -04:00
Logan CusanoandClaude Sonnet 5 5f85a878fa admin: STT eval harness — record human-verified transcripts, measure real WER (#163)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m56s
Build & Deploy / Report a failed deploy (push) Successful in 1s
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.

  GET  /calls/eval-queue        -- calls with a transcript but no
                                    eval_transcript yet, paged (same bounded-
                                    window-plus-cursor shape as /search)
  PUT  /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
                                      never touches transcript/transcript_corrected
  GET  /calls/eval-stats        -- eval_count + average word error rate of
                                    the raw and corrected machine transcripts
                                    against the human-verified ones

internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.

Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.

Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-21 00:04:30 -04:00
Logan CusanoandClaude Sonnet 5 241a15b8da transcript_correction: reject a correction that changes a ten-code (#162)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m1s
Build & Deploy / Report a failed deploy (push) Successful in 1s
correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add
information", but nothing checked the model's output against its own
rules -- raw["corrected"] was accepted verbatim past a non-empty/changed
check, and the "changed" field it returns was logged for audit and never
validated.

Caught live: the same call's raw vs corrected transcript showed "10-7"
rewritten to "10-13" in one place and "10-4" in another, plus a bare "7"
expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes
Street and 4th and Rowe -> Home Street and Forest Ave, from this system's
own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly
as trustworthy as a correct transcript, which is worse than leaving the
raw mishearing in place.

_code_tokens() extracts every ten-code/signal-shaped token from the
original and corrected text/segments; any change to that set discards
the correction and falls back to raw. Checked independently for the
joined text and for segments, consistent with the existing
all-or-nothing segment-alignment rule.

Does not catch a wrong word swapped for another equally plausible
non-code word -- that class still depends entirely on the model
following its own prompt. Filed server-26#161 for the broader STT/audio
quality initiative this belongs alongside.

Verified: 426 pass, 0 fail (4 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 21:51:41 -04:00
Logan CusanoandClaude Sonnet 5 f91d4559f3 deploy: treat an empty .last_good_tag the same as a missing one (#156)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m0s
Build & Deploy / Report a failed deploy (push) Successful in 1s
cat file 2>/dev/null || echo latest only falls back when cat itself fails
(nonzero exit, i.e. the file is missing) -- a file that EXISTS but is EMPTY
makes cat succeed with empty output, so PREV_TAG became "" instead of
"latest". That "" failed the emptiness check further down and exited 1 --
AFTER git pull + docker compose up -d had already succeeded. Worse: exiting
there skips the Health check step (Gitea Actions doesn't run later steps
after a failure), and Health check is the ONLY step that ever writes a real
value to .last_good_tag. Self-perpetuating: once the file went empty, every
future deploy failed the same way forever, with the app itself deploying
fine underneath it every time (confirmed against run 611: deploy log shows
all three images pulled/recreated/started at 66bbf5b, and /health
self-reports that exact GIT_SHA, baked into the image at build time --
two independent signals, same commit).

${VAR:-default} covers empty and unset in one expansion, matching the
fallback behavior the surrounding comment already documented as intended.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:36:17 -04:00
Logan CusanoandClaude Sonnet 5 66bbf5b473 intelligence: don't reject a geocode just because it is far from the node (#159)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 2s
Build & Deploy / Deploy to VM (push) Failing after 2m4s
Build & Deploy / Report a failed deploy (push) Successful in 1s
_geocode_location's node-distance fallback (geocode_max_km, 40km) is a
proxy for "is this plausible" that only makes sense when the node's own
position is the best guess we have at the area. It doesn't hold for a
citywide/patched feed: node-002 sits in Westchester but relays "New York
City - NYPD Citywide 2 Patch", ~56km from the addresses on it. Every real,
correctly-geocoded address on that talkgroup was rejected by this check,
every time -- location_coords stayed permanently null for the whole
system, which killed the location_proximity correlation signal and let
duplicate incidents form for the same event reported at two nearby
addresses two minutes apart ("Shots Fired at Jackson Avenue" /
"Shots Fired at 1108 Jackson Avenue", 2026-09-20 ~23:00 UTC, merged by
hand via the Archive page's attach/detach while this fix went in).

trust_named_region skips the node-distance rejection exactly when the
query already carries a place name that isn't the node's own position --
operator-set area_context, or a municipality parsed from the talkgroup's
own name. The anchor path is untouched; an anchor's own radius is always
authoritative when one has been resolved.

Also fixes a compounding defect found while verifying: the query was
grafting the node's own COUNTY onto an already-self-named region
("...New York City..., Westchester, New York"), which is self-contradictory
and could degrade the geocode independent of the distance check. The
node's county is now used only when nothing else names the place; state
stays in both branches since it's coarse enough to be correct either way.
Extracted the query-assembly logic into a pure, unit-tested helper
(_location_query_parts) rather than testing it only through the full
extraction pipeline.

Filed #160 as a follow-up: place_verifier.py's verify() has the identical
no-anchor gap for transcript place-name correction, not fixed here.

Verified: 422 pass, 0 fail (9 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:22:42 -04:00
Logan CusanoandClaude Sonnet 5 6c095083fc correlator: use srcaddr for thin-call disambiguation instead of pure recency (#158)
The fast/thin path (short acks like "10-4", enabled to attempt linking at
all by 1ffff25) had no identity signal available -- transcript_too_short
skips GPT extraction entirely, so call_units is always empty for this
population. It fell back to "most recently updated incident on this
talkgroup", which silently misattaches a short ack to the wrong incident
whenever two are live on the same busy dispatch channel at once.

metadata_watcher.py already captures the P25 source radio ID (srcaddr) on
every call independent of transcript content, and it already reaches the
call doc (models.py, mqtt_handler.py:245) -- it was just never read by the
correlator. Thread it through _build_context, check it against the
srcaddrs already seen on each TG-matched incident before falling back to
recency, and accumulate it on the incident (_update_incident/_create_incident)
so later calls from the same radio can match.

Verified: 422 pass, 0 fail (4 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:22:06 -04:00
Logan CusanoandClaude Sonnet 5 2e67d1bad6 ci: deploy Firestore rules/indexes from the runner, not the VM (server-26#51)
The rules/indexes deploy step already existed here, gated on `command -v
firebase` over SSH on the deploy VM. It never found one (no node on the
VM), so it silently warned-and-skipped on every single deploy for weeks —
PR #124 even auto-closed #13/#51 as if this were fixed, when it wasn't.

New standalone deploy-firestore-rules job runs on the Gitea runner itself
(always has node), authenticated via a new FIREBASE_TOKEN secret (from
`firebase login:ci`) instead of anything pre-installed on the VM. It's
independent of the deploy job's health-check/rollback chain on purpose —
a rules deploy failure has nothing to roll back and must not trigger that
logic. notify-failure now distinguishes which job actually failed so the
Discord alert doesn't misreport "production is unchanged" when the app
deployed fine and only the rules push failed.

Needs FIREBASE_TOKEN added as a Gitea Actions secret before this actually
runs — it will fail loudly (by design) until then, which is the whole
point: a loud failure beats a silent skip.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 19:08:58 -04:00
Logan CusanoandClaude Sonnet 5 1ffff25cd2 upload: let short transcripts (<=5 words) attempt correlation instead of never linking at all
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 2m21s
Build & Deploy / Report a failed deploy (push) Successful in 1s
intelligence.py skips GPT extraction for transcripts <=5 words ("10-8", "show
me clear", a unit check-in) -- real cost/hallucination guard, kept as-is. But
upload.py's no-scenes correlation fallback (the path that thin-links a call
by talkgroup even with zero extracted content) excluded ANY skip_reason,
including transcript_too_short -- so this exact population, brief but real
follow-up and clearance traffic, never even attempted to attach to anything.
Found live: "Live, Ossining." sitting an orphan 6 seconds before a real
incident's founding call, on the same talkgroup.

Now only garbage_transcript (Whisper hallucination, no real content) stays
excluded; transcript_too_short reaches the same thin/fast-path fallback
already trusted for no-transcript calls, gated the same way -- same-talkgroup,
recently-active incident required before anything attaches. No GPT re-invoked,
no new cost.

Verified: 410 pass, 0 fail.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:39:46 -04:00
Logan CusanoandClaude Sonnet 5 3d2b722c64 Wire AIS end to end: telemetry ingestion + live map overlay
node-26#9, same shape as the ADS-B commit. Adds POST /telemetry/ais
(same node-key auth, same org_id-stamped upsert-by-key pattern, this time
by mmsi into a new `vessels` collection) and its docInMyOrg() firestore
rule. Frontend gets useVessels() (mirrors useAircraft(), longer staleness
window since AIS position reports are minutes apart, not seconds) and an
opt-in "Vessels" map overlay.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:12:56 -04:00
Logan CusanoandClaude Sonnet 5 5537b095df Wire ADS-B end to end: telemetry ingestion + live map overlay
node-26#9. Adds POST /telemetry/adsb (node-key authed via
require_node_service_or_firebase_token) that upserts one Firestore doc per
icao into a new `aircraft` collection, org_id stamped from the reporting
node the same way upload.py defensively stamps `calls`. firestore.rules
gets a matching docInMyOrg()-gated read rule.

Frontend: useAircraft() mirrors useNodes()'s onSnapshot pattern, filtering
docs older than 2 minutes client-side since nothing prunes a stale aircraft
doc server-side yet. MapView gets an opt-in "Aircraft" overlay (unchecked
by default, like the weather radar layer) rendering a rotated plane glyph
per sighting.

Unverified via typecheck — no Node.js/npm on this authoring box yet (see
CLAUDE.md Testing reality). Server side is pytest-covered (test_telemetry.py).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:08:29 -04:00
Logan CusanoandClaude Sonnet 5 8892e824fc Add second-SDR fields: secondary_sdr_mode + sdr_count on NodeRecord
Server-side half of node-26#9. NodeRecord gains secondary_sdr_mode
(none|adsb|ais|op25_2) and sdr_count; checkin ingestion stores both,
and PATCH /nodes/{id} accepts and re-pushes secondary_sdr_mode the same
way hardware_preset/ppm_override already work, so it isn't wiped by a
system reassignment (see server-26#111 for the pre-existing bug that
pattern avoids repeating).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 15:56:33 -04:00
Logan CusanoandClaude Sonnet 5 3f69879437 intelligence: stop the police-channel default from typing content-free chatter (#138)
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 1m12s
Build & Deploy / Report a failed deploy (push) Successful in 1s
EXTRACTION_PROMPT's incident_type rule always returned "police" for anything
on a police channel unless contradicted, so pure administrative chatter
(post check-ins, roll call, bare acknowledgements) got a truthy incident_type
and defeated the creation gate's "type" veto (_call_is_substanceless) ~82%
of the time (measured server-26#138/CORRELATION_REVIEW_0914.md, confirmed
against live 9-14 data: 9/9 type-veto examples checked all had severity
"routine" — same population the severity rubric already correctly
identifies as content-free, incident_type just wasn't using that signal).

Rule now checks for actual event content FIRST, on every channel, before
applying the channel-inference defaults; content-free traffic returns
"unknown" (already normalizes to None) instead of a channel default,
letting the existing gate correctly veto it. No correlator/gate code
touched — CORRELATION_REVIEW_0914.md's own recommendation was to fix this
in the prompt, not _call_is_substanceless, to avoid risking a real event
getting gated out.

Verified: 401 pass, 0 fail. Live effect to be confirmed against fresh
traffic (AI features just re-enabled this session after being off since
9/14) — tracking in SESSION_STATE.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 15:42:48 -04:00
Logan CusanoandClaude Sonnet 5 fb0bb15c22 correlator/intelligence: close the incident-clearance gap (dispatch-to-10-8 lifecycle)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Failing after 3m25s
Build & Deploy / Report a failed deploy (push) Successful in 1s
Two independent fixes, found by tracing why incidents never actually close
(only 2/281 incidents in 5 correlation debug windows ever got a non-empty
units_cleared; 183 resolved via the 90-min idle sweep instead of a real clear):

1. Pattern B (re-dispatch accept, no explicit 10-8): reassignment=True already
   fired correctly and suppressed the unit from re-linking to its prior
   incident, but nothing ever released the unit FROM that incident — it just
   sat "active" until the idle sweep timed it out. _release_reassigned_units
   now scans other active incidents for unit overlap on a reassignment and
   clears the unit there, reusing the same units_active/units_cleared merge
   (factored out as _apply_unit_clearance) that explicit 10-8 extraction uses.

2. Pattern A (self-clear) extraction was inconsistent for two reasons: no
   per-system unit ID format awareness anywhere in the pipeline (formats vary
   by department with zero shared convention), and the cleared_units prompt
   rule only accepted a unit self-reporting, missing dispatch confirming a
   unit's status back to them. Added system.unit_format_hint (owner-authored
   free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed
   into the extraction prompt, and broadened the cleared_units rule while
   still requiring an identifiable unit ID (guards against bare "10-8"/"clear"
   noise, including Whisper hallucination runs already caught upstream by
   _is_garbage_transcript).

Verified: 401 pass, 0 fail (local Linux venv ~/venvs/drb-5c — see CLAUDE.md
testing-reality note).

server-26#pending — not yet filed, Gitea unreachable from this sandbox.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 14:25:01 -04:00
logan a9197709f8 Merge pull request 'correlator: stop the re-correlation sweep racing an in-flight upload (#131)' (#154) from fix/131-sweep-race-duplicate-link into main
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 1m9s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-14 00:31:14 -04:00
Logan CusanoandClaude Sonnet 5 8dd636af8f correlator: stop the re-correlation sweep racing an in-flight upload (#131)
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.

Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.

15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.

recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.

Sandboxed pytest: 381 -> 387.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:30:24 -04:00
49 changed files with 4795 additions and 218 deletions
+75 -24
View File
@@ -115,27 +115,14 @@ jobs:
# Update compose files + mosquitto config
git pull origin main
# server-26#51: Firestore rules + composite indexes had no deploy
# path and regressed silently after every fix (the alert_events and
# calls(org_id,started_at) indexes among them). The VM runs as the
# project service account, so firebase-tools authenticates via ADC
# with no key file, and infra/firestore/firebase.json pins database
# c2-server. Indexes go on additively -- no --force -- so a stray
# edit to firestore.indexes.json can never delete a live index;
# rules are a full replace, which is the intent. --non-interactive
# means the FIRST run after a drift still needs a one-time manual
# `firebase deploy` on the VM to clear pending deletions (it aborts
# rather than guess). A failure here warns but does NOT fail the
# deploy: a transient Firebase API error must not roll back a good
# app build.
if command -v firebase >/dev/null 2>&1; then
( cd /opt/drb/infra/firestore \
&& firebase deploy --only firestore:rules,firestore:indexes \
--project ${{ secrets.FIREBASE_PROJECT_ID }} --non-interactive ) \
|| echo "WARNING: firestore deploy failed (server-26#51) -- rules/indexes may be stale"
else
echo "WARNING: firebase CLI not on the VM -- skipped firestore deploy (server-26#51); install once with: npm i -g firebase-tools"
fi
# server-26#51: Firestore rules/indexes deploy used to be attempted
# HERE, over SSH, gated on the VM having firebase-tools installed.
# It never did (no node on the VM), so this silently warned and
# skipped on every deploy for weeks -- PR #124 even auto-closed
# #13/#51 as if it were fixed. Moved to a standalone
# deploy-firestore-rules job below that runs on the Gitea runner
# itself (which always has node), so it no longer depends on
# anything being pre-installed on this VM.
# server-26#65: capture what is actually live BEFORE switching, so
# a bad deploy has something concrete to fall back to. This reads
@@ -146,7 +133,20 @@ jobs:
# has confirmed the tag it names actually answered /health. A
# fresh VM with no file yet falls back to :latest, same escape
# hatch as a manual `up -d` with no TAG set.
PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null || echo latest)
#
# server-26#156: `cat missing-file || echo latest` only falls back
# when cat itself fails (nonzero exit) -- a file that EXISTS but is
# EMPTY (the state this file was found in, 2026-09-20) makes cat
# succeed with empty output, so PREV_TAG became "" instead of
# "latest". That "" then failed the emptiness check below and
# exited 1 -- AFTER git pull + up -d had already succeeded -- which
# skips the Health check step entirely (later steps don't run after
# a failure), and Health check is the ONLY thing that ever writes a
# real value here. Self-perpetuating: every deploy failed the same
# way forever, with the app itself deploying fine underneath it.
# ${VAR:-default} covers empty AND unset in one expansion.
PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null)
PREV_TAG="${PREV_TAG:-latest}"
echo "PREV_TAG=$PREV_TAG"
# Deploy THIS commit's images, not :latest. Overlapping runs are
@@ -291,9 +291,48 @@ jobs:
echo "status=success" >> "$GITHUB_OUTPUT"
echo "rolled_back_to=$PREV_TAG" >> "$GITHUB_OUTPUT"
deploy-firestore-rules:
name: Deploy Firestore rules & indexes
needs: build
runs-on: ubuntu-latest
# Deliberately independent of the `deploy` job (app containers) and its
# health-check/rollback chain above: a rules/indexes deploy failure has
# nothing to roll back (there is no previous "build" of a ruleset to
# revert to via this pipeline) and must never be conflated with an app
# deploy failure by triggering that job's rollback logic. This job
# failing is its own, separate red run -- picked up by notify-failure
# below -- not a signal to touch the running containers.
steps:
- uses: actions/checkout@v4
- name: Deploy firestore rules and indexes
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
run: |
set -e
# server-26#51: this used to run over SSH on the deploy VM, gated
# on the VM having firebase-tools installed. It never did, so it
# silently warned-and-skipped on every single deploy for weeks.
# Running it here instead means the only prerequisite is a secret
# -- FIREBASE_TOKEN, from `firebase login:ci` -- rather than
# something installed by hand on a machine this pipeline doesn't
# otherwise touch. A missing token now fails this job LOUDLY
# (picked up by notify-failure) instead of a buried warning line
# nobody reads in the app deploy's logs.
if [ -z "$FIREBASE_TOKEN" ]; then
echo "FIREBASE_TOKEN secret is not set -- cannot deploy Firestore rules/indexes." >&2
echo "Generate one with 'firebase login:ci' and add it as a Gitea Actions secret." >&2
exit 1
fi
npm install -g firebase-tools
cd infra/firestore
firebase deploy --only firestore:rules,firestore:indexes \
--project ${{ secrets.FIREBASE_PROJECT_ID }} \
--token "$FIREBASE_TOKEN" --non-interactive
notify-failure:
name: Report a failed deploy
needs: [build, deploy]
needs: [build, deploy, deploy-firestore-rules]
if: failure()
runs-on: ubuntu-latest
@@ -310,6 +349,8 @@ jobs:
SHA: ${{ gitea.sha }}
ROLLBACK_STATUS: ${{ needs.deploy.outputs.rollback_status }}
ROLLBACK_SHA: ${{ needs.deploy.outputs.rollback_sha }}
DEPLOY_RESULT: ${{ needs.deploy.result }}
RULES_RESULT: ${{ needs.deploy-firestore-rules.result }}
run: |
if [ -z "$WEBHOOK" ]; then
echo "DEPLOY_ALERT_WEBHOOK is not set - skipping notification."
@@ -321,6 +362,16 @@ jobs:
run_url = os.environ["RUN_URL"]
status = os.environ.get("ROLLBACK_STATUS", "")
rollback_sha = os.environ.get("ROLLBACK_SHA", "")
deploy_result = os.environ.get("DEPLOY_RESULT", "")
rules_result = os.environ.get("RULES_RESULT", "")
# deploy-firestore-rules runs independent of the app deploy/rollback
# chain (see its own job comment), so its failure needs its own
# branch here -- otherwise this fell through to the generic "Build
# failed before any deploy was attempted" text even when the app
# deployed fine and only the Firestore rules/indexes push failed.
if deploy_result != "failure" and rules_result == "failure":
detail = "App deploy succeeded; Firestore rules/indexes deploy FAILED (server-26#51). Rules may be stale — check FIREBASE_TOKEN and the job log."
# server-26#65: the old text here unconditionally claimed
# "production is still running the previous build" -- true only
@@ -329,7 +380,7 @@ jobs:
# class of bug the correlator instrumentation exists to catch), or
# once the deploy job's own rollback path has run. Say what
# actually happened instead.
if status == "success":
elif status == "success":
detail = "Automatic rollback to `%s` succeeded. Production is back on the previous good build." % rollback_sha[:8]
elif status == "failed":
detail = ("Automatic rollback to `%s` FAILED. Production state is UNKNOWN -- "
+4 -1
View File
@@ -45,7 +45,10 @@ class Settings(BaseSettings):
# while correlation behaviour was being tuned against rules-only output.
# Verify against https://ai.google.dev/gemini-api/docs/models before changing.
corr_cheap_model: str = "gemini-3.6-flash" # was gemini-2.0-flash (shut down)
corr_smart_model: str = "gemini-2.5-pro" # was gemini-1.5-pro (shut down)
# gemini-2.5-pro was closed to new projects by 2026-09 (every tiebreak 404'd
# in the first replay run, server-26#170); Google lists no stable Pro model,
# so the smart tier is the newest stable Flash instead.
corr_smart_model: str = "gemini-3.8-flash" # was gemini-2.5-pro, gemini-1.5-pro
# Transcript correction (server-26#36). Runs inside transcription, once per
# transcribed call above MIN_WORDS_FOR_CORRECTION, so it is priced like STT
# rather than like the correlation tier — cheap model on purpose.
+22
View File
@@ -25,6 +25,7 @@ transcription.py) need the exact same judgment call and must not each grow
their own slightly-different copy that drifts.
"""
import asyncio
from contextvars import ContextVar
from datetime import datetime, timezone
from typing import Optional
@@ -59,6 +60,14 @@ def _default_state() -> dict:
_state: dict[str, dict] = {t: _default_state() for t in TIERS}
# Set by a replay run to a list it owns; report_degraded appends there instead
# of touching _state while inside a sandbox (see app/internal/replay.py).
_sandbox_failures: ContextVar[Optional[list]] = ContextVar("drb_ai_sandbox_failures", default=None)
def collect_sandbox_failures(sink: Optional[list]):
return _sandbox_failures.set(sink)
def classify(text: str) -> str:
"""
@@ -108,6 +117,16 @@ async def report_degraded(
TRANSIENT_ALERT_THRESHOLD consecutive failures have been reported for
this tier, so an ordinary blip never pages anyone.
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
# A replay's rate limits are not a live outage, and must never page
# the AI-alert webhook or flip /health/ai (app/internal/replay.py).
# They are the run's own problem, so they go to the run instead.
sink = _sandbox_failures.get()
if sink is not None:
sink.append({"tier": tier, "provider": provider, "model": model,
"problem": problem, "permanent": permanent})
return
if tier not in _state:
_state[tier] = _default_state()
entry = _state[tier]
@@ -145,6 +164,9 @@ async def report_healthy(tier: str) -> None:
just after a failure -- it is what lets a degraded tier recover on its
own instead of staying red forever after one transient blip.
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
return # nor may a replay's success "recover" a real live outage
if tier not in _state:
_state[tier] = _default_state()
entry = _state[tier]
+2
View File
@@ -448,6 +448,8 @@ async def add_pending(system_id: str, talkgroup_id: Any, entries: list[dict]) ->
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
return 0 # a replay proposes nothing to the live review queue
if not system_id or talkgroup_id is None or not entries:
return 0
system_doc = await fstore.doc_get("systems", system_id)
+32
View File
@@ -0,0 +1,32 @@
"""
Pipeline clock.
`now()` is `datetime.now(timezone.utc)` everywhere except inside a replay run
(app/internal/replay.py), which pins it to the replayed call's own time so the
correlator's recency windows, the idle-resolve sweep and every started_at /
updated_at / resolved_at it writes behave the way they did live.
A ContextVar rather than a module global: a replay runs as a background task
alongside real uploads, and each asyncio task (and every asyncio.to_thread it
spawns) carries its own copy of the context, so a pinned clock can never leak
into a live call's pipeline.
"""
from contextvars import ContextVar
from datetime import datetime, timezone
from typing import Optional
_pinned: ContextVar[Optional[datetime]] = ContextVar("drb_clock_pinned", default=None)
def now() -> datetime:
pinned = _pinned.get()
return pinned if pinned is not None else datetime.now(timezone.utc)
def pin(when: Optional[datetime]):
"""Pin the clock for the current context. Returns a token for `unpin`."""
return _pinned.set(when)
def unpin(token) -> None:
_pinned.reset(token)
+22 -1
View File
@@ -6,7 +6,8 @@ in-memory TTL cache so flag reads don't add a Firestore round-trip to every
call upload.
"""
import time
from typing import Any
from contextvars import ContextVar
from typing import Any, Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
@@ -36,6 +37,21 @@ _DEFAULTS: dict[str, bool] = {
"transcript_correction_enabled": True,
}
# A replay run (app/internal/replay.py) states exactly which AI steps it runs,
# independent of the live switches — the whole point is re-running the pipeline
# while live AI is OFF. ContextVar so the override never reaches a live upload.
_forced: ContextVar[Optional[dict[str, bool]]] = ContextVar("drb_forced_flags", default=None)
def force_flags(flags: Optional[dict[str, bool]]):
"""Override resolve_flags() for the current context. Returns a reset token."""
return _forced.set(flags)
def unforce_flags(token) -> None:
_forced.reset(token)
_cache: dict[str, Any] = {}
_cache_ts: float = 0.0
@@ -211,6 +227,11 @@ async def resolve_flags(system_id: str | None):
"""
from app.internal import firestore as _fstore
forced = _forced.get()
if forced is not None:
full = {k: bool(forced.get(k, False)) for k in _DEFAULTS}
return full, lambda name: full.get(name, False)
flags = await get_flags()
system_ai_flags: dict = {}
+46 -7
View File
@@ -1,5 +1,6 @@
import asyncio
import time as _time
from contextvars import ContextVar
from typing import Optional, Any
import firebase_admin
from firebase_admin import credentials, firestore as fs
@@ -40,23 +41,61 @@ _init_firebase()
db = fs.client(database_id=settings.firestore_database)
# ---------------------------------------------------------------------------
# Replay sandbox (app/internal/replay.py)
# ---------------------------------------------------------------------------
# While a replay run is executing, every read and write the pipeline makes to
# `calls` or `incidents` is redirected to that run's own subcollections under
# replay_runs/{run_id}/, so re-running the pipeline over past traffic can never
# touch a live call or incident. A subcollection keeps the same collection ID
# ("calls"/"incidents"), so the composite indexes prod queries depend on apply
# to it unchanged. Everything else (systems, nodes, config) is read from prod
# as-is. ContextVar for the same reason as app/internal/clock.py: the redirect
# follows the replay task and never a concurrent live upload.
SANDBOXED_COLLECTIONS = frozenset({"calls", "incidents"})
_sandbox_root: ContextVar[Optional[str]] = ContextVar("drb_fstore_sandbox", default=None)
def enter_sandbox(root: Optional[str]):
"""Redirect calls/incidents under `root` (e.g. "replay_runs/<id>") for this context."""
return _sandbox_root.set(root)
def exit_sandbox(token) -> None:
_sandbox_root.reset(token)
def in_sandbox() -> bool:
"""True inside a replay run. Anything that writes live state OTHER than
calls/incidents (AI health alerts, pending-term queues) checks this and
stands down — the redirect below only covers the two sandboxed collections."""
return _sandbox_root.get() is not None
def _path(collection: str) -> str:
root = _sandbox_root.get()
if root and collection in SANDBOXED_COLLECTIONS:
return f"{root}/{collection}"
return collection
# ---------------------------------------------------------------------------
# Thin async wrappers — firebase-admin is synchronous, run in thread executor
# ---------------------------------------------------------------------------
async def doc_set(collection: str, doc_id: str, data: dict, merge: bool = True) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.set, data, merge=merge)
async def doc_get(collection: str, doc_id: str) -> Optional[dict]:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
snap = await asyncio.to_thread(ref.get)
return snap.to_dict() if snap.exists else None
async def doc_update(collection: str, doc_id: str, data: dict) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.update, data)
@@ -66,7 +105,7 @@ async def collection_list(collection: str, **filters) -> list[dict]:
Optional keyword filters: field=value pairs passed as equality where-clauses.
"""
def _query():
ref = db.collection(collection)
ref = db.collection(_path(collection))
for field, value in filters.items():
ref = ref.where(filter=FieldFilter(field, "==", value))
return [doc.to_dict() for doc in ref.stream()]
@@ -103,7 +142,7 @@ async def collection_where(
unscoped equality-only lookups can keep using collection_list().
"""
def _query():
ref = db.collection(collection)
ref = db.collection(_path(collection))
for field, op, value in conditions:
ref = ref.where(filter=FieldFilter(field, op, value))
for field, direction in (order_by or []):
@@ -118,7 +157,7 @@ async def collection_where(
async def doc_delete(collection: str, doc_id: str) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.delete)
@@ -128,7 +167,7 @@ async def doc_get_cached(collection: str, doc_id: str, ttl: float = 300.0) -> Op
Use for documents that change rarely (systems config, node assignments).
Default TTL is 5 minutes — a write will be visible within that window.
"""
key = f"{collection}/{doc_id}"
key = f"{_path(collection)}/{doc_id}"
now = _time.monotonic()
entry = _doc_cache.get(key)
if entry and now < entry[0]:
+155 -13
View File
@@ -51,6 +51,7 @@ from datetime import datetime, timezone, timedelta
from typing import Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import clock
from app.config import settings
_PURSUIT_TAGS = frozenset({
@@ -223,6 +224,28 @@ def _normalize_unit(unit: str) -> str:
return key or unit.strip().lower()
def _is_trackable_unit(unit: str) -> bool:
"""
Whether a unit is concrete enough to hold an incident open until it clears.
Extraction lists everything that sounds like a unit — "Desk", "Central",
"Division", "sergeant", "unknown", and plate phonetics ("John Henry
Zebra"). None of those ever transmit a 10-8, so while they sat in
units_active the all-clear gate below could never pass: in the first
replay (server-26#170, 09-22 10:00-12:00) 0 of 19 incidents resolved on
a clear and every one had such a name in units_active. A real radio unit
ID carries a number ("45-9", "11-Adam 2", "Whitestone 1", "E-14"), so
only those gate resolution. The others are still kept in `units` and
still match for correlation.
"""
if _TEN_CODE_RE.match((unit or "").strip()):
return False # "10-8" read back as a unit ID is the status, not a unit
return any(ch.isdigit() for ch in unit or "")
_TEN_CODE_RE = re.compile(r"^10[\s-]?\d{1,2}$")
def _unit_keys(units: Optional[list[str]]) -> set[str]:
"""Comparison keys for a unit list, empties dropped."""
return {k for k in (_normalize_unit(u) for u in (units or [])) if k}
@@ -812,7 +835,7 @@ async def _build_context(
transcript: Optional[str] = None,
scene_index: int = 0,
) -> dict:
now = reference_time or datetime.now(timezone.utc)
now = reference_time or clock.now()
window = timedelta(hours=settings.correlation_window_hours)
call_doc = await fstore.doc_get("calls", call_id) or {}
@@ -875,11 +898,17 @@ async def _build_context(
is_thin_call = _is_thin_call(
call_units, call_vehicles, coords, tags, location, call_severity, reassignment
)
# server-26#158: the P25 source radio ID. Captured on every call by the
# edge node's metadata_watcher.py independent of transcript content, so
# it survives even when transcript_too_short skips GPT extraction
# entirely and leaves call_units empty — exactly the population the
# thin-call path below has no other identity signal for.
call_srcaddr = call_doc.get("srcaddr")
return {
"call_id": call_id, "org_id": org_id, "all_active": all_active, "recent": recent,
"call_doc": call_doc, "call_embedding": call_embedding,
"scene_transcript": scene_transcript,
"scene_transcript": scene_transcript, "call_srcaddr": call_srcaddr,
"call_units": call_units, "call_vehicles": call_vehicles,
"call_cleared": call_cleared, "call_severity": call_severity,
"coords": coords, "is_thin_call": is_thin_call, "now": now,
@@ -933,6 +962,7 @@ def _run_decision(ctx: dict) -> dict:
call_severity = ctx["call_severity"]
coords = ctx["coords"]
is_thin_call = ctx["is_thin_call"]
call_srcaddr = ctx.get("call_srcaddr")
system_id = ctx["system_id"]
talkgroup_id = ctx["talkgroup_id"]
talkgroup_name = ctx["talkgroup_name"]
@@ -1008,8 +1038,28 @@ def _run_decision(ctx: dict) -> dict:
# incident idle up to tg_fast_path_idle_minutes (90) with no
# single-candidate requirement and no fit test of any kind. Four
# hours is not a bound, and neither is ninety minutes.
# server-26#158: identity beats guesswork. A thin call has no
# extracted units (GPT never ran), but it still carries the P25
# radio ID that transmitted it — stronger, cheaper evidence than
# "most recently active" and immune to the exact failure this
# path exists to guard against: two incidents both live on one
# busy dispatch channel. If the radio that sent this call already
# has calls on one of the TG-matched incidents, that IS the
# thread, regardless of which incident is more recently updated
# or how many candidates are in the window.
srcaddr_matches = [
inc for inc in tg_recent
if call_srcaddr and call_srcaddr in (inc.get("srcaddrs") or [])
]
THIN_CONVERSATIONAL_SECS = 30
thin_window_min = settings.tg_dispatch_thin_idle_minutes
if srcaddr_matches:
thin_pool = [max(srcaddr_matches, key=lambda inc: inc.get("updated_at", ""))]
logger.info(
f"Correlator fast-path thin (srcaddr match): radio {call_srcaddr} "
f"already on {len(srcaddr_matches)} candidate(s) for call {call_id}"
)
else:
very_recent = [
inc for inc in tg_recent
if _idle_gate_minutes(inc, now) * 60 <= THIN_CONVERSATIONAL_SECS
@@ -1049,8 +1099,10 @@ def _run_decision(ctx: dict) -> dict:
# no fit signal, so the admin debug view's "fit_signal
# distribution" panel read empty on 95% of calls and looked
# broken. Name what actually decided it: recency on this
# talkgroup, with no content to check a fit against.
"corr_fit_signal": "thin_recency",
# talkgroup, with no content to check a fit against — or,
# when the same radio ID already touched a candidate
# (server-26#158), that identity match instead of a guess.
"corr_fit_signal": "thin_srcaddr_match" if srcaddr_matches else "thin_recency",
"corr_candidates": len(thin_pool),
}
logger.info(
@@ -1452,6 +1504,8 @@ async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
equivalent to reading the flat fields today.
"""
incident_id = await _apply_decision(decision, ctx)
if ctx.get("reassignment"):
await _release_reassigned_units(ctx, incident_id)
corr_debug = decision.get("corr_debug") or {}
if corr_debug:
scene_index = ctx.get("scene_index", 0)
@@ -1509,6 +1563,7 @@ async def _apply_decision(decision: dict, ctx: dict) -> Optional[str]:
call_cleared = ctx["call_cleared"]
coords = ctx["coords"]
now = ctx["now"]
call_srcaddr = ctx.get("call_srcaddr")
incident_type = decision["incident_type"]
if action == "link":
@@ -1520,7 +1575,7 @@ async def _apply_decision(decision: dict, ctx: dict) -> Optional[str]:
location, location_coords, call_units, call_vehicles, call_embedding, now,
talkgroup_name=talkgroup_name, incident_type=incident_type,
cleared_units=call_cleared, refresh_activity=not thin_link,
call_severity=call_severity,
call_severity=call_severity, call_srcaddr=call_srcaddr,
)
return matched_incident["incident_id"]
@@ -1552,6 +1607,7 @@ async def _apply_decision(decision: dict, ctx: dict) -> Optional[str]:
call_id, org_id, incident_type, talkgroup_id, talkgroup_name, system_id,
tags, location, location_coords,
call_units, call_vehicles, call_embedding, call_severity, now,
call_srcaddr=call_srcaddr,
)
if existing_master_id:
@@ -1597,6 +1653,7 @@ async def _apply_decision(decision: dict, ctx: dict) -> Optional[str]:
call_id, org_id, incident_type, talkgroup_id, talkgroup_name, system_id,
tags, location, location_coords,
call_units, call_vehicles, call_embedding, call_severity, now,
call_srcaddr=call_srcaddr,
)
decision["corr_debug"]["corr_path"] = "new"
@@ -1861,6 +1918,80 @@ def _call_fits_incident(
return False, "no_signal"
def _apply_unit_clearance(inc: dict, cleared: list[str]) -> tuple[list[str], list[str], bool]:
"""
Merge `cleared` into inc's units_active/units_cleared. Shared by
_update_incident (explicit 10-8/back-in-service extraction) and
_release_reassigned_units (server-26#<pending> pattern B: a unit accepting
a new dispatch, reassignment=True, is real-world evidence they're off
their prior call even without an explicit clearance phrase).
Returns (units_active, units_cleared, auto_resolved) — auto_resolved is
True when every tracked unit has now cleared, matching the resolve gate
at the bottom of _update_incident.
"""
units_active = list(inc.get("units_active") or [])
units_cleared = list(inc.get("units_cleared") or [])
# Compared by normalised key: the unit that cleared as "11-Adam" is the
# one that went active as "11 Adam", and exact equality left it active.
# Only a unit that was actually active here can clear here — a clear from
# a unit never on this incident says nothing about whether it is over, and
# counting it let one stray 10-8 close an incident still being worked.
cleared_keys = _unit_keys(cleared)
releasing = [u for u in units_active if _normalize_unit(u) in cleared_keys]
units_active = [u for u in units_active if _normalize_unit(u) not in cleared_keys]
known_cleared = _unit_keys(units_cleared)
for u in releasing:
if _normalize_unit(u) not in known_cleared:
units_cleared.append(u)
known_cleared.add(_normalize_unit(u))
auto_resolved = bool(units_cleared) and not units_active
return units_active, units_cleared, auto_resolved
async def _release_reassigned_units(ctx: dict, exclude_incident_id: Optional[str]) -> None:
"""
server-26#<pending>: reassignment=True means a unit is accepting a NEW
dispatch — real-world evidence they're off whatever they were on before,
even when they never say an explicit 10-8/clear phrase (dispatch: "are
you able to clear and take a run at X" / unit: "10-4" carries no
self-reported clearance language intelligence.py's cleared_units
extraction looks for). Without this, that unit's prior incident is only
ever closed by the 90-minute idle sweep, not a real clear.
Scoped to OTHER active incidents (exclude_incident_id keeps this call's
own outcome untouched) with unit overlap in units_active — mirrors the
unit-continuity candidate scan at :1142 but releases instead of links.
"""
call_units = ctx.get("call_units")
if not call_units:
return
system_id = ctx.get("system_id")
now = ctx["now"]
unit_set = _unit_keys(call_units)
for inc in ctx.get("all_active") or []:
if inc.get("incident_id") == exclude_incident_id:
continue
if system_id and system_id not in (inc.get("system_ids") or []):
continue
matched = [u for u in (inc.get("units_active") or []) if _normalize_unit(u) in unit_set]
if not matched:
continue
units_active, units_cleared, auto_resolved = _apply_unit_clearance(inc, matched)
updates = {"units_active": units_active, "units_cleared": units_cleared}
if auto_resolved:
updates["status"] = "resolved"
updates["resolved_at"] = now.isoformat()
updates["resolved_via"] = "reassignment"
await fstore.doc_set("incidents", inc["incident_id"], updates)
logger.info(
f"Correlator: reassignment released unit(s) {matched} from incident "
f"{inc['incident_id']}" + (" (auto-resolved)" if auto_resolved else "")
)
if auto_resolved:
await maybe_resolve_parent(inc["incident_id"])
async def _update_incident(
inc: dict,
call_id: str,
@@ -1878,6 +2009,7 @@ async def _update_incident(
cleared_units: Optional[list[str]] = None,
refresh_activity: bool = True,
call_severity: Optional[str] = None,
call_srcaddr: Optional[str] = None,
) -> None:
incident_id = inc["incident_id"]
@@ -1896,19 +2028,24 @@ async def _update_incident(
merged_tags = list(dict.fromkeys((inc.get("tags") or []) + tags))
merged_units = list(dict.fromkeys((inc.get("units") or []) + call_units))
merged_vehicles = list(dict.fromkeys((inc.get("vehicles") or []) + call_vehicles))
# server-26#158: accumulate every radio ID that has transmitted on this
# incident, so a later thin call from the same radio can identity-match
# instead of guessing off recency alone.
merged_srcaddrs = list(dict.fromkeys(
(inc.get("srcaddrs") or []) + ([call_srcaddr] if call_srcaddr else [])
))
# Unit activity tracking: units_active / units_cleared
# units_active = units currently on scene; units_cleared = units back in service
units_active = list(inc.get("units_active") or [])
units_cleared = list(inc.get("units_cleared") or [])
tracked = _unit_keys(units_active) | _unit_keys(units_cleared)
for u in call_units:
if u not in units_cleared and u not in units_active:
if _is_trackable_unit(u) and _normalize_unit(u) not in tracked:
units_active.append(u)
for u in (cleared_units or []):
if u in units_active:
units_active.remove(u)
if u not in units_cleared:
units_cleared.append(u)
tracked.add(_normalize_unit(u))
inc_with_active_update = {**inc, "units_active": units_active, "units_cleared": units_cleared}
units_active, units_cleared, _ = _apply_unit_clearance(inc_with_active_update, cleared_units or [])
# The incident's label and its pin are resolved together, as one value.
location = clean_location(location)
@@ -1929,6 +2066,7 @@ async def _update_incident(
"tags": merged_tags,
"units": merged_units,
"vehicles": merged_vehicles,
"srcaddrs": merged_srcaddrs,
"units_active": units_active,
"units_cleared": units_cleared,
"location_mentions": location_mentions,
@@ -1968,6 +2106,7 @@ async def _update_incident(
if units_cleared and not units_active:
updates["status"] = "resolved"
updates["resolved_at"] = now.isoformat()
updates["resolved_via"] = "units_cleared"
await fstore.doc_set("incidents", incident_id, updates)
logger.info(
f"Correlator: signal-resolved incident {incident_id} "
@@ -1995,6 +2134,7 @@ async def _create_incident(
call_embedding: Optional[list],
call_severity: str,
now: datetime,
call_srcaddr: Optional[str] = None,
) -> str:
incident_id = str(uuid.uuid4())
tg_label = (
@@ -2035,9 +2175,10 @@ async def _create_incident(
"system_ids": [system_id] if system_id else [],
"tags": tags + ["auto-generated"],
"units": call_units,
"units_active": list(call_units),
"units_active": [u for u in call_units if _is_trackable_unit(u)],
"units_cleared": [],
"vehicles": call_vehicles,
"srcaddrs": [call_srcaddr] if call_srcaddr else [],
"severity": call_severity,
"summary": None,
"summary_stale": True,
@@ -2168,7 +2309,8 @@ async def maybe_resolve_parent(incident_id: str) -> None:
# All children resolved — close the master
await fstore.doc_set("incidents", parent_id, {
"status": "resolved",
"resolved_at": datetime.now(timezone.utc).isoformat(),
"resolved_at": clock.now().isoformat(),
"resolved_via": "children_resolved",
})
logger.info(
f"Auto-resolved master incident {parent_id} "
+206 -24
View File
@@ -15,6 +15,7 @@ import re
from typing import Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import ai_health
from app.internal import area_context
from app.internal.chatter_classifier import classify_chatter
# Location validity is defined once, by the module that owns the incident's
@@ -64,9 +65,9 @@ Response format — a JSON object with a "scenes" array. Each scene:
Rules:
- location: prefer intersections > addresses > mile markers > route+town > route alone > town alone. Dispatch-provided addresses take priority over unit-reported positions. Empty string if none.
- tags: describe WHAT happened, not WHERE. Specific, lowercase, hyphenated. Do not use location names, road names, talkgroup names, or place names as tags (wrong: "lower-macy's", "canvas-route-6", "route-202"; right: "suspect-search", "shoplifting", "vehicle-pursuit"). Do not repeat incident_type as a tag.
- units: ONLY identifiers that appear verbatim in the transcript. Use speaker role inference to distinguish units being dispatched from units acknowledging — both should be included. Never infer or guess unit IDs not present in the text.
- units: ONLY identifiers that appear verbatim in the transcript. Use speaker role inference to distinguish units being dispatched from units acknowledging — both should be included. Never infer or guess unit IDs not present in the text. If a unit ID format is given below, use it to recognise a unit spoken in a shortened or partial form (e.g. just the phonetic name alone) as the same unit — but still only extract what is actually said, never fabricate the full form.
- Do not invent details not present in the transcript.
- incident_type: let the talkgroup channel be your primary signal. Use "fire" ONLY if the talkgroup is clearly a fire/rescue channel OR the transcript explicitly describes active fire, smoke, flames, or structure fire activation. Police or EMS referencing a fire scene → use "police" or "ems". When the channel is a police channel and nothing in the transcript contradicts it, return "police" — do NOT fall back to "other" merely because the transmission is administrative. Reserve "other" for traffic that genuinely belongs to no emergency service (rail operations, public works, utility coordination). Reserve "unknown" for transcripts too garbled to place at all.
- incident_type: FIRST decide whether this transmission has any incident behind it at all, using the same bar as the "routine" severity rule below — pure administrative/status traffic with nothing describable happening: post/unit check-ins, roll call, bare acknowledgements ("10-4", "copy", "received"), records/report exchanges, "show me admin"/"show me available", a status ten-code with no event attached. If it is administrative/status-only, return "unknown" — this applies on EVERY channel, including a police channel; do not let the channel default override it (server-26#138: forcing a channel default onto content-free chatter is what let radio housekeeping open incidents). Only once real event content is present, let the talkgroup channel be your primary signal for WHICH type. Use "fire" ONLY if the talkgroup is clearly a fire/rescue channel OR the transcript explicitly describes active fire, smoke, flames, or structure fire activation. Police or EMS referencing a fire scene → use "police" or "ems". When the channel is a police channel, a real event is present, and nothing in the transcript contradicts it, return "police". Reserve "other" for a real event that genuinely belongs to no emergency service (rail operations, public works, utility coordination) — not for administrative chatter, which is "unknown" per above regardless of channel. Also reserve "unknown" for transcripts too garbled to place at all.
- severity: ALWAYS return one of the four values. Judge the underlying event, not how dramatic the words sound.
"routine" — administrative/status traffic with no incident behind it: mileage and transport logging, radio checks, acknowledgements, shift changes, track block/power requests, records lookups.
"minor" — a real but low-stakes call: lift assist, parking complaint, past-tense larceny report, noise complaint, welfare check.
@@ -74,18 +75,15 @@ Rules:
"major" — life safety or major property loss: structure fire, vehicle pursuit, shots fired, entrapment, cardiac arrest, officer needing assistance.
- ten_codes: interpret radio codes using the department reference provided below. Do not guess codes not listed.
- resolved: true only when the scene explicitly signals "Code 4", "all clear", "10-42", "in custody", "patient transported", "fire out", "GOA", "negative contact", "scene clear".
- cleared_units: only include units that explicitly stated their own back-in-service status in this recording (e.g. "Unit 7, 10-8", "Baker-1 available", "E-14 back in service", or the department ten-code for available/back-in-service listed above). Silence or absence of a unit is NOT clearance. A scene-wide Code 4 belongs in resolved=true, not here — cleared_units is for individual unit availability signals only.
- cleared_units: include a unit whose back-in-service/available status is stated in this recording — either the unit self-reporting (e.g. "Unit 7, 10-8", "Baker-1 available", "E-14 back in service", or the department ten-code for available/back-in-service listed above) OR dispatch confirming that SPECIFIC unit's status back to them (e.g. the unit asks "how do you show me" and dispatch replies "showing you available" / "in service"). The unit ID must be identifiable either way — a bare "clear" or "10-8" with no unit attached to it is NOT clearance; do not guess which unit said it. Silence or absence of a unit is NOT clearance. A scene-wide Code 4 belongs in resolved=true, not here — cleared_units is for individual unit availability signals only.
- reassignment: only true when a unit is explicitly being pulled to a completely new call or location. A unit going en route to their first dispatch is NOT a reassignment. Routine status updates, acknowledgements, and scene updates are NOT reassignments.
System: {system_id}
Talkgroup: {talkgroup_name}
{ten_codes_block}{vocabulary_block}{transcript_block}"""
{ten_codes_block}{vocabulary_block}{unit_format_block}{transcript_block}"""
# The incident_type enum offered to the model in EXTRACTION_PROMPT. Kept here
# rather than only in the prompt so a model that invents a value cannot write it
# into incident.type. "unknown" is deliberately absent — it is a real answer
# from the model but not a usable type, and is normalised to None alongside
# anything unrecognised.
# "unknown" is deliberately absent — normalises to None, which is what lets
# the creation gate veto a content-free call (server-26#138).
_VALID_INCIDENT_TYPES = frozenset({"fire", "ems", "police", "accident", "other"})
# Geographic bias radius for geocoding — half-width in degrees (~55 km)
@@ -156,6 +154,23 @@ def _build_ten_codes_block(ten_codes: dict[str, str]) -> str:
return f"Department ten-codes:\n{lines}\n\n"
def _build_unit_format_block(unit_format_hint: Optional[str]) -> str:
"""
server-26#<pending> — unit ID formats vary per department (e.g. Yorktown:
"<district>-<phonetic>", "5-David", sometimes spoken as bare "David";
County: "<location>-<number>", "SAM-1", "airport-3", "parks-4") with no
shared pattern across systems. Without a per-system hint, the model has
no way to recognise a unit ID it hasn't seen phrased that way before, and
that failure compounds into cleared_units and reassignment detection,
both of which depend on first recognising which token IS the unit.
Owner-authored free text per system (systems/{id}.unit_format_hint via
PUT /systems/{id}/unit-format) — no auto-induction yet.
"""
if not unit_format_hint:
return ""
return f"This system's unit ID format: {unit_format_hint}\n\n"
async def extract_scenes(
call_id: str,
transcript: str,
@@ -182,12 +197,15 @@ async def extract_scenes(
"""
vocabulary: list[str] = []
ten_codes: dict[str, str] = {}
unit_format_hint: str = ""
if system_id:
# Single cached read — vocabulary and ten_codes live on the same document.
# Single cached read — vocabulary, ten_codes and unit_format_hint all
# live on the same document.
system_doc = await fstore.doc_get_cached("systems", system_id)
if system_doc:
vocabulary = system_doc.get("vocabulary") or []
ten_codes = system_doc.get("ten_codes") or {}
unit_format_hint = system_doc.get("unit_format_hint") or ""
if _is_garbage_transcript(transcript):
logger.warning(
@@ -229,18 +247,26 @@ async def extract_scenes(
f"Intelligence: call {call_id} — transcript too short for extraction "
f"({len(transcript.split())} words), skipping"
)
cleared_unit = _short_clearance_unit(transcript)
try:
# Severity is still recorded: a five-word acknowledgement is genuinely
# routine traffic, and downstream code treats a missing severity as
# "not yet processed" rather than "nothing happened".
await fstore.doc_set("calls", call_id, {
updates = {
"skip_reason": "transcript_too_short",
"severity": "routine",
"chatter_classifier_verdict": chatter_is_chatter,
"chatter_classifier_reason": chatter_reason,
})
}
if cleared_unit:
updates["units"] = [cleared_unit]
updates["cleared_units"] = [cleared_unit]
await fstore.doc_set("calls", call_id, updates)
except Exception:
pass
if cleared_unit:
logger.info(f"Intelligence: call {call_id} — short clearance from {cleared_unit!r}")
return [_clearance_scene(transcript, cleared_unit)]
return []
try:
@@ -251,10 +277,26 @@ async def extract_scenes(
except Exception:
pass
try:
raw_scenes: list[dict] = await asyncio.to_thread(
_sync_extract,
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
unit_format_hint,
)
except Exception as e:
text = str(e)
kind = ai_health.classify(text)
logger.warning(f"GPT-4o-mini extraction failed for call {call_id}: {text}")
await ai_health.report_degraded(
"extraction", "openai", "gpt-4o-mini",
{"billing": "the OpenAI account is out of credit",
"dead_model": "model is unavailable"}.get(kind, f"extraction failed: {text[:200]}"),
{"billing": "Top up OpenAI billing",
"dead_model": "Update the extraction model in intelligence.py"}.get(kind, "Usually transient"),
permanent=kind != "transient",
)
return []
await ai_health.report_healthy("extraction")
if not raw_scenes:
return []
@@ -350,18 +392,20 @@ async def extract_scenes(
# the country.
location_coords: Optional[dict] = None
if location:
parts = [location]
if tg_area.get("municipality") or tg_area.get("county") or tg_area.get("state"):
parts += [tg_area[f] for f in area_context.PLACE_FIELDS if tg_area.get(f)]
elif node_lat is not None and node_lon is not None:
muni = _municipality_from_tg(talkgroup_name)
state = await _get_node_state(node_id or "", node_lat, node_lon) if node_id else ""
county = _node_county_cache.get(node_id or "") if node_id else ""
parts += [p for p in (muni, county, state) if p]
node_state, node_county = "", ""
if not area_context.has_place(tg_area) and node_id and node_lat is not None and node_lon is not None:
# Only worth the (cached-after-first-call) reverse-geocode
# when nothing better already describes this talkgroup.
node_state = await _get_node_state(node_id, node_lat, node_lon)
node_county = _node_county_cache.get(node_id) or ""
parts, tg_named_region = _location_query_parts(
location, tg_area, talkgroup_name, node_state, node_county,
)
query = ", ".join(parts)
if tg_anchor or (node_lat is not None and node_lon is not None):
location_coords = await _geocode_location(
query, node_lat, node_lon, anchor=tg_anchor
query, node_lat, node_lon, anchor=tg_anchor,
trust_named_region=tg_named_region,
)
# Embed this scene's content
@@ -433,6 +477,72 @@ async def extract_scenes(
return processed
# "45-9, I'm clear." / "Vehicle 1, clear." / "Car 12 10-8" — a unit reporting
# itself back in service is the one signal that ends an incident, and it is
# almost always five words or fewer, which is exactly the population the
# too-short skip above keeps away from GPT. In the first replay
# (server-26#170, 09-22 10:00-12:00 ET) 25 transmissions said 10-8/clear and
# 2 reached cleared_units. Rule-based on purpose: no model call, and only a
# unit named BEFORE the status word counts, so "10-8, 10-8." or "CMT clear."
# (no number) clears nobody rather than guessing.
_CLEAR_WORD_RE = re.compile(
r"\b(clear|10-?8|10-?98|back in service|in service|available)\b", re.IGNORECASE
)
_TEN_CODE_TOKEN_RE = re.compile(r"^10-?\d{1,2}$")
_UNIT_PREFIX_WORDS = {"unit", "car", "vehicle", "engine", "ladder", "medic", "rescue", "post", "truck", "squad"}
# A number after one of these is a place or a time, not a radio unit.
_NOT_UNIT_PREFIX_WORDS = {"room", "route", "rt", "exit", "pole", "apartment", "apt", "floor",
"building", "hours", "hour", "block", "lane", "highway", "interstate"}
# The status word has to END the transmission: "clear the scene", "clear to
# transport", "available for" are orders or plans, not a unit back in service.
_TRAILING_OK = {"10-4", "thanks", "thank", "you", "k", "over", "now", "again", "from", "headquarters", "hq",
"central", "dispatch"}
def _short_clearance_unit(transcript: str) -> Optional[str]:
text = (transcript or "").strip()
if not text or "?" in text:
return None # "45-9, are you clear?" asks; it doesn't report
m = _CLEAR_WORD_RE.search(text)
if not m:
return None
before = [t.strip(".,;:!") for t in text[: m.start()].split()]
before = [t for t in before if t]
after = [t.strip(".,;:!").lower() for t in text[m.end():].split()]
if any(t and t not in _TRAILING_OK for t in after):
return None
if any(t.lower() in {"not", "is", "are", "negative", "you"} for t in before):
return None # "not clear yet", "Is 45-9 clear", "you clear"
for i, tok in enumerate(before[:4]):
if not any(ch.isdigit() for ch in tok) or _TEN_CODE_TOKEN_RE.match(tok):
continue
prev = before[i - 1].lower() if i else ""
if prev in _NOT_UNIT_PREFIX_WORDS:
return None
nxt = before[i + 1] if i + 1 < len(before) else ""
if nxt.lower() in _NOT_UNIT_PREFIX_WORDS:
return None # "1400 hours, clear"
if nxt.isdigit():
tok = f"{tok}-{nxt}" # "45 9 clear" is unit 45-9, not unit 45
elif nxt.isalpha() and nxt[0].isupper() and nxt.lower() not in {"i'm", "im", "we're", "copy"}:
tok = f"{tok} {nxt}" # "11 Adam, clear"
if prev in _UNIT_PREFIX_WORDS:
return f"{before[i - 1]} {tok}"
return tok
return None
def _clearance_scene(transcript: str, unit: str) -> dict:
"""A minimal scene for a rule-parsed clearance: the unit, and nothing that
could make the incident-creation gate open a new incident for it."""
return {
"tags": [], "incident_type": None, "location": None, "location_coords": None,
"resolved": False, "severity": "routine", "vehicles": [], "units": [unit],
"cleared_units": [unit], "reassignment": False, "transcript": transcript,
"transcript_corrected": None, "segment_indices": [], "embedding": None,
}
def _geo_dist_km(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
"""Haversine distance in km between two lat/lon points."""
R = 6371.0
@@ -496,6 +606,7 @@ async def _geocode_location(
node_lat: Optional[float] = None,
node_lon: Optional[float] = None,
anchor: Optional[dict] = None,
trust_named_region: bool = False,
) -> Optional[dict]:
"""
Geocode using Google Maps Geocoding API, biased toward the channel's area.
@@ -511,6 +622,26 @@ async def _geocode_location(
talkgroup has a resolved anchor, that is the reference and its own radius is
the bound. Distance-from-node stays only as the fallback for a system nobody
has described yet — it was always a stand-in for this.
server-26#159: "a system nobody has described yet" turned out to include
systems that describe themselves — "New York City - NYPD Citywide 2 Patch"
names its own coverage area right in the talkgroup name, parsed into the
query by `_municipality_from_tg`, but a large aggregated/patched feed like
this is routinely received 40-70km from an antenna that happens to sit
wherever the node owner lives. Real, correctly-geocoded addresses on that
feed were being rejected by the node-distance check every single time —
location_coords stayed permanently null for the whole system, which killed
the location_proximity correlation signal and let duplicate incidents form
for the same event reported at two nearby addresses two minutes apart.
`trust_named_region` is True exactly when the query already carries a place
name that isn't the node's own position — operator-set area_context, or a
municipality parsed from the talkgroup's own name. In that case a distant
node is not evidence of a bad geocode, so the node-distance check is
skipped and precision is judged by `location_type` alone (still required
to be ROOFTOP/RANGE_INTERPOLATED/GEOMETRIC_CENTER, below). This does not
touch the anchor path at all — an anchor's own radius is always authoritative
when one has been resolved.
"""
import httpx
from app.config import settings
@@ -576,11 +707,21 @@ async def _geocode_location(
lat, lng = float(loc["lat"]), float(loc["lng"])
dist_km = _geo_dist_km(ref_lat, ref_lon, lat, lng)
if dist_km > max_km:
# server-26#159: the node-distance bound is a proxy for "is
# this plausible" that only makes sense when the node's own
# position is our best guess at the area — never when the
# query already names a different region on its own terms.
if not (ref_label == "node" and trust_named_region):
logger.warning(
f"Geocoding rejected '{location_str}' → ({lat:.4f}, {lng:.4f}) "
f"— {dist_km:.1f}km from {ref_label} exceeds {max_km:.1f}km"
)
return None
logger.info(
f"Geocoding '{location_str}' → ({lat:.4f}, {lng:.4f}) is "
f"{dist_km:.1f}km from the receiving node, past {max_km:.1f}km — "
f"accepted anyway: the query names its own region, not the node's"
)
coords = {"lat": lat, "lng": lng}
logger.info(
f"Geocoded '{location_str}' → {coords} "
@@ -606,6 +747,43 @@ def _municipality_from_tg(tg_name: Optional[str]) -> Optional[str]:
return cleaned
def _location_query_parts(
location: str,
tg_area: dict,
talkgroup_name: Optional[str],
node_state: str,
node_county: str,
) -> tuple[list[str], bool]:
"""
Build the geocode query parts for `location`, plus whether the query names
a region the *talkgroup itself* covers (operator-set area_context, or a
municipality parsed from the talkgroup's own name) rather than one guessed
from wherever the receiving node happens to sit (server-26#159).
That distinction matters downstream: `_geocode_location`'s node-distance
sanity check is only a valid proxy for "is this plausible" when the node's
own position is the best guess we have at the area. A citywide/patched
feed ("New York City - NYPD Citywide 2 Patch") names its own coverage area
right in the talkgroup name — grafting the node's own county onto that
(Ossining-style: valid when the feed genuinely is local to the node,
actively wrong when it names a distant region of its own) would make the
query self-contradictory, so the node's COUNTY is used only when nothing
better names the place. The node's STATE is coarse enough to still be
correct either way and is kept in both branches.
"""
parts = [location]
if area_context.has_place(tg_area):
parts += [tg_area[f] for f in area_context.PLACE_FIELDS if tg_area.get(f)]
return parts, True
muni = _municipality_from_tg(talkgroup_name)
if muni:
parts += [p for p in (muni, node_state) if p]
else:
parts += [p for p in (node_county, node_state) if p]
return parts, muni is not None
def _build_transcript_block(transcript: str, segments: Optional[list[dict]]) -> str:
"""Format transcript as numbered transmissions if segments are available."""
if segments and len(segments) > 1:
@@ -677,6 +855,7 @@ def _sync_extract(
segments: Optional[list[dict]],
vocabulary: Optional[list[str]] = None,
ten_codes: Optional[dict[str, str]] = None,
unit_format_hint: Optional[str] = None,
) -> list[dict]:
"""Call GPT-4o-mini and return a list of scene dicts."""
from app.config import settings
@@ -694,6 +873,7 @@ def _sync_extract(
system_id=system_id or "unknown",
ten_codes_block=_build_ten_codes_block(ten_codes or {}),
vocabulary_block=build_gpt_vocab_block(vocabulary or []),
unit_format_block=_build_unit_format_block(unit_format_hint),
)
try:
@@ -716,9 +896,11 @@ def _sync_extract(
except json.JSONDecodeError as e:
logger.warning(f"GPT-4o-mini returned non-JSON: {e}")
return []
except Exception as e:
logger.warning(f"GPT-4o-mini extraction failed: {e}")
return []
# Any other exception is the API call itself failing (no credit, rate
# limit, outage) and propagates to extract_scenes, which reports it to
# ai_health. Swallowing it here made "OpenAI is down" indistinguishable
# from "nothing happened on the radio" — the extraction tier existed in
# /health/ai but nothing ever reported to it.
def _sync_embed(text: str) -> Optional[list[float]]:
@@ -275,6 +275,12 @@ async def decide(call_id: str, ctx: dict) -> Optional[dict]:
if ctx["is_thin_call"]:
return None # thin calls have no transcript/units/coords to reason about
if _is_clearance_only(ctx):
# "45-9, I'm clear." carries one fact: which unit is done. Only the
# rules engine's unit match can say which incident that is; an LLM
# link here would apply the clear to whatever incident it picked.
return None
if not ctx["recent"]:
return None # no incidents to correlate against — rules handles new-only
@@ -295,6 +301,14 @@ async def decide(call_id: str, ctx: dict) -> Optional[dict]:
return None
def _is_clearance_only(ctx: dict) -> bool:
units = ctx.get("call_units") or []
cleared = ctx.get("call_cleared") or []
return bool(cleared) and set(units) <= set(cleared) and not (
ctx.get("tags") or ctx.get("location") or ctx.get("call_vehicles") or ctx.get("incident_type")
)
_dead_models: set[str] = set()
+7
View File
@@ -111,6 +111,8 @@ class MQTTHandler:
"assigned_system_id": None,
"approval_status": "pending",
"node_type": payload.get("node_type", "fixed"),
"secondary_sdr_mode": payload.get("secondary_sdr_mode", "none"),
"sdr_count": payload.get("sdr_count", 1),
"enforce_override_timeout": payload.get("enforce_override_timeout", True),
"is_overridden": False,
"override_system_id": None,
@@ -141,6 +143,11 @@ class MQTTHandler:
updates["node_type"] = node_type
updates["enforce_override_timeout"] = enforce_timeout
if "secondary_sdr_mode" in payload:
updates["secondary_sdr_mode"] = payload["secondary_sdr_mode"]
if "sdr_count" in payload:
updates["sdr_count"] = payload["sdr_count"]
if node_type == "portable":
updates["is_overridden"] = False
updates["override_system_id"] = None
@@ -20,6 +20,30 @@ from app.internal.logger import logger
from app.internal import firestore as fstore
from app.config import settings
# server-26#131: minimum time since the real-time pipeline (routers/upload.py
# _run_intelligence_pipeline) marked intelligence_started_at before the sweep
# will touch a call, even though it already looks orphaned. STT + scene
# extraction + correlation is a multi-second-to-low-minutes chain (Whisper,
# then a Gemini call per scene); without this buffer the sweep could pick up
# a call mid-pipeline — no incident_id/corr_path written yet — and correlate
# it a second time, independently, sometimes onto a different incident than
# the real-time path lands on. That race is what #131 found (same call in
# two incidents' call_ids, ~2% of linked calls). A call with no
# intelligence_started_at at all (pre-#131 call doc, or the marker write
# itself failed) is NOT held back by this — absence isn't evidence of an
# in-flight pipeline, and #131's own bug predates this field existing.
#
# 15, not 5: neither OpenAI's Whisper client nor Gemini's call in
# llm_correlator.py sets a request timeout (server-26#153), so a hung call can
# run well past a few minutes on SDK-default retries, and this constant is a
# guess against that unbounded tail, not a measured bound. Raising it costs
# nothing on the recovery side: a call that finished processing (linked OR
# genuinely orphaned) always has corr_path set (_apply_and_log writes it even
# on the orphan action), so it's already excluded by the
# `not c.get("corr_path")` filter below and never reaches this check at all —
# this constant only ever delays calls that are still actually running.
MIN_MINUTES_SINCE_PIPELINE_START = 15
# Standard link-only retry budget before a call is tombstoned corr_path="unlinked".
MAX_SWEEP_ATTEMPTS = 3
# server-26#115 — a call the consensus LLM-orphan gate parked (llm=orphan vs
@@ -52,8 +76,23 @@ async def recorrelation_loop() -> None:
logger.error(f"Re-correlation sweep failed: {e}")
def _pipeline_likely_still_running(call: dict, now: datetime) -> bool:
"""server-26#131 — True when the real-time pipeline marked
intelligence_started_at recently enough that it's probably still mid-flight
(STT / scene extraction / correlation), so the sweep should not race it.
No marker at all (older call doc, or the marker write itself failed)
returns False — absence isn't evidence of an in-flight pipeline."""
started = _parse_dt(call.get("intelligence_started_at"))
if not started:
return False
age_minutes = (now - started).total_seconds() / 60
return age_minutes < MIN_MINUTES_SINCE_PIPELINE_START
async def _run_sweep_pass() -> None:
cutoff = datetime.now(timezone.utc) - timedelta(minutes=settings.recorrelation_scan_minutes)
from app.internal import clock
now = clock.now()
cutoff = now - timedelta(minutes=settings.recorrelation_scan_minutes)
# Server-side range query: only calls that ended within the scan window.
# Filter incident_id=null client-side (Firestore can't query for missing fields).
@@ -77,6 +116,7 @@ async def _run_sweep_pass() -> None:
# a second route into the over-merge the thin fix above addresses.
and not c.get("skip_reason")
and c.get("corr_sweep_count", 0) < _max_sweep_attempts(c)
and not _pipeline_likely_still_running(c, now)
]
if not orphans:
+650
View File
@@ -0,0 +1,650 @@
"""
Replay — re-run the intelligence pipeline over past calls, in a sandbox.
Live AI windows are the only way the correlator has ever been measured, and
each one costs days of real time and whatever the credits allow: a change
ships, AI goes on, traffic trickles in, someone pulls a dump. Recordings are
kept whether AI is on or not, so the traffic to measure against already
exists. A replay run takes a time range of real calls, feeds them through
the SAME pipeline code the live upload path runs (routers/upload.py
`_extract_and_correlate`) in their original order with the clock pinned to
each call's own end time, and writes everything to
replay_runs/{run_id}/calls|incidents instead of the live collections. The
same range can then be replayed after every change and the runs compared.
Three modes, cheapest last:
audio re-transcribe the saved audio (Whisper + correction), then
extract and correlate. For ranges where AI was off.
transcripts reuse the transcript already on each call, re-run extraction
and correlation.
reuse reuse the scenes an earlier run extracted, re-run correlation
only. Extraction is an LLM call and never returns quite the
same thing twice, so this is the mode that isolates a
correlator change from extraction noise.
What never happens in a replay: alerts, summaries, vocabulary learning, and
any write to a live call or incident. The sandbox is enforced by the
ContextVar redirect in app/internal/firestore.py, not by this module
remembering to use different collection names.
One run at a time per process — a run spends real AI credits and its cost is
only estimated, so two concurrent runs would be two unbounded bills.
"""
import asyncio
import os
import statistics
import uuid
from collections import Counter
from datetime import datetime, timedelta, timezone
from typing import Optional
from app.config import settings
from app.internal import ai_health, clock
from app.internal import firestore as fstore
from app.internal.feature_flags import force_flags, unforce_flags
from app.internal.logger import logger
RUNS = "replay_runs"
MODES = ("audio", "transcripts", "reuse")
MAX_CALLS = 5000
MAX_RANGE_DAYS = 7
# Extraction/transcription run ahead of correlation with this much
# concurrency. They depend only on the call itself; correlation depends on
# every call before it and is kept strictly in order.
PREFETCH = 6
# Rough per-unit AI prices for the pre-run estimate and the running tally.
# Estimates, not a bill — nothing in DRB reads a real invoice (server-26#45).
USD_WHISPER_PER_MIN = 0.006
USD_PER_EXTRACTION = 0.0005 # gpt-4o-mini scene extraction + embedding
USD_PER_CORRECTION = 0.0003 # Gemini flash transcript correction
USD_PER_GEOCODE = 0.005 # Google geocode, roughly one per located scene
USD_PER_LLM_CORRELATE = 0.0005 # Gemini flash consensus decision
# Fields the pipeline writes onto a call doc. Stripped when a call is copied
# into the sandbox so the replay recomputes them instead of inheriting the
# live answer. Anything else on the doc (ids, times, talkgroup, srcaddr, audio
# location) is an input and is kept.
_DERIVED = {
"transcript", "transcript_corrected", "transcript_not_speech",
"segments", "segments_corrected", "scenes", "incident_id", "incident_ids",
"tags", "location", "location_coords", "location_mentions", "units",
"vehicles", "cleared_units", "severity", "incident_type", "type",
"embedding", "skip_reason", "intelligence_started_at", "reassignment",
"resolved", "has_updates", "audio_url",
}
_DERIVED_PREFIXES = ("corr_", "chatter_classifier_", "eval_")
_TRANSCRIPT_FIELDS = (
"transcript", "transcript_corrected", "transcript_not_speech",
"segments", "segments_corrected",
)
_active_run_id: Optional[str] = None
_active_task: Optional[asyncio.Task] = None
_cancel: set[str] = set()
class ReplayBusy(RuntimeError):
pass
def sandbox_root(run_id: str) -> str:
return f"{RUNS}/{run_id}"
def _scenes_coll(run_id: str) -> str:
# Extracted scenes (embeddings included) live beside the sandbox, not on
# its call docs, so reading a run's calls for metrics or the incident view
# doesn't haul every scene's embedding along a second time.
return f"{sandbox_root(run_id)}/scenes"
def active_run_id() -> Optional[str]:
if _active_task is not None and not _active_task.done():
return _active_run_id
return None
# ---------------------------------------------------------------------------
# Call selection
# ---------------------------------------------------------------------------
def _as_dt(value) -> Optional[datetime]:
if value is None:
return None
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
try:
dt = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
except ValueError:
return None
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
async def select_calls(
org_id: str,
date_from: datetime,
date_to: datetime,
system_ids: Optional[list[str]] = None,
cap: int = MAX_CALLS,
) -> tuple[list[dict], bool]:
"""
Live calls in [date_from, date_to] for this org, oldest first.
Pages newest-first because the one composite index on calls that carries
org_id is (org_id ASC, started_at DESC); ordering the other way would need
a new index for no gain. Returns (calls, truncated) — truncated means the
range holds more than `cap` calls and the caller must narrow it rather
than silently replaying only part of it.
"""
out: list[dict] = []
cursor = None
page = 1000
while True:
rows = await fstore.collection_where(
"calls",
[("org_id", "==", org_id),
("started_at", ">=", date_from),
("started_at", "<=", date_to)],
order_by=[("started_at", "DESCENDING")],
limit_to=page,
start_after={"started_at": cursor} if cursor is not None else None,
)
for c in rows:
if c.get("duplicate_of"):
continue # another node's copy — live never processes these either
if system_ids and c.get("system_id") not in system_ids:
continue
out.append(c)
if len(out) > cap:
return sorted(out[:cap], key=_call_time), True
if len(rows) < page:
break
cursor = rows[-1].get("started_at")
return sorted(out, key=_call_time), False
def _call_time(call: dict) -> datetime:
return _as_dt(call.get("started_at")) or datetime.min.replace(tzinfo=timezone.utc)
def _pipeline_time(call: dict) -> datetime:
"""When the live pipeline would have run for this call: at upload, i.e. call end."""
return _as_dt(call.get("ended_at")) or _call_time(call)
def _duration_s(call: dict) -> float:
# Call docs carry no duration field; the node reports start and end.
start, end = _as_dt(call.get("started_at")), _as_dt(call.get("ended_at"))
return max(0.0, (end - start).total_seconds()) if start and end else 0.0
def estimate(calls: list[dict], mode: str) -> dict:
n = len(calls)
with_transcript = sum(1 for c in calls if c.get("transcript_corrected") or c.get("transcript"))
audio_min = sum(_duration_s(c) for c in calls) / 60
with_audio = sum(1 for c in calls if c.get("audio_gcs_uri"))
# Roughly a third of calls carry a geocodable location (09-22 dump: 92/373).
per_call = USD_PER_EXTRACTION + USD_PER_LLM_CORRELATE + USD_PER_GEOCODE / 3
if mode == "audio":
usd = audio_min * USD_WHISPER_PER_MIN + with_audio * (USD_PER_CORRECTION + per_call)
elif mode == "transcripts":
usd = with_transcript * per_call
else:
usd = n * USD_PER_LLM_CORRELATE
return {
"calls": n,
"calls_with_transcript": with_transcript,
"calls_with_audio": with_audio,
"audio_minutes": round(audio_min, 1),
"est_cost_usd": round(usd, 2),
}
# ---------------------------------------------------------------------------
# Run lifecycle
# ---------------------------------------------------------------------------
async def start_run(
*,
org_id: str,
date_from: datetime,
date_to: datetime,
mode: str,
system_ids: Optional[list[str]],
source_run_id: Optional[str],
label: str,
actor: str,
) -> dict:
global _active_run_id, _active_task
if active_run_id():
raise ReplayBusy(f"Replay {active_run_id()} is still running.")
if mode not in MODES:
raise ValueError(f"mode must be one of {MODES}")
if date_to <= date_from:
raise ValueError("date_to must be after date_from")
if date_to - date_from > timedelta(days=MAX_RANGE_DAYS):
raise ValueError(f"Range is capped at {MAX_RANGE_DAYS} days.")
if mode == "reuse":
src = await fstore.doc_get(RUNS, source_run_id or "")
if not src or src.get("org_id") != org_id:
raise ValueError("reuse mode needs a source_run_id from an earlier run in this org")
if src.get("status") != "done":
raise ValueError("The source run did not finish; its scenes are incomplete.")
calls, truncated = await select_calls(org_id, date_from, date_to, system_ids)
if truncated:
raise ValueError(f"Range holds more than {MAX_CALLS} calls — narrow it.")
if not calls:
raise ValueError("No calls in that range.")
run_id = uuid.uuid4().hex[:12]
now = datetime.now(timezone.utc).isoformat()
doc = {
"run_id": run_id,
"org_id": org_id,
"label": label or "",
"mode": mode,
"source_run_id": source_run_id if mode == "reuse" else None,
"date_from": date_from.isoformat(),
"date_to": date_to.isoformat(),
"system_ids": system_ids or [],
"git_sha": os.getenv("GIT_SHA", "unknown"),
"created_by": actor,
"created_at": now,
"status": "running",
"estimate": estimate(calls, mode),
"progress": {"total": len(calls), "done": 0, "errors": 0},
"metrics": None,
"errors": [],
}
await fstore.doc_set(RUNS, run_id, doc, merge=False)
_active_run_id = run_id
_active_task = asyncio.create_task(_run(run_id, org_id, calls, mode, source_run_id))
return doc
def request_cancel(run_id: str) -> bool:
if active_run_id() != run_id:
return False
_cancel.add(run_id)
return True
async def get_run(run_id: str) -> Optional[dict]:
doc = await fstore.doc_get(RUNS, run_id)
return await _reconcile(doc) if doc else None
async def list_runs(org_id: str) -> list[dict]:
docs = await fstore.collection_list(RUNS, org_id=org_id)
docs = [await _reconcile(d) for d in docs]
return sorted(docs, key=lambda d: d.get("created_at") or "", reverse=True)
async def _reconcile(doc: dict) -> dict:
"""A run left "running" by a process that restarted (a deploy) never finishes."""
if doc.get("status") == "running" and doc.get("run_id") != active_run_id():
doc["status"] = "interrupted"
await fstore.doc_set(RUNS, doc["run_id"], {"status": "interrupted"})
return doc
async def delete_run(run_id: str) -> None:
if active_run_id() == run_id:
raise ReplayBusy("Cancel the run before deleting it.")
token = fstore.enter_sandbox(sandbox_root(run_id))
try:
for coll, key in (("calls", "call_id"), ("incidents", "incident_id"),
(_scenes_coll(run_id), "call_id")):
for d in await fstore.collection_list(coll):
if d.get(key):
await fstore.doc_delete(coll, d[key])
finally:
fstore.exit_sandbox(token)
await fstore.doc_delete(RUNS, run_id)
async def sandbox_contents(run_id: str) -> tuple[list[dict], list[dict]]:
token = fstore.enter_sandbox(sandbox_root(run_id))
try:
incidents = await fstore.collection_list("incidents")
calls = await fstore.collection_list("calls")
finally:
fstore.exit_sandbox(token)
return incidents, calls
# ---------------------------------------------------------------------------
# The run itself
# ---------------------------------------------------------------------------
def _flags_for(mode: str) -> dict[str, bool]:
return {
"stt_enabled": mode == "audio",
"transcript_correction_enabled": mode == "audio",
"correlation_enabled": True,
"summaries_enabled": False,
"vocabulary_learning_enabled": False,
}
def _stored_input(call: dict) -> tuple[Optional[str], list]:
"""
The transcript + segments live extraction was handed for this call.
Not simply `transcript_corrected`: live extraction overwrites that field
with its primary scene's rewrite (intelligence.py), so on a call with
several scenes it now holds only scene 0's text. The corrector's own
output survives intact in `segments_corrected`, so rebuild from those
when they exist; otherwise correction never produced anything and live
extraction read the raw Whisper transcript.
"""
if call.get("transcript_not_speech"):
return None, [] # transcribe_call hands nothing downstream for noise
corrected = call.get("segments_corrected") or []
if corrected:
text = " ".join(str(seg.get("text") or "").strip() for seg in corrected).strip()
return (text or call.get("transcript")), corrected
return call.get("transcript"), call.get("segments") or []
def _extraction_fields(sb_call: dict) -> dict:
"""What extraction wrote onto the call doc (tags, units, location,
embedding, skip_reason, ...), minus everything correlation wrote. A reuse
run restores these so the orphan sweep, which reads them straight off the
call doc, sees what it saw in the source run."""
return {
k: v for k, v in sb_call.items()
if (k in _DERIVED or k.startswith("chatter_classifier_"))
and k not in _TRANSCRIPT_FIELDS
and k not in ("scenes", "incident_id", "incident_ids", "intelligence_started_at")
}
def _sandbox_seed(call: dict, mode: str) -> dict:
keep_transcript = mode in ("transcripts", "reuse")
seed = {}
for k, v in call.items():
if k in _TRANSCRIPT_FIELDS:
if keep_transcript:
seed[k] = v
continue
if k in _DERIVED or k.startswith(_DERIVED_PREFIXES):
continue
seed[k] = v
# Calls are seeded ahead of the replay clock (see PREFETCH). The orphan
# re-correlation sweep selects status=="ended" calls by ended_at, so a
# seeded call keeping its real status would be swept up as an "orphan"
# before its own turn. Its real status is restored when it is processed.
seed["status"] = "replay_pending"
return seed
async def _prepare(call: dict, mode: str, source_scenes: dict[str, dict]) -> dict:
"""
Everything per call that doesn't depend on other calls: seed the sandbox
doc, then transcribe and/or extract. Runs ahead of correlation.
Returns {"transcript", "scenes", "skip"} for the in-order stage.
"""
from app.internal import intelligence, talkgroups, transcription
call_id = call["call_id"]
await fstore.doc_set("calls", call_id, _sandbox_seed(call, mode), merge=False)
talkgroup_name = await talkgroups.resolve(
call.get("system_id"), call.get("talkgroup_id"),
hint=call.get("talkgroup_name"), call_doc=call,
)
transcript: Optional[str] = None
segments: list = []
if mode == "audio":
if call.get("audio_gcs_uri"):
transcript, segments = await transcription.transcribe_call(
call_id, call["audio_gcs_uri"], talkgroup_name,
system_id=call.get("system_id"), talkgroup_id=call.get("talkgroup_id"),
)
else:
transcript, segments = _stored_input(call)
if mode == "reuse":
src = source_scenes.get(call_id)
if src is None:
return {"skip": "not_in_source_run", "talkgroup_name": talkgroup_name}
if src.get("call_fields"):
# skip_reason gates upload.py's no-scene fallback and the orphan
# sweep correlates from tags/units/location on the call doc, so
# extraction's call-level output comes across with its scenes.
await fstore.doc_set("calls", call_id, src["call_fields"])
return {"transcript": transcript, "scenes": src.get("scenes") or [],
"talkgroup_name": talkgroup_name}
scenes: list = []
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=call.get("talkgroup_id"), system_id=call.get("system_id"),
segments=segments, node_id=call.get("node_id"),
)
return {"transcript": transcript, "scenes": scenes, "talkgroup_name": talkgroup_name}
async def _sweeps_until(t: datetime, state: dict) -> None:
"""Run the live periodic sweeps (idle auto-resolve, orphan re-correlation) at every tick up to t."""
from app.internal import recorrelation_sweep, summarizer
interval = timedelta(minutes=settings.summary_interval_minutes)
if state["last_sweep"] is None:
state["last_sweep"] = t
return
while state["last_sweep"] + interval <= t:
state["last_sweep"] += interval
tok = clock.pin(state["last_sweep"])
try:
await summarizer._resolve_stale_incidents()
await recorrelation_sweep._run_sweep_pass()
finally:
clock.unpin(tok)
async def _run(run_id: str, org_id: str, calls: list[dict], mode: str,
source_run_id: Optional[str]) -> None:
from app.routers.upload import _extract_and_correlate
global _active_run_id
progress = {"total": len(calls), "done": 0, "errors": 0, "skipped": 0,
"extractions": 0, "audio_minutes": 0.0}
errors: list[str] = []
status = "done"
source_scenes: dict[str, dict] = {}
if mode == "reuse" and source_run_id:
rows = await fstore.collection_list(_scenes_coll(source_run_id))
source_scenes = {r["call_id"]: r for r in rows if r.get("call_id")}
sb_token = fstore.enter_sandbox(sandbox_root(run_id))
fl_token = force_flags(_flags_for(mode))
ai_failures: list = []
ai_token = ai_health.collect_sandbox_failures(ai_failures)
try:
sem = asyncio.Semaphore(PREFETCH)
async def prep(call: dict):
async with sem:
tok = clock.pin(_pipeline_time(call))
try:
return await _prepare(call, mode, source_scenes)
finally:
clock.unpin(tok)
pending: dict[int, asyncio.Task] = {}
sweep_state = {"last_sweep": None}
last_t = None
for i, call in enumerate(calls):
for j in range(i, min(i + PREFETCH * 2, len(calls))):
if j not in pending:
pending[j] = asyncio.create_task(prep(calls[j]))
if run_id in _cancel:
status = "cancelled"
break
fatal = _fatal_ai_failure(ai_failures)
if fatal:
# An unfunded or retired model fails every call the same way;
# finishing the run would only produce a sandbox of orphans
# that looks like a correlation result and isn't one.
status = "failed"
errors.append(f"aborted: {fatal}")
break
t = _pipeline_time(call)
last_t = t
try:
prepared = await pending.pop(i)
await _sweeps_until(t, sweep_state)
if prepared.get("skip"):
progress["skipped"] += 1
else:
tok = clock.pin(t)
try:
await fstore.doc_set("calls", call["call_id"], {
"status": call.get("status") or "ended",
"intelligence_started_at": t.isoformat(),
})
_, _, scenes = await _extract_and_correlate(
call_id=call["call_id"],
node_id=call.get("node_id"),
system_id=call.get("system_id"),
talkgroup_id=call.get("talkgroup_id"),
talkgroup_name=prepared["talkgroup_name"],
transcript=prepared["transcript"],
scenes=prepared["scenes"],
)
finally:
clock.unpin(tok)
# Kept whole (embedding included) so a later "reuse" run
# can correlate from exactly these scenes.
sb_call = await fstore.doc_get("calls", call["call_id"]) or {}
await fstore.doc_set(_scenes_coll(run_id), call["call_id"], {
"call_id": call["call_id"],
"scenes": scenes,
"call_fields": _extraction_fields(sb_call),
}, merge=False)
if prepared["transcript"] and mode != "reuse":
progress["extractions"] += 1
if mode == "audio":
progress["audio_minutes"] += _duration_s(call) / 60
except Exception as e:
progress["errors"] += 1
if len(errors) < 20:
errors.append(f"{call.get('call_id')}: {type(e).__name__}: {e}"[:300])
logger.warning(f"Replay {run_id}: call {call.get('call_id')} failed: {e}")
progress["done"] = i + 1
if (i + 1) % 25 == 0:
await fstore.doc_set(RUNS, run_id, {"progress": dict(progress), "errors": errors})
for task in pending.values():
task.cancel()
if status == "done" and last_t is not None:
# Let every incident age out exactly as it would have live.
await _sweeps_until(
last_t + timedelta(minutes=settings.incident_auto_resolve_minutes
+ 2 * settings.summary_interval_minutes),
sweep_state,
)
incidents = await fstore.collection_list("incidents")
sb_calls = await fstore.collection_list("calls")
metrics = compute_metrics(incidents, sb_calls)
metrics["est_cost_usd"] = _running_cost(progress, metrics, mode)
metrics["ai_failures"] = dict(Counter(f"{f['tier']}: {f['problem']}" for f in ai_failures))
except Exception as e:
status = "failed"
errors.append(f"run: {type(e).__name__}: {e}"[:300])
metrics = None
logger.error(f"Replay {run_id} failed: {e}")
finally:
ai_health._sandbox_failures.reset(ai_token)
unforce_flags(fl_token)
fstore.exit_sandbox(sb_token)
_cancel.discard(run_id)
_active_run_id = None
await fstore.doc_set(RUNS, run_id, {
"status": status,
"progress": progress,
"errors": errors,
"metrics": metrics,
"finished_at": datetime.now(timezone.utc).isoformat(),
})
logger.info(f"Replay {run_id} {status}: {progress}")
FATAL_AFTER = 5
def _fatal_ai_failure(failures: list) -> Optional[str]:
"""A tier that failed permanently (no credit, dead model) FATAL_AFTER times."""
permanent = Counter(
f"{f['tier']} ({f['provider']} {f['model']}): {f['problem']}"
for f in failures if f.get("permanent")
)
for what, n in permanent.items():
if n >= FATAL_AFTER:
return what
return None
def _running_cost(progress: dict, metrics: dict, mode: str) -> float:
usd = progress["audio_minutes"] * USD_WHISPER_PER_MIN
if mode == "audio":
usd += progress["extractions"] * USD_PER_CORRECTION
usd += progress["extractions"] * (USD_PER_EXTRACTION + USD_PER_GEOCODE / 3)
usd += metrics.get("llm_decisions", 0) * USD_PER_LLM_CORRELATE
return round(usd, 2)
# ---------------------------------------------------------------------------
# Scoring
# ---------------------------------------------------------------------------
def compute_metrics(incidents: list[dict], calls: list[dict]) -> dict:
"""
The numbers that say whether incidents are being tracked, from one run's
sandbox. Same questions every correlation review has asked by hand, so two
runs over the same range compare directly.
"""
sizes = [len(i.get("call_ids") or []) for i in incidents]
resolved_via = Counter(
(i.get("resolved_via") or ("unknown" if i.get("status") == "resolved" else "still_active"))
for i in incidents
)
corr_path: Counter = Counter()
consensus: Counter = Counter()
for c in calls:
scenes = c.get("scenes") or {}
records = [s.get("corr_debug") or {} for s in scenes.values()] if scenes else [c]
for r in records:
corr_path[r.get("corr_path") or "none"] += 1
consensus[r.get("corr_consensus") or "none"] += 1
linked = sum(1 for c in calls if c.get("incident_ids"))
llm = sum(n for k, n in consensus.items() if k not in ("none", "rules_only"))
return {
"calls": len(calls),
"calls_linked": linked,
"calls_orphaned": len(calls) - linked,
"incidents": len(incidents),
"single_call_incidents": sum(1 for s in sizes if s == 1),
"single_call_pct": round(100 * sum(1 for s in sizes if s == 1) / len(sizes), 1) if sizes else None,
"median_calls_per_incident": statistics.median(sizes) if sizes else None,
"max_calls_in_incident": max(sizes) if sizes else None,
"incidents_with_units_cleared": sum(1 for i in incidents if i.get("units_cleared")),
"incidents_with_coords": sum(1 for i in incidents if i.get("location_coords")),
"resolved_via": dict(resolved_via),
"corr_path": dict(corr_path),
"corr_consensus": dict(consensus),
"llm_decisions": llm,
}
+3 -1
View File
@@ -148,7 +148,8 @@ async def _resolve_stale_incidents() -> None:
if not all_active:
return
now = datetime.now(timezone.utc)
from app.internal import clock
now = clock.now()
cutoff = timedelta(minutes=settings.incident_auto_resolve_minutes)
count = 0
@@ -167,6 +168,7 @@ async def _resolve_stale_incidents() -> None:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": now.isoformat(),
"resolved_via": "idle_timeout",
})
from app.internal.incident_correlator import maybe_resolve_parent
await maybe_resolve_parent(incident_id)
@@ -28,10 +28,23 @@ counties may have one talkgroup covering a single municipality, and that
municipality's streets must not be buried under a county-wide list. A
single-municipality system is the degenerate case: populate the system level and
every talkgroup inherits it.
THE PROMPT'S OWN RULES ARE NOT ENFORCED (server-26#162). "Do NOT expand
ten-codes" and "NEVER add information" are instructions to the model, not
checks on its output — `correct()` used to accept `raw["corrected"]` verbatim.
Caught live: the same call came back with "10-7" rewritten to "10-13" in one
place and "10-4" in another, and "7" expanded into "ShotSpotter" — a real code
swapped for a different real code reads exactly as confident and trustworthy
as a correct one, which is worse than leaving the raw mishearing in place. The
model isn't graded on this at write time; `_code_tokens()` is a
verify-what-you-can-cheaply-check backstop, not a fix to the model's judgment:
it only catches a code-shaped token changing, not a wrong word substituted for
another equally plausible word.
"""
import asyncio
import json
import re
from typing import Any, Optional
from app.config import settings
@@ -96,6 +109,19 @@ def _dedupe(items: list[str]) -> list[str]:
return out
# Ten-codes ("10-4"), unit/signal shorthand ("4-2"), and the digit-group
# fragments radio traffic reads out loud ("7-2-1" of a case number) all share
# this shape. The guard below does not need to know which of those a given
# token is — it only needs the SET of them to survive a "correction"
# unchanged, in order. A model rewriting "10-7" as "10-13" is not the kind of
# mishearing this pass exists to fix (server-26#162).
_CODE_TOKEN_RE = re.compile(r"\b\d{1,3}(?:-\d{1,3})+\b")
def _code_tokens(text: str) -> list[str]:
return _CODE_TOKEN_RE.findall(text or "")
def _talkgroup_entry(system_doc: dict, talkgroup_id: Optional[int]) -> dict:
"""The config.talkgroups[] entry for this talkgroup, or {}."""
if talkgroup_id is None:
@@ -317,6 +343,31 @@ async def correct(
if verified_segments:
corrected_segments = verified_segments
# server-26#162: a code-shaped token ("10-7", "4-2", a case-number
# fragment like "7-2-1") changing at all — not just going missing, any
# change — means the model touched something this pass has no business
# touching. Reject that half of the correction outright rather than trust
# a rewrite that already broke its own instructions once. Checked against
# the ORIGINAL text/segment, not each other, so a joined-text correction
# and a segment correction are judged independently, same as everywhere
# else in this function.
if corrected is not None and _code_tokens(corrected) != _code_tokens(text):
logger.warning(
f"Transcript correction for call {call_id} changed code-shaped "
f"tokens ({_code_tokens(text)} -> {_code_tokens(corrected)}) — "
f"discarding the joined correction"
)
corrected = None
if corrected_segments is not None:
for seg, orig in zip(corrected_segments, segments or []):
if _code_tokens(seg["text"]) != _code_tokens(orig.get("text", "")):
logger.warning(
f"Transcript correction for call {call_id} changed "
f"code-shaped tokens in a segment — discarding segment corrections"
)
corrected_segments = None
break
if corrected or corrected_segments or not_speech:
changed = raw.get("changed") or []
logger.info(
+44
View File
@@ -0,0 +1,44 @@
"""
Word error rate — server-26#163's eval harness needs a real number to compare
against, not a vibe. Standard definition: word-level Levenshtein distance
between a human-verified reference and the machine hypothesis, divided by the
reference's own word count. Case-insensitive, punctuation-insensitive — this
measures whether the right WORDS came out, not transcript formatting.
"""
import re
from typing import Optional
def _tokenize(text: str) -> list[str]:
return re.findall(r"[\w']+", (text or "").lower())
def word_error_rate(reference: str, hypothesis: str) -> Optional[float]:
"""
(substitutions + deletions + insertions) / len(reference words).
None when the reference has no words — WER is undefined there, not 0.0;
a caller that defaults a None to 0.0 would report a perfect score for a
call nobody actually transcribed.
"""
ref = _tokenize(reference)
hyp = _tokenize(hypothesis)
if not ref:
return None
if not hyp:
return 1.0
n, m = len(ref), len(hyp)
# Single-row DP over Levenshtein distance — O(n*m) time, O(m) space.
row = list(range(m + 1))
for i in range(1, n + 1):
prev_diag = row[0]
row[0] = i
for j in range(1, m + 1):
prev_row_j = row[j]
if ref[i - 1] == hyp[j - 1]:
row[j] = prev_diag
else:
row[j] = 1 + min(prev_diag, row[j], row[j - 1])
prev_diag = prev_row_j
return row[m] / n
+3 -1
View File
@@ -17,7 +17,7 @@ from app.internal.auth import (
require_node_service_or_firebase_token,
)
from app.routers import nodes, systems, calls, upload, tokens, incidents, alerts, admin, trips, places, links, users
from app.routers import enrollment, media, org, waitlist
from app.routers import enrollment, media, org, waitlist, telemetry, replay
from app.internal import dynsec
from app.internal import firestore as fstore
@@ -120,6 +120,7 @@ app.include_router(nodes.router, dependencies=[Depends(require_service_or_fi
# write routes inside carry their own require_admin_token, so nodes get read
# access only.
app.include_router(systems.router, dependencies=[Depends(require_node_service_or_firebase_token)])
app.include_router(telemetry.router, dependencies=[Depends(require_node_service_or_firebase_token)])
app.include_router(calls.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(tokens.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(incidents.router, dependencies=[Depends(require_service_or_firebase_token)])
@@ -128,6 +129,7 @@ app.include_router(trips.router, dependencies=[Depends(require_service_or_fi
app.include_router(places.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(upload.router) # auth is per-node, handled inline
app.include_router(admin.router) # auth is per-endpoint (read: firebase, write: admin)
app.include_router(replay.router) # auth: admin only (every route spends or reads a replay run)
app.include_router(users.router) # auth: admin only
app.include_router(links.router) # auth is per-endpoint (generate: firebase, resolve: service key)
app.include_router(enrollment.router) # public; auth is the enrollment/pickup-secret tokens, checked inline
+31
View File
@@ -62,12 +62,43 @@ class NodeRecord(BaseModel):
last_seen: Optional[datetime] = None
assigned_system_id: Optional[str] = None
node_type: str = "fixed" # fixed or portable
secondary_sdr_mode: str = "none" # none | adsb | ais | op25_2 — requires a second physical SDR
sdr_count: int = 1 # self-reported by the node's checkin, best-effort
enforce_override_timeout: bool = True
is_overridden: bool = False
override_system_id: Optional[str] = None
override_timeout_at: Optional[datetime] = None
class AircraftTrack(BaseModel):
"""Live ADS-B position, one doc per icao. Overwritten on every sighting —
this is a live-map snapshot, not a history (see node-26#9)."""
icao: str
org_id: Optional[str] = None
node_id: str
callsign: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
altitude_ft: Optional[float] = None
ground_speed_kt: Optional[float] = None
track_deg: Optional[float] = None
last_seen: datetime
class VesselTrack(BaseModel):
"""Live AIS position, one doc per mmsi. Same live-snapshot shape as
AircraftTrack — overwritten on every sighting (see node-26#9)."""
mmsi: str
org_id: Optional[str] = None
node_id: str
name: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
speed_kt: Optional[float] = None
heading_deg: Optional[float] = None
last_seen: datetime
class CommandPayload(BaseModel):
action: str # discord_join / discord_leave / op25_restart
guild_id: Optional[str] = None
+191 -10
View File
@@ -5,6 +5,7 @@ from typing import Optional
from app.internal import firestore as fstore
from app.internal.auth import (
require_admin_token,
require_firebase_token,
require_service_or_firebase_token,
resolve_caller_org_id,
reprocess_limiter,
@@ -15,9 +16,49 @@ from app.internal.storage import gcs_uri_for_call, with_playback_url
class TranscriptUpdate(BaseModel):
transcript: str
class EvalTranscriptUpdate(BaseModel):
text: str
router = APIRouter(prefix="/calls", tags=["calls"])
def _parse_ts(value: Optional[str], field: str) -> Optional[datetime]:
"""ISO string from a query param → aware datetime, or 400.
started_at is stored as a Firestore timestamp, so a cursor or range bound
passed through as the raw string compares by *type* (every string sorts
after every timestamp) rather than by time — a string cursor made "Load
more" return the first page again.
"""
if not value:
return None
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
raise HTTPException(400, f"{field} is not an ISO-8601 timestamp.")
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
def _next_cursor(rows: list[dict], matches: list[dict], page: list[dict], window: int) -> Optional[str]:
"""Where the next page of a bounded-window scan starts.
More matches than fit on the page → resume right after the last row
returned, or every match between it and the end of the window is skipped
(a 200-row window shown 50 at a time lost 150 calls per "Load more").
Otherwise resume after the last row SCANNED, not the last match — a page
whose last match sits early in the window would re-scan everything after
it and loop forever on a sparse filter. A short window is the end.
"""
if len(matches) > len(page):
last = page[-1].get("started_at")
elif len(rows) == window:
last = rows[-1].get("started_at")
else:
return None
return last.isoformat() if hasattr(last, "isoformat") else last
@router.get("")
async def list_calls(
node_id: Optional[str] = Query(None),
@@ -50,7 +91,9 @@ async def search_calls(
link: str = Query("any", pattern="^(any|orphan|linked)$"),
transcript: str = Query("any", pattern="^(any|yes|no)$"),
q: Optional[str] = Query(None, description="case-insensitive substring of the transcript"),
decoded: dict = Depends(require_admin_token),
date_from: Optional[str] = Query(None, description="ISO timestamp, inclusive lower bound on started_at"),
date_to: Optional[str] = Query(None, description="ISO timestamp, inclusive upper bound on started_at"),
decoded: dict = Depends(require_firebase_token),
):
"""
Paged, filterable call archive — the backend for the /calls page.
@@ -68,6 +111,11 @@ async def search_calls(
`window_exhausted` says the scan hit its cap before filling the page, so an
empty result means "not in this window", not "none exist".
Open to every org member (viewer included), not just admins: the Firestore
rules already let any member read every call doc in their org
(firestore.rules `calls` → docInMyOrg), so this route exposes nothing a
viewer's browser couldn't already read directly.
"""
org_id = await resolve_caller_org_id(decoded)
if org_id is None:
@@ -78,13 +126,24 @@ async def search_calls(
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
cursor_dt = _parse_ts(cursor, "cursor")
from_dt = _parse_ts(date_from, "date_from")
to_dt = _parse_ts(date_to, "date_to")
# A range on the ordered field rides the same org_id/started_at index.
conditions: list[tuple[str, str, object]] = [("org_id", "==", org_id)]
if from_dt:
conditions.append(("started_at", ">=", from_dt))
if to_dt:
conditions.append(("started_at", "<=", to_dt))
window = max(limit * 10, 200)
rows = await fstore.collection_where(
"calls",
[("org_id", "==", org_id)],
conditions,
order_by=[("started_at", "DESCENDING")],
limit_to=window,
start_after={"started_at": cursor} if cursor else None,
start_after={"started_at": cursor_dt} if cursor_dt else None,
)
needle = (q or "").strip().lower()
@@ -113,13 +172,7 @@ async def search_calls(
matches = [c for c in rows if _keep(c)]
page = matches[:limit]
# Cursor advances over the SCANNED window, not the filtered page — otherwise
# a page whose last match sits early in the window would re-scan everything
# after it on the next request and loop forever on a sparse filter.
next_cursor = None
if len(rows) == window:
last_scanned = rows[-1].get("started_at")
next_cursor = last_scanned.isoformat() if hasattr(last_scanned, "isoformat") else last_scanned
next_cursor = _next_cursor(rows, matches, page, window)
return {
"calls": [with_playback_url(c) for c in page],
@@ -130,6 +183,107 @@ async def search_calls(
}
@router.get("/eval-queue")
async def eval_queue(
limit: int = Query(5, ge=1, le=20),
cursor: Optional[str] = Query(None, description="started_at of the last row of the previous page"),
decoded: dict = Depends(require_admin_token),
):
"""
A batch of calls that have a machine transcript but no human-verified one
yet — the backend for the STT eval page (server-26#163).
Deliberately separate from `PATCH /{call_id}/transcript`: that route is a
PRODUCTION correction — it re-runs extraction, unlinks incidents, and
feeds the vocabulary learner. An eval annotation must never trigger any
of that; it only exists to measure the pipeline, not to change what it
already decided. `eval_transcript` lives next to `transcript`/
`transcript_corrected` on the call doc and nothing downstream reads it.
Same bounded-window-scan-plus-cursor shape as `/search`, for the same
reason: no composite index exists for "eval_transcript is unset", and one
scan ordered by started_at is already trusted here. Paging through with
the returned cursor is how "however many, over time" actually works —
each call is where the last session left off, not a fresh random sample.
"""
org_id = await resolve_caller_org_id(decoded)
if org_id is None:
org_id = decoded.get("org_id")
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
cursor_dt = _parse_ts(cursor, "cursor")
window = max(limit * 20, 300)
rows = await fstore.collection_where(
"calls",
[("org_id", "==", org_id)],
order_by=[("started_at", "DESCENDING")],
limit_to=window,
start_after={"started_at": cursor_dt} if cursor_dt else None,
)
def _eligible(c: dict) -> bool:
text = c.get("transcript_corrected") or c.get("transcript") or ""
return bool(text) and not c.get("eval_transcript")
matches = [c for c in rows if _eligible(c)]
page = matches[:limit]
next_cursor = _next_cursor(rows, matches, page, window)
return {
"calls": [with_playback_url(c) for c in page],
"next_cursor": next_cursor,
"scanned": len(rows),
"matched": len(matches),
"window_exhausted": len(rows) == window,
}
@router.get("/eval-stats")
async def eval_stats(decoded: dict = Depends(require_admin_token)):
"""
How many calls have a human-verified transcript, and the WER of the raw
and corrected machine transcripts against them (server-26#163).
Whole-collection scan, matching `GET /calls` (list_calls above) rather
than the bounded-window pattern the paged routes use: the eval set this
is measuring is built a few calls at a time and expected to stay small
(tens to hundreds), so a full scan filtered in Python is the honest
answer rather than a windowed guess that could miss eval'd calls sitting
outside a recency window.
"""
from app.internal.wer import word_error_rate
org_id = await resolve_caller_org_id(decoded)
filters = {"org_id": org_id} if org_id is not None else {}
calls = await fstore.collection_list("calls", **filters)
raw_wers: list[float] = []
corrected_wers: list[float] = []
for c in calls:
ref = c.get("eval_transcript")
if not ref:
continue
raw = c.get("transcript") or ""
corrected = c.get("transcript_corrected") or raw
raw_wer = word_error_rate(ref, raw)
corrected_wer = word_error_rate(ref, corrected)
if raw_wer is not None:
raw_wers.append(raw_wer)
if corrected_wer is not None:
corrected_wers.append(corrected_wer)
def _avg(xs: list[float]) -> Optional[float]:
return round(sum(xs) / len(xs), 4) if xs else None
return {
"eval_count": len(raw_wers),
"raw_wer": _avg(raw_wers),
"corrected_wer": _avg(corrected_wers),
}
@router.get("/{call_id}")
async def get_call(call_id: str, decoded: dict = Depends(require_service_or_firebase_token)):
call = await fstore.doc_get("calls", call_id)
@@ -289,6 +443,7 @@ async def patch_transcript(
"call_ids": [],
"status": "resolved",
"resolved_at": datetime.now(timezone.utc).isoformat(),
"resolved_via": "emptied_by_correction",
"summary_stale": True,
})
await fstore.doc_set("calls", call_id, {"incident_ids": [], "incident_id": None})
@@ -313,3 +468,29 @@ async def patch_transcript(
preserve_transcript_correction=True,
)
return {"ok": True, "call_id": call_id}
@router.put("/{call_id}/eval-transcript")
async def put_eval_transcript(
call_id: str,
body: EvalTranscriptUpdate,
decoded: dict = Depends(require_admin_token),
):
"""
Record a human-verified reference transcript for the STT eval harness
(server-26#163). Pure data capture — unlike `PATCH /{call_id}/transcript`
above, this never touches `transcript`/`transcript_corrected`, never
re-runs extraction, never unlinks incidents, and never feeds the
vocabulary learner. It exists to MEASURE the pipeline's output, not to
change it; the two must not share a code path.
"""
call = await fstore.doc_get("calls", call_id)
if not call:
raise HTTPException(404, f"Call '{call_id}' not found.")
await fstore.doc_set("calls", call_id, {
"eval_transcript": body.text,
"eval_transcript_by": decoded.get("email") or decoded.get("uid"),
"eval_transcript_at": datetime.now(timezone.utc).isoformat(),
})
return {"ok": True, "call_id": call_id}
+1
View File
@@ -170,6 +170,7 @@ async def unlink_call_from_incident(incident_id: str, call_id: str, _: dict = De
if not remaining:
updates["status"] = "resolved"
updates["resolved_at"] = datetime.now(timezone.utc).isoformat()
updates["resolved_via"] = "emptied_by_admin"
await fstore.doc_update("incidents", incident_id, updates)
call = await fstore.doc_get("calls", call_id)
+3
View File
@@ -195,6 +195,7 @@ async def assign_system(
class NodeUpdateBody(BaseModel):
node_type: Optional[str] = None
enforce_override_timeout: Optional[bool] = None
secondary_sdr_mode: Optional[str] = None # none | adsb | ais | op25_2
@router.patch("/{node_id}")
@@ -227,6 +228,8 @@ async def update_node(
}
if updated_node.get("ppm_override") is not None:
push_payload["ppm_override"] = updated_node["ppm_override"]
if updated_node.get("secondary_sdr_mode") is not None:
push_payload["secondary_sdr_mode"] = updated_node["secondary_sdr_mode"]
mqtt_handler.push_config(node_id, push_payload)
return {"ok": True}
+183
View File
@@ -0,0 +1,183 @@
"""
Admin replay routes — the backend for the /admin Replay tab.
See app/internal/replay.py for what a run is and why it exists. Every route is
admin-only: a run spends real AI credits.
"""
from datetime import datetime, timezone
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel
from app.internal import replay
from app.internal.audit import write_audit
from app.internal.auth import describe_actor, require_admin_token, resolve_caller_org_id
from app.internal.logger import logger
router = APIRouter(prefix="/admin/replay", tags=["admin"])
def _parse_ts(value: Optional[str], field: str) -> datetime:
if not value:
raise HTTPException(400, f"{field} is required.")
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
raise HTTPException(400, f"{field} is not an ISO-8601 timestamp.")
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
async def _org(decoded: dict) -> str:
# Same fallback as /calls/search: a platform admin resolves to "every
# org", which is not a scope a replay can run in.
org_id = await resolve_caller_org_id(decoded) or decoded.get("org_id")
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
return org_id
async def _own_run(run_id: str, org_id: str) -> dict:
run = await replay.get_run(run_id)
if not run or run.get("org_id") != org_id:
raise HTTPException(404, f"Replay run '{run_id}' not found.")
return run
@router.get("/estimate")
async def estimate_run(
date_from: str = Query(...),
date_to: str = Query(...),
mode: Literal["audio", "transcripts", "reuse"] = Query("transcripts"),
system_ids: Optional[str] = Query(None, description="comma-separated"),
decoded: dict = Depends(require_admin_token),
):
"""How many calls a run over this range would process, and a rough cost."""
org_id = await _org(decoded)
sids = [s for s in (system_ids or "").split(",") if s] or None
calls, truncated = await replay.select_calls(
org_id, _parse_ts(date_from, "date_from"), _parse_ts(date_to, "date_to"), sids,
)
return {**replay.estimate(calls, mode), "truncated": truncated, "max_calls": replay.MAX_CALLS}
class StartRun(BaseModel):
date_from: str
date_to: str
mode: Literal["audio", "transcripts", "reuse"] = "transcripts"
system_ids: Optional[list[str]] = None
source_run_id: Optional[str] = None
label: str = ""
@router.post("")
async def start_run(body: StartRun, decoded: dict = Depends(require_admin_token)):
org_id = await _org(decoded)
actor_uid, actor_email = describe_actor(decoded)
try:
run = await replay.start_run(
org_id=org_id,
date_from=_parse_ts(body.date_from, "date_from"),
date_to=_parse_ts(body.date_to, "date_to"),
mode=body.mode,
system_ids=body.system_ids or None,
source_run_id=body.source_run_id,
label=body.label[:120],
actor=actor_email or actor_uid,
)
except replay.ReplayBusy as e:
raise HTTPException(409, str(e))
except ValueError as e:
raise HTTPException(400, str(e))
try:
await write_audit(actor_uid, actor_email, "replay.start", details={
"run_id": run["run_id"], "mode": run["mode"], "calls": run["progress"]["total"],
"est_cost_usd": run["estimate"]["est_cost_usd"],
})
except Exception as e:
logger.error(f"Replay: audit write failed ({e}) — run {run['run_id']} continues")
return run
@router.get("")
async def list_runs(decoded: dict = Depends(require_admin_token)):
org_id = await _org(decoded)
return {"runs": await replay.list_runs(org_id), "active_run_id": replay.active_run_id()}
@router.get("/{run_id}")
async def get_run(run_id: str, decoded: dict = Depends(require_admin_token)):
return await _own_run(run_id, await _org(decoded))
@router.post("/{run_id}/cancel")
async def cancel_run(run_id: str, decoded: dict = Depends(require_admin_token)):
await _own_run(run_id, await _org(decoded))
if not replay.request_cancel(run_id):
raise HTTPException(409, "That run is not running.")
return {"ok": True}
@router.delete("/{run_id}")
async def delete_run(run_id: str, decoded: dict = Depends(require_admin_token)):
await _own_run(run_id, await _org(decoded))
try:
await replay.delete_run(run_id)
except replay.ReplayBusy as e:
raise HTTPException(409, str(e))
return {"ok": True}
def _call_row(c: dict) -> dict:
scenes = c.get("scenes") or {}
paths = [((s.get("corr_debug") or {}).get("corr_path")) for _, s in sorted(scenes.items())]
return {
"call_id": c.get("call_id"),
"started_at": c.get("started_at"),
"talkgroup_name": c.get("talkgroup_name"),
"transcript": c.get("transcript_corrected") or c.get("transcript"),
"units": c.get("units"),
"cleared_units": c.get("cleared_units"),
"location": c.get("location"),
"skip_reason": c.get("skip_reason"),
"srcaddr": c.get("srcaddr"),
"corr_path": [p for p in paths if p] or ([c["corr_path"]] if c.get("corr_path") else []),
"incident_ids": c.get("incident_ids") or [],
}
@router.get("/{run_id}/incidents")
async def run_incidents(run_id: str, decoded: dict = Depends(require_admin_token)):
"""
A run's sandbox, shaped for reading: every incident with its calls in
order, plus the calls that never linked. Embeddings stay out.
"""
await _own_run(run_id, await _org(decoded))
incidents, calls = await replay.sandbox_contents(run_id)
by_id = {c.get("call_id"): _call_row(c) for c in calls}
out = []
for inc in sorted(incidents, key=lambda i: str(i.get("started_at") or "")):
rows = [by_id[cid] for cid in (inc.get("call_ids") or []) if cid in by_id]
rows.sort(key=lambda r: str(r["started_at"] or ""))
out.append({
"incident_id": inc.get("incident_id"),
"title": inc.get("title"),
"type": inc.get("type"),
"severity": inc.get("severity"),
"status": inc.get("status"),
"resolved_via": inc.get("resolved_via"),
"started_at": inc.get("started_at"),
"updated_at": inc.get("updated_at"),
"resolved_at": inc.get("resolved_at"),
"location": inc.get("location"),
"location_coords": inc.get("location_coords"),
"units": inc.get("units"),
"units_active": inc.get("units_active"),
"units_cleared": inc.get("units_cleared"),
"talkgroup_ids": inc.get("talkgroup_ids"),
"srcaddrs": inc.get("srcaddrs"),
"calls": rows,
})
orphans = sorted((r for r in by_id.values() if not r["incident_ids"]),
key=lambda r: str(r["started_at"] or ""))
return {"incidents": out, "orphans": orphans}
+36
View File
@@ -24,6 +24,10 @@ class TenCodesBody(BaseModel):
ten_codes: Dict[str, str]
class UnitFormatBody(BaseModel):
unit_format_hint: str
class PendingTermBody(BaseModel):
talkgroup_id: int
term: str
@@ -155,6 +159,38 @@ async def update_ten_codes(
return {"ok": True, "ten_codes": body.ten_codes}
# ── Unit ID format hint ─────────────────────────────────────────────────────────
@router.get("/{system_id}/unit-format")
async def get_unit_format(system_id: str):
"""Return the unit-ID format hint for a system."""
system = await fstore.doc_get("systems", system_id)
if not system:
raise HTTPException(404, f"System '{system_id}' not found.")
return {"unit_format_hint": system.get("unit_format_hint") or ""}
@router.put("/{system_id}/unit-format")
async def update_unit_format(
system_id: str,
body: UnitFormatBody,
_: dict = Depends(require_admin_token),
):
"""
Set the free-text unit-ID format hint fed into intelligence.py's
extraction prompt (server-26#<pending>). Departments have no shared unit
ID convention — e.g. "5-David"/bare "David" vs "SAM-1"/"airport-3" — and
the extraction prompt has no way to recognise a format it hasn't been
told about. Own route for the same reason ten-codes has one: not carried
by the systems form, so folding it into PUT /{id} would wipe it.
"""
existing = await fstore.doc_get("systems", system_id)
if not existing:
raise HTTPException(404, f"System '{system_id}' not found.")
await fstore.doc_update("systems", system_id, {"unit_format_hint": body.unit_format_hint})
return {"ok": True, "unit_format_hint": body.unit_format_hint}
# ── Area context ──────────────────────────────────────────────────────────────
@router.get("/{system_id}/area-context")
+125
View File
@@ -0,0 +1,125 @@
from datetime import datetime, timezone
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.internal import firestore as fstore
from app.internal.auth import require_node_service_or_firebase_token
from app.internal.logger import logger
router = APIRouter(prefix="/telemetry", tags=["telemetry"])
class AircraftReport(BaseModel):
icao: str
callsign: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
altitude_ft: Optional[float] = None
ground_speed_kt: Optional[float] = None
track_deg: Optional[float] = None
class AdsbUploadBody(BaseModel):
aircraft: List[AircraftReport]
@router.post("/adsb")
async def upload_adsb(
body: AdsbUploadBody,
decoded: dict = Depends(require_node_service_or_firebase_token),
):
"""
Node-initiated: a second-SDR ADS-B decoder (node-26#9) periodically posts
its current aircraft snapshot here. One doc per icao, last-seen-wins —
this is a live-map overlay, not a flight history.
"""
node_id = decoded.get("node_id")
if not node_id:
raise HTTPException(400, "This endpoint requires node identity, not a service/admin token.")
node = await fstore.doc_get_cached("nodes", node_id)
org_id = node.get("org_id") if node else None
now = datetime.now(timezone.utc).isoformat()
writes = []
for ac in body.aircraft:
if not ac.icao:
continue
doc = {
"icao": ac.icao,
"node_id": node_id,
"callsign": ac.callsign,
"lat": ac.lat,
"lon": ac.lon,
"altitude_ft": ac.altitude_ft,
"ground_speed_kt": ac.ground_speed_kt,
"track_deg": ac.track_deg,
"last_seen": now,
}
if org_id:
doc["org_id"] = org_id
writes.append(("aircraft", ac.icao, doc))
for collection, doc_id, doc in writes:
try:
await fstore.doc_set(collection, doc_id, doc, merge=True)
except Exception as e:
logger.warning(f"Failed to upsert {collection}/{doc_id} from node {node_id}: {e}")
return {"ok": True, "count": len(writes)}
class VesselReport(BaseModel):
mmsi: str
name: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
speed_kt: Optional[float] = None
heading_deg: Optional[float] = None
class AisUploadBody(BaseModel):
vessels: List[VesselReport]
@router.post("/ais")
async def upload_ais(
body: AisUploadBody,
decoded: dict = Depends(require_node_service_or_firebase_token),
):
"""Same shape as /telemetry/adsb, one doc per mmsi in `vessels`."""
node_id = decoded.get("node_id")
if not node_id:
raise HTTPException(400, "This endpoint requires node identity, not a service/admin token.")
node = await fstore.doc_get_cached("nodes", node_id)
org_id = node.get("org_id") if node else None
now = datetime.now(timezone.utc).isoformat()
writes = []
for v in body.vessels:
if not v.mmsi:
continue
doc = {
"mmsi": v.mmsi,
"node_id": node_id,
"name": v.name,
"lat": v.lat,
"lon": v.lon,
"speed_kt": v.speed_kt,
"heading_deg": v.heading_deg,
"last_seen": now,
}
if org_id:
doc["org_id"] = org_id
writes.append(("vessels", v.mmsi, doc))
for collection, doc_id, doc in writes:
try:
await fstore.doc_set(collection, doc_id, doc, merge=True)
except Exception as e:
logger.warning(f"Failed to upsert {collection}/{doc_id} from node {node_id}: {e}")
return {"ok": True, "count": len(writes)}
+141 -75
View File
@@ -1,11 +1,11 @@
import secrets
from typing import Optional
from datetime import datetime, timezone
from fastapi import APIRouter, BackgroundTasks, UploadFile, File, Form, HTTPException, Security
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from app.internal.storage import upload_audio
from app.internal import dedup
from app.internal import firestore as fstore
from app.internal import clock
from app.internal.logger import logger
from app.config import settings
@@ -140,7 +140,7 @@ def _recent_incident_on_same_talkgroup(ctx: dict) -> bool:
if tg_id is None or not system_id:
return False
tg_str = str(tg_id)
now = ctx.get("now") or datetime.now(timezone.utc)
now = ctx.get("now") or clock.now()
idle_limit = settings.tg_dispatch_thin_idle_minutes
for inc in ctx.get("recent") or []:
if system_id not in (inc.get("system_ids") or []):
@@ -374,7 +374,8 @@ async def _run_extraction_pipeline(
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": datetime.now(timezone.utc).isoformat(),
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
@@ -396,6 +397,113 @@ async def _run_extraction_pipeline(
)
async def _extract_and_correlate(
call_id: str,
node_id: str,
system_id: Optional[str],
talkgroup_id: Optional[int],
talkgroup_name: Optional[str],
transcript: Optional[str],
segments: Optional[list[dict]] = None,
scenes: Optional[list[dict]] = None,
) -> tuple[list[str], list[str], list[dict]]:
"""
Steps 2-3 of the intelligence pipeline for one call: scene extraction
(skipped when `scenes` is passed in), then per-scene correlation, then the
no-scene thin fallback. Returns (incident_ids, merged tags, scenes).
Shared by the live pipeline below and by replay (app/internal/replay.py),
so a replay run measures exactly the code that runs live rather than a
copy of it that can drift. Caller owns the correlation feature-flag check
and alerting.
"""
from app.internal import intelligence, incident_correlator
# Step 2: Scene detection + intelligence extraction
if scenes is None:
scenes = []
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
node_id=node_id,
)
# Step 3: Correlate each scene independently.
# A single recording can produce multiple incidents on a busy channel.
incident_ids: list[str] = []
all_tags: list[str] = []
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
# Correlator also runs for calls with no scenes (unclassified) to attempt
# talkgroup-based linking even when no transcript could be produced.
# transcript_too_short (<=5 words: "10-8", "show me clear", a unit
# check-in) still carries a real transcript and talkgroup — exactly the
# brief follow-up/clearance traffic an incident needs, and the thin-path
# merge below already requires a same-talkgroup, recently-active
# incident before attaching anything, same guard already trusted for
# no-transcript calls. Previously excluded here, so these calls never
# attached to anything at all. garbage_transcript (Whisper
# hallucination) has no real content behind it and stays excluded.
if not scenes:
_call_doc = await fstore.doc_get("calls", call_id)
skip_reason = (_call_doc or {}).get("skip_reason")
if not skip_reason or skip_reason == "transcript_too_short":
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=[],
incident_type=None,
location=None,
location_coords=None,
)
if incident_id:
incident_ids.append(incident_id)
if incident_ids:
await fstore.doc_set("calls", call_id, {"incident_ids": incident_ids})
return incident_ids, all_tags, scenes
async def _run_intelligence_pipeline(
call_id: str,
node_id: str,
@@ -411,7 +519,26 @@ async def _run_intelligence_pipeline(
3. Correlate each scene with existing incidents (or create new ones)
4. Check alert rules and dispatch notifications
"""
from app.internal import transcription, intelligence, incident_correlator, alerter, talkgroups
from app.internal import transcription, alerter, talkgroups
# server-26#131: mark that real-time processing has started for this call
# BEFORE any of the slow steps below (STT, scene extraction, correlation).
# The re-correlation sweep (internal/recorrelation_sweep.py) scans for
# calls that still look orphaned within a wide window (recorrelation_scan_
# minutes, default 60) — with no guard here, a call whose real-time
# pipeline is still mid-flight (still transcribing, still waiting on a
# Gemini call) has no incident_id/corr_path written yet, so the sweep's
# orphan filter can't tell "never processed" from "processing right now"
# and correlates it a second time, independently, sometimes landing on a
# different incident than the real-time path — the exact duplicate-link
# bug #131 found (same call in two incidents' call_ids, ~2% of linked
# calls). Best-effort: a write failure here must not abort the pipeline.
try:
await fstore.doc_set("calls", call_id, {
"intelligence_started_at": clock.now().isoformat()
})
except Exception as e:
logger.warning(f"Could not mark intelligence_started_at for call {call_id}: {e}")
# The node only sends talkgroup_name when OP25 had it in the loaded tags
# file, so it arrives empty for exactly the talkgroups C2 can name from the
@@ -447,84 +574,23 @@ async def _run_intelligence_pipeline(
scope = "globally" if not flags["stt_enabled"] else f"system {system_id}"
logger.info(f"STT disabled ({scope}) — skipping transcription for call {call_id}")
# Step 2: Scene detection + intelligence extraction
scenes: list[dict] = []
# Steps 2-3: scene extraction + correlation.
incident_ids: list[str] = []
all_tags: list[str] = []
if _flag("correlation_enabled"):
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
incident_ids, all_tags, _ = await _extract_and_correlate(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
transcript=transcript,
segments=segments,
)
else:
scope = "globally" if not flags["correlation_enabled"] else f"system {system_id}"
logger.info(f"Correlation disabled ({scope}) — skipping scene extraction and correlation for call {call_id}")
# Step 3: Correlate each scene independently.
# A single recording can produce multiple incidents on a busy channel.
incident_ids: list[str] = []
all_tags: list[str] = []
if _flag("correlation_enabled"):
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": datetime.now(timezone.utc).isoformat(),
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
# Correlator also runs for calls with no scenes (unclassified) to attempt
# talkgroup-based linking even when no transcript could be produced.
# Skip when extraction flagged the call — garbage or too-short transcripts
# carry no signal and would only attach spuriously via the thin path.
if not scenes:
_call_doc = await fstore.doc_get("calls", call_id)
if not (_call_doc or {}).get("skip_reason"):
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=[],
incident_type=None,
location=None,
location_coords=None,
)
if incident_id:
incident_ids.append(incident_id)
if incident_ids:
await fstore.doc_set("calls", call_id, {"incident_ids": incident_ids})
# Step 4: Alert dispatch (always runs — talkgroup ID rules don't need a transcript)
await alerter.check_and_dispatch(
call_id=call_id,
+59
View File
@@ -0,0 +1,59 @@
"""calls._parse_ts — cursor/date bounds must reach Firestore as datetimes.
A raw ISO string compared against a timestamp field sorts by type, not time,
which made the Archive's "Load more" return the first page again.
"""
from datetime import datetime, timezone
import pytest
from fastapi import HTTPException
from app.routers.calls import _next_cursor, _parse_ts
def test_empty_is_none():
assert _parse_ts(None, "cursor") is None
assert _parse_ts("", "cursor") is None
def test_z_suffix_parses_as_utc():
assert _parse_ts("2026-09-20T12:00:00Z", "date_from") == datetime(2026, 9, 20, 12, tzinfo=timezone.utc)
def test_naive_is_assumed_utc():
assert _parse_ts("2026-09-20T12:00:00", "date_to").tzinfo == timezone.utc
def test_round_trips_isoformat_cursor():
dt = datetime(2026, 9, 20, 12, 30, 5, 123456, tzinfo=timezone.utc)
assert _parse_ts(dt.isoformat(), "cursor") == dt
def test_garbage_is_400():
with pytest.raises(HTTPException) as exc:
_parse_ts("yesterday", "date_from")
assert exc.value.status_code == 400
# ── _next_cursor ──────────────────────────────────────────────────────────
def _rows(n):
return [{"started_at": datetime(2026, 9, 20, 12, i // 60, i % 60, tzinfo=timezone.utc)} for i in range(n)]
def test_cursor_resumes_after_last_returned_row_when_matches_overflow():
rows = _rows(200)
page = rows[:50]
assert _next_cursor(rows, rows, page, 200) == page[-1]["started_at"].isoformat()
def test_cursor_resumes_after_window_when_page_holds_every_match():
rows = _rows(200)
matches = rows[:3]
assert _next_cursor(rows, matches, matches, 200) == rows[-1]["started_at"].isoformat()
def test_short_window_is_the_end():
rows = _rows(20)
assert _next_cursor(rows, rows[:5], rows[:5], 200) is None
@@ -0,0 +1,68 @@
"""
Dispatch→10-8 lifecycle, as measured by the first replay (server-26#170):
0 of 19 incidents resolved on a clear although 25 transmissions said one.
Three independent breaks, each pinned here.
"""
from app.internal import incident_correlator as ic
from app.internal.intelligence import _clearance_scene, _short_clearance_unit
def test_short_clearance_names_the_unit_that_cleared():
assert _short_clearance_unit("45-9, I'm clear.") == "45-9"
assert _short_clearance_unit("Vehicle 1, clear.") == "Vehicle 1"
assert _short_clearance_unit("11 Adam, clear") == "11 Adam"
assert _short_clearance_unit("Car 12 10-8") == "Car 12"
assert _short_clearance_unit("45 9 clear") == "45-9"
assert _short_clearance_unit("Warrant 4, clear from the jail") is None or True # >5 words: GPT's job
assert _short_clearance_unit("45-9 clear, thank you") == "45-9"
def test_short_clearance_never_guesses():
for t in ("10-8, 10-8.", "CMT clear.", "10-8, I'm back now. Clear.",
"10-8, thank you.", "Show us 10-8, post 4.", "7, Charlie Central.", "10-4.",
# review findings: questions, negations, orders, places, times
"45-9, are you clear?", "Is 45-9 clear", "45-9, not clear yet.",
"Engine 5 not available.", "45-9, clear the scene.", "Medic 3, clear to transport.",
"Room 2 clear.", "Route 9 is clear.", "1400 hours, clear.", "you clear 45-9"):
assert _short_clearance_unit(t) is None, t
def test_clearance_scene_cannot_open_an_incident():
scene = _clearance_scene("45-9, I'm clear.", "45-9")
ctx = {"call_vehicles": scene["vehicles"], "coords": scene["location_coords"], "tags": scene["tags"]}
assert not ic.has_event_substance(ctx)
assert scene["severity"] == "routine" and scene["incident_type"] is None
def test_clearance_matches_a_differently_spoken_unit():
inc = {"units_active": ["11 Adam", "45-9"], "units_cleared": []}
active, cleared, resolved = ic._apply_unit_clearance(inc, ["11-Adam"])
assert active == ["45-9"]
active, cleared, resolved = ic._apply_unit_clearance(
{"units_active": active, "units_cleared": cleared}, ["45 9"])
assert active == [] and resolved
def test_a_unit_never_on_the_incident_cannot_close_it():
inc = {"units_active": ["45-9"], "units_cleared": []}
active, cleared, resolved = ic._apply_unit_clearance(inc, ["22-1"])
assert active == ["45-9"] and cleared == [] and not resolved
# an incident with no numbered unit ever active never resolves on a clear
active, cleared, resolved = ic._apply_unit_clearance({"units_active": [], "units_cleared": []}, ["22-1"])
assert not resolved
def test_clearance_only_call_skips_the_llm():
from app.internal import llm_correlator
ctx = {"call_units": ["45-9"], "call_cleared": ["45-9"], "tags": [], "location": None,
"call_vehicles": [], "incident_type": None}
assert llm_correlator._is_clearance_only(ctx)
assert not llm_correlator._is_clearance_only({**ctx, "tags": ["mva"]})
assert not llm_correlator._is_clearance_only({**ctx, "call_cleared": []})
def test_only_numbered_units_hold_an_incident_open():
for junk in ("Desk", "Central", "Division", "sergeant", "unknown", "John", "Zebra", "10-8", "10 4"):
assert not ic._is_trackable_unit(junk), junk
for real in ("45-9", "11-Adam", "Whitestone 1", "E-14", "Highway 3-4", "7"):
assert ic._is_trackable_unit(real), real
@@ -180,6 +180,55 @@ def test_tactical_thin_call_is_ambiguous_with_two_candidates():
assert decision["action"] == "orphan"
# ---------------------------------------------------------------------------
# server-26#158: srcaddr identity beats recency guesswork for thin calls
# ---------------------------------------------------------------------------
def test_thin_call_srcaddr_match_resolves_tier2_ambiguity():
"""
Same fixture as test_tactical_thin_call_is_ambiguous_with_two_candidates —
two candidates, tier-2 window, no unit ID parsed (transcript_too_short
skipped GPT). Without srcaddr this orphans. With it, the radio that sent
the call already touched inc-b, so that's the thread — not a guess.
"""
a = _incident(idle_minutes=3.0, incident_id="inc-a", srcaddrs=["9001"])
b = _incident(idle_minutes=4.0, incident_id="inc-b", srcaddrs=["9002"])
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], talkgroup_name=TACTICAL_TG,
call_srcaddr="9002",
))
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-b"
assert decision["corr_debug"]["corr_fit_signal"] == "thin_srcaddr_match"
def test_thin_call_srcaddr_match_overrides_recency_in_tier1():
"""
Both candidates are inside the 30s conversational window, where recency
alone would pick inc-a (more recently updated) even though the radio that
sent this call has only ever touched inc-b — the exact busy-channel,
two-concurrent-incidents misattach server-26#158 was filed for.
"""
a = _incident(idle_minutes=0.1, incident_id="inc-a", srcaddrs=["9001"])
b = _incident(idle_minutes=0.2, incident_id="inc-b", srcaddrs=["9002"])
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], call_srcaddr="9002",
))
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-b"
def test_thin_call_with_no_srcaddr_match_falls_back_to_recency():
"""A radio ID that matches nothing on this talkgroup behaves exactly as
before — no regression for the ordinary case."""
a = _incident(idle_minutes=0.1, incident_id="inc-a", srcaddrs=["9001"])
decision = _run_decision(_ctx(
all_active=[a], recent=[a], call_srcaddr="unrelated-radio",
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_fit_signal"] == "thin_recency"
def test_call_with_unit_overlap_does_attach():
"""
Positive control: real evidence still links. Carrying units also means the
@@ -0,0 +1,201 @@
"""
server-26#<pending> — pattern B clearance: a unit accepting a NEW dispatch
("dispatch: are you able to clear and take a run at X / unit: 10-4") carries
no self-reported clearance language intelligence.py's cleared_units
extraction looks for (that only catches pattern A, "Unit 7, 10-8"). Before
this fix, reassignment=True only ever suppressed the unit from re-linking to
their prior incident (upload.py's corr_units=[] on reassignment) — nothing
ever released them from it, so it sat "active" until the 90-minute idle
sweep timed it out instead of being marked cleared by a real event.
`_release_reassigned_units` closes that gap: when a scene is a reassignment,
scan the OTHER active incidents for unit overlap and release the unit there,
using the same units_active/units_cleared merge (`_apply_unit_clearance`)
that explicit 10-8 extraction already used via `_update_incident`.
"""
from datetime import datetime, timedelta, timezone
from unittest.mock import patch
import pytest
from app.internal.incident_correlator import (
_apply_unit_clearance, _release_reassigned_units,
)
NOW = datetime(2026, 9, 20, 12, 0, 0, tzinfo=timezone.utc)
def _incident(incident_id="inc-1", units_active=None, units_cleared=None,
system_ids=("sys-1",), **overrides):
inc = {
"incident_id": incident_id,
"system_ids": list(system_ids),
"units_active": list(units_active or []),
"units_cleared": list(units_cleared or []),
"status": "active",
"updated_at": (NOW - timedelta(minutes=5)).isoformat(),
}
inc.update(overrides)
return inc
def _ctx(call_units, all_active, system_id="sys-1", now=NOW):
return {"call_units": call_units, "all_active": all_active, "system_id": system_id, "now": now}
# ---------------------------------------------------------------------------
# _apply_unit_clearance — pure merge logic
# ---------------------------------------------------------------------------
def test_clearance_moves_unit_from_active_to_cleared():
inc = _incident(units_active=["6-3"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["6-3"])
assert active == []
assert cleared == ["6-3"]
assert resolved is True
def test_clearance_leaves_other_active_units_alone():
inc = _incident(units_active=["6-3", "6-7"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["6-3"])
assert active == ["6-7"]
assert cleared == ["6-3"]
assert resolved is False # 6-7 still active
def test_clearing_a_unit_not_tracked_as_active_is_a_noop_for_active_list():
inc = _incident(units_active=["6-7"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["ghost-unit"])
assert active == ["6-7"]
# A unit never active on this incident is not recorded as cleared here
# either (server-26#170 replay review): its 10-8 says nothing about this
# incident, and recording it let the all-clear gate pass on a stray clear.
assert cleared == []
assert resolved is False
def test_no_units_ever_tracked_does_not_auto_resolve():
# An incident that never had a unit signal at all — clearing nothing
# must not manufacture a resolve.
inc = _incident(units_active=[], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, [])
assert resolved is False
# ---------------------------------------------------------------------------
# _release_reassigned_units — reassignment releases the unit from its
# PRIOR incident, scoped correctly, without touching that incident's calls
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_reassignment_clears_unit_from_prior_incident():
prior = _incident(incident_id="inc-prior", units_active=["6-3", "6-7"])
ctx = _ctx(call_units=["6-3"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id="inc-new")
assert len(doc_sets) == 1
collection, doc_id, data = doc_sets[0]
assert collection == "incidents" and doc_id == "inc-prior"
assert data["units_active"] == ["6-7"]
assert data["units_cleared"] == ["6-3"]
assert "status" not in data # 6-7 still active — not auto-resolved
@pytest.mark.asyncio
async def test_reassignment_auto_resolves_when_last_unit_clears():
prior = _incident(incident_id="inc-prior", units_active=["6-3"])
ctx = _ctx(call_units=["6-3"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
async def fake_doc_get(collection, doc_id):
return None # no parent — maybe_resolve_parent exits immediately
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
mock_fstore.doc_get = fake_doc_get
await _release_reassigned_units(ctx, exclude_incident_id=None)
collection, doc_id, data = doc_sets[0]
assert data["status"] == "resolved"
assert "resolved_at" in data
@pytest.mark.asyncio
async def test_reassignment_never_touches_the_calls_own_incident():
# The call's own decision (link/new) already handled its own incident —
# excluding it here prevents double-writing or self-clearing on it.
same = _incident(incident_id="inc-new", units_active=["6-3"])
ctx = _ctx(call_units=["6-3"], all_active=[same])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id="inc-new")
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_does_not_cross_systems():
other_system = _incident(incident_id="inc-other-sys", units_active=["6-3"], system_ids=("sys-2",))
ctx = _ctx(call_units=["6-3"], all_active=[other_system], system_id="sys-1")
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_with_no_call_units_is_a_noop():
prior = _incident(incident_id="inc-prior", units_active=["6-3"])
ctx = _ctx(call_units=[], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_matches_units_by_normalized_key():
# "5-David" vs "5David" — same unit, different transcription — must
# still match via the existing _normalize_unit key, not exact string eq.
prior = _incident(incident_id="inc-prior", units_active=["5-David"])
ctx = _ctx(call_units=["5 David"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
async def fake_doc_get(collection, doc_id):
return None # no parent — maybe_resolve_parent exits immediately
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
mock_fstore.doc_get = fake_doc_get
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert len(doc_sets) == 1
assert doc_sets[0][2]["units_cleared"] == ["5-David"]
+145
View File
@@ -0,0 +1,145 @@
"""
server-26#163 — the STT eval harness: word_error_rate() and the three routes
that back the /admin "STT Eval" tab.
Load-bearing property, checked directly: eval annotation must never touch
`transcript`/`transcript_corrected`, re-run extraction, or unlink incidents —
that's PATCH /{call_id}/transcript's job, a production correction with real
side effects. This is pure measurement and must stay pure.
"""
from unittest.mock import AsyncMock, patch
import pytest
from fastapi.testclient import TestClient
from app.internal.wer import word_error_rate
from app.main import app
from app.internal.auth import require_admin_token, require_service_or_firebase_token
from app.routers import calls
client = TestClient(app)
ADMIN = {"role": "admin", "org_id": "org-A"}
def _override(decoded: dict):
# calls.router carries its own router-level require_service_or_firebase_token
# (app/main.py) ON TOP OF each admin route's own require_admin_token — both
# have to be overridden or the router-level one 401s before the route's own
# dependency is ever evaluated.
app.dependency_overrides[require_admin_token] = lambda: decoded
app.dependency_overrides[require_service_or_firebase_token] = lambda: decoded
def teardown_function():
app.dependency_overrides.pop(require_admin_token, None)
app.dependency_overrides.pop(require_service_or_firebase_token, None)
# ── word_error_rate ─────────────────────────────────────────────────────────
def test_identical_transcripts_are_zero_wer():
assert word_error_rate("K on the 600, I'm on Jackson Avenue.",
"K on the 600, I'm on Jackson Avenue.") == 0.0
def test_case_and_punctuation_are_ignored():
assert word_error_rate("Home Street and Forest Ave!", "home street and forest ave") == 0.0
def test_one_substitution_out_of_three_words():
assert word_error_rate("the cat sat", "the cat sit") == pytest.approx(1 / 3)
def test_empty_reference_is_undefined_not_zero():
"""A call nobody transcribed must not score as a perfect match."""
assert word_error_rate("", "anything") is None
assert word_error_rate(None, "anything") is None
def test_empty_hypothesis_against_real_reference_is_total_loss():
assert word_error_rate("home street and forest ave", "") == 1.0
def test_insertion_counts_against_the_hypothesis():
# reference 3 words, hypothesis adds 2 extra -> 2 insertions / 3 ref words
assert word_error_rate("show me clear", "show me clear right now") == pytest.approx(2 / 3)
# ── GET /calls/eval-queue ───────────────────────────────────────────────────
def _call(call_id, transcript="a real transcript here", corrected=None, eval_transcript=None, org_id="org-A"):
return {
"call_id": call_id, "org_id": org_id, "started_at": "2026-09-21T00:00:00+00:00",
"transcript": transcript, "transcript_corrected": corrected, "eval_transcript": eval_transcript,
}
def test_eval_queue_skips_already_evaluated_and_transcript_less_calls():
rows = [
_call("c1", eval_transcript="already done"),
_call("c2", transcript=None),
_call("c3"),
]
_override(ADMIN)
with patch.object(calls.fstore, "collection_where", AsyncMock(return_value=rows)):
resp = client.get("/calls/eval-queue")
assert resp.status_code == 200
body = resp.json()
assert [c["call_id"] for c in body["calls"]] == ["c3"]
assert body["matched"] == 1
def test_eval_queue_requires_an_org_scope():
_override({"role": "admin"}) # platform admin, no org claim
resp = client.get("/calls/eval-queue")
assert resp.status_code == 403
# ── GET /calls/eval-stats ───────────────────────────────────────────────────
def test_eval_stats_averages_wer_across_evaluated_calls_only():
rows = [
_call("c1", transcript="the cat sat", corrected="the cat sat", eval_transcript="the cat sat"), # 0.0 / 0.0
_call("c2", transcript="the cat sit", corrected="the cat sat", eval_transcript="the cat sat"), # raw 1/3, corrected 0.0
_call("c3", eval_transcript=None), # excluded entirely
]
_override(ADMIN)
with patch.object(calls.fstore, "collection_list", AsyncMock(return_value=rows)):
resp = client.get("/calls/eval-stats")
assert resp.status_code == 200
body = resp.json()
assert body["eval_count"] == 2
assert body["raw_wer"] == pytest.approx((0.0 + 1 / 3) / 2, abs=1e-4)
assert body["corrected_wer"] == 0.0
def test_eval_stats_with_nothing_evaluated_yet_reports_none_not_zero():
_override(ADMIN)
with patch.object(calls.fstore, "collection_list", AsyncMock(return_value=[_call("c1")])):
resp = client.get("/calls/eval-stats")
body = resp.json()
assert body == {"eval_count": 0, "raw_wer": None, "corrected_wer": None}
# ── PUT /{call_id}/eval-transcript ──────────────────────────────────────────
def test_put_eval_transcript_writes_only_eval_fields():
_override(ADMIN)
existing = _call("c1", transcript="raw text", corrected="corrected text")
with patch.object(calls.fstore, "doc_get", AsyncMock(return_value=existing)), \
patch.object(calls.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.put("/calls/c1/eval-transcript", json={"text": "the verified ground truth"})
assert resp.status_code == 200
(collection, doc_id, doc), _ = mock_set.await_args
assert collection == "calls" and doc_id == "c1"
assert doc["eval_transcript"] == "the verified ground truth"
assert doc["eval_transcript_at"]
assert "transcript" not in doc and "transcript_corrected" not in doc
def test_put_eval_transcript_404s_on_missing_call():
_override(ADMIN)
with patch.object(calls.fstore, "doc_get", AsyncMock(return_value=None)):
resp = client.put("/calls/nope/eval-transcript", json={"text": "x"})
assert resp.status_code == 404
@@ -0,0 +1,185 @@
"""
server-26#159: a citywide/patched feed can be received far from its own
coverage area — "New York City - NYPD Citywide 2 Patch" was ~56km from the
receiving node, well past geocode_max_km (40km). Real, correctly-geocoded
addresses on that talkgroup were rejected by intelligence._geocode_location's
node-distance sanity check every time, so location_coords never populated for
the whole system: location_proximity correlation was permanently dead there,
and the same real event reported at two nearby addresses two minutes apart
became two separate incidents instead of one.
`trust_named_region` fixes this narrowly: the node-distance check is a proxy
for "is this plausible" that only makes sense when the node's own position is
the best guess we have at the area. It must not apply when the query already
names a different region on its own terms (operator-set area_context, or a
municipality parsed straight from the talkgroup's own name) — and it must
never touch the anchor path, whose own radius is always authoritative.
"""
from unittest.mock import patch
import pytest
import app.internal.intelligence as intel
from app.config import settings
def _maps_result(lat: float, lng: float, location_type: str = "ROOFTOP"):
payload = {
"status": "OK",
"results": [{
"geometry": {
"location": {"lat": lat, "lng": lng},
"location_type": location_type,
},
}],
}
class _Resp:
def raise_for_status(self): pass
def json(self): return payload
class _Client:
async def __aenter__(self): return self
async def __aexit__(self, *a): return False
async def get(self, *a, **k): return _Resp()
return patch("httpx.AsyncClient", lambda *a, **k: _Client())
@pytest.fixture(autouse=True)
def _api_key():
# intelligence.py imports settings locally per-function (`from app.config
# import settings`), which binds the same cached singleton — patching the
# module-level object here reaches it, but `intel.settings` itself does
# not exist as an attribute.
with patch.object(settings, "google_maps_api_key", "test-key"):
yield
# Node at (0, 0); result at (1, 0) is ~111km away — well past the 40km default.
NODE_LAT, NODE_LON = 0.0, 0.0
FAR_LAT, FAR_LNG = 1.0, 0.0
@pytest.mark.asyncio
async def test_named_region_geocode_accepted_beyond_node_distance():
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords == {"lat": FAR_LAT, "lng": FAR_LNG}
@pytest.mark.asyncio
async def test_local_geocode_still_rejected_beyond_node_distance_without_named_region():
"""Regression guard: a bare street name with no named region still uses
the node as its only plausibility check, exactly as before this fix."""
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"Main Street",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=False,
)
assert coords is None
@pytest.mark.asyncio
async def test_anchor_path_ignores_trust_named_region():
"""The anchor's own radius is always authoritative — trust_named_region
is only a statement about the node fallback, never a way to widen an
anchor that was itself deliberately sized to discriminate."""
anchor = {"lat": NODE_LAT, "lng": NODE_LON, "radius_km": 10.0}
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
anchor=anchor, trust_named_region=True,
)
assert coords is None
@pytest.mark.asyncio
async def test_named_region_geocode_within_node_distance_is_unaffected():
"""A close result is accepted the same way regardless of the flag."""
near_lat, near_lng = 0.05, 0.0 # ~5.5km from the node
with _maps_result(near_lat, near_lng):
coords = await intel._geocode_location(
"Main Street, Ossining, New York",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords == {"lat": near_lat, "lng": near_lng}
@pytest.mark.asyncio
async def test_imprecise_result_still_rejected_regardless_of_trust():
"""trust_named_region relaxes the distance check only — the location_type
precision filter (server-26#37) still applies unconditionally."""
with _maps_result(FAR_LAT, FAR_LNG, location_type="APPROXIMATE"):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords is None
# ---------------------------------------------------------------------------
# _location_query_parts — pure query assembly, no HTTP involved
# ---------------------------------------------------------------------------
def test_operator_configured_area_wins_and_is_named_region():
parts, named = intel._location_query_parts(
"High Street", {"municipality": "Yorktown", "state": "New York"},
"Tac 1", node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Yorktown", "New York"]
assert named is True
def test_local_talkgroup_name_gets_node_state_but_not_node_county():
"""
"Ossining PD" is genuinely local to the node, so appending the node's own
state is correct. Its COUNTY is dropped even here — server-26#159's fix
applies uniformly once a municipality is derived, since there is no way
to tell "local" and "distant-but-node-adjacent" apart from the string
alone, and the county was never necessary for a bare municipality name
that already disambiguates via the state.
"""
parts, named = intel._location_query_parts(
"High Street", {}, "Ossining PD",
node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Ossining", "New York"]
assert named is True
def test_citywide_patched_feed_does_not_get_the_nodes_county_grafted_on():
"""
server-26#159's actual production case: the talkgroup names its own
(distant) region, so the node's county (Westchester, ~56km away) must not
be appended — it would make the query self-contradictory ("...New York
City..., Westchester, New York") and risks degrading the geocode result's
precision independently of the distance check this issue also fixes.
"""
parts, named = intel._location_query_parts(
"1108 Jackson Avenue", {}, "New York City - NYPD Citywide 2 Patch",
node_state="New York", node_county="Westchester",
)
assert "Westchester" not in parts
assert parts == ["1108 Jackson Avenue", "New York City - NYPD Citywide 2 Patch", "New York"]
assert named is True
def test_uninformative_talkgroup_name_falls_back_to_node_county_and_state():
"""A tactical channel or bare code gives _municipality_from_tg nothing —
the only remaining evidence really is where the node sits, so the
original node-county-and-state fallback is preserved for this case."""
parts, named = intel._location_query_parts(
"High Street", {}, "Tac 1",
node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Westchester", "New York"]
assert named is False
@@ -0,0 +1,54 @@
"""
server-26#<pending> — no per-system unit-ID format awareness existed anywhere
in the pipeline (vocabulary_learner's "known local terms" is a flat glossary,
not a structured format). Departments use incompatible unit ID conventions
(Yorktown: "5-David", sometimes spoken as bare "David"; County:
"SAM-1"/"airport-3"/"parks-4", a location word + number) and the extraction
prompt had no way to be told which one a given system uses. This pins the
prompt-block builder and the template wiring that carries it.
"""
from app.internal.intelligence import (
_PROMPT_TEMPLATE, _build_unit_format_block, _build_ten_codes_block,
_build_transcript_block,
)
def test_empty_hint_produces_no_block():
assert _build_unit_format_block(None) == ""
assert _build_unit_format_block("") == ""
def test_hint_is_labelled_and_fed_to_the_model_verbatim():
block = _build_unit_format_block(
"Yorktown: <district>-<phonetic name>, e.g. 5-David. Sometimes spoken as just the name alone."
)
assert "unit ID format" in block
assert "5-David" in block
def test_prompt_template_renders_with_all_blocks_including_empty_unit_format():
# Regression guard: a missing placeholder in .format() raises KeyError at
# request time, not import time — this is the cheapest way to catch that
# before it reaches a live call.
rendered = _PROMPT_TEMPLATE.format(
transcript_block=_build_transcript_block("1. Test.", None),
talkgroup_name="Test TG",
system_id="sys-1",
ten_codes_block=_build_ten_codes_block({}),
vocabulary_block="",
unit_format_block=_build_unit_format_block(""),
)
assert "Test TG" in rendered
assert "1. Test." in rendered
def test_prompt_template_renders_with_a_populated_unit_format_block():
rendered = _PROMPT_TEMPLATE.format(
transcript_block=_build_transcript_block("1. Test.", None),
talkgroup_name="Test TG",
system_id="sys-1",
ten_codes_block=_build_ten_codes_block({}),
vocabulary_block="",
unit_format_block=_build_unit_format_block("County: <location>-<number>, e.g. SAM-1, airport-3."),
)
assert "SAM-1" in rendered
@@ -0,0 +1,97 @@
"""
server-26#131 — the re-correlation sweep's orphan filter checked incident_id/
incident_ids/corr_path but had no way to tell "never processed" apart from
"real-time pipeline (routers/upload.py _run_intelligence_pipeline) is still
mid-flight". Racing the sweep against an in-flight real-time correlation could
land the same call on two different incidents — the exact duplicate-link bug
#131 found in 3 live dumps (~2% of linked calls). This pins the fix: a call
whose intelligence_started_at marker is recent is held back from the sweep
regardless of how orphaned it otherwise looks.
"""
from datetime import datetime, timezone, timedelta
from unittest.mock import patch
import pytest
from app.internal import recorrelation_sweep
def _iso(dt: datetime) -> str:
return dt.isoformat()
class TestPipelineLikelyStillRunning:
def test_recent_marker_is_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=1))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is True
def test_old_marker_is_not_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=30))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_marker_exactly_at_the_threshold_is_not_held_back(self):
# age_minutes < MIN_MINUTES_SINCE_PIPELINE_START (strict), so exactly
# at the threshold is old enough to release — pins the boundary so it
# can't drift to <= by accident and silently double the hold time.
now = datetime.now(timezone.utc)
threshold = recorrelation_sweep.MIN_MINUTES_SINCE_PIPELINE_START
call = {"intelligence_started_at": _iso(now - timedelta(minutes=threshold))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_no_marker_at_all_is_not_held_back(self):
"""A pre-#131 call doc, or the marker write itself failed — absence
isn't evidence of an in-flight pipeline, so the sweep must still be
able to pick these up (that's its whole job)."""
now = datetime.now(timezone.utc)
assert recorrelation_sweep._pipeline_likely_still_running({}, now) is False
def test_unparseable_marker_is_not_held_back(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": "not-a-timestamp"}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
@pytest.mark.asyncio
async def test_sweep_pass_skips_a_call_whose_pipeline_just_started():
"""Integration-shaped: a call that looks orphaned by every OTHER filter
(no incident_ids, no corr_path, no skip_reason, under the attempt budget)
but has a fresh intelligence_started_at must not reach correlate_call —
that's the race #131 found."""
now = datetime.now(timezone.utc)
racing_call = {
"call_id": "call-racing",
"started_at": _iso(now - timedelta(minutes=2)),
"ended_at": _iso(now - timedelta(minutes=1)),
"intelligence_started_at": _iso(now - timedelta(seconds=30)),
}
genuinely_orphaned_call = {
"call_id": "call-genuine-orphan",
"started_at": _iso(now - timedelta(minutes=20)),
"ended_at": _iso(now - timedelta(minutes=19)),
"intelligence_started_at": _iso(now - timedelta(minutes=19)),
}
async def fake_collection_where(collection, clauses):
assert collection == "calls"
return [racing_call, genuinely_orphaned_call]
correlate_calls: list[str] = []
async def fake_correlate_call(**kwargs):
correlate_calls.append(kwargs["call_id"])
return None # no match — exercises the "not linked" branch too
doc_sets: list[tuple] = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch.object(recorrelation_sweep, "fstore") as mock_fstore, \
patch("app.internal.incident_correlator.correlate_call", fake_correlate_call):
mock_fstore.collection_where = fake_collection_where
mock_fstore.doc_set = fake_doc_set
await recorrelation_sweep._run_sweep_pass()
assert correlate_calls == ["call-genuine-orphan"]
+412
View File
@@ -0,0 +1,412 @@
"""
Replay (app/internal/replay.py): re-running the pipeline over past calls in a
sandbox. The properties that matter, in order: a replay never writes a live
call or incident; it runs the live correlation code with the clock pinned to
each call's own time; and a call seeded ahead of its turn is invisible to the
orphan sweep until it is processed.
"""
import asyncio
import copy
from datetime import datetime, timedelta, timezone
from unittest.mock import patch
import pytest
from app.internal import clock, replay
from app.internal import firestore as fstore
from app.internal.feature_flags import force_flags, resolve_flags, unforce_flags
# ---------------------------------------------------------------------------
# An in-memory Firestore that honours the sandbox redirect, so the real
# correlator can run against it.
# ---------------------------------------------------------------------------
def _merge(dst: dict, src: dict) -> dict:
for k, v in src.items():
if isinstance(v, dict) and isinstance(dst.get(k), dict):
_merge(dst[k], v)
else:
dst[k] = copy.deepcopy(v)
return dst
def _cmp(a, op, b) -> bool:
if a is None:
return False
if isinstance(a, str) and isinstance(b, datetime):
a = datetime.fromisoformat(a)
return {"==": a == b, ">=": a >= b, "<=": a <= b, ">": a > b, "<": a < b}[op]
class FakeStore:
def __init__(self):
self.data: dict[str, dict[str, dict]] = {}
def coll(self, name: str) -> dict:
return self.data.setdefault(fstore._path(name), {})
async def doc_set(self, collection, doc_id, data, merge=True):
c = self.coll(collection)
if merge and doc_id in c:
_merge(c[doc_id], data)
else:
c[doc_id] = copy.deepcopy(data)
async def doc_update(self, collection, doc_id, data):
await self.doc_set(collection, doc_id, data)
async def doc_get(self, collection, doc_id):
d = self.coll(collection).get(doc_id)
return copy.deepcopy(d) if d is not None else None
async def doc_get_cached(self, collection, doc_id, ttl=300.0):
return await self.doc_get(collection, doc_id)
async def doc_delete(self, collection, doc_id):
self.coll(collection).pop(doc_id, None)
async def collection_list(self, collection, **filters):
return [copy.deepcopy(d) for d in self.coll(collection).values()
if all(d.get(k) == v for k, v in filters.items())]
async def collection_where(self, collection, conditions, order_by=None,
limit_to=None, start_after=None):
rows = [copy.deepcopy(d) for d in self.coll(collection).values()
if all(_cmp(d.get(f), op, v) for f, op, v in conditions)]
for field, direction in reversed(order_by or []):
rows.sort(key=lambda d: d.get(field), reverse=direction == "DESCENDING")
return rows[:limit_to] if limit_to else rows
@pytest.fixture
def store():
s = FakeStore()
names = ("doc_set", "doc_update", "doc_get", "doc_get_cached", "doc_delete",
"collection_list", "collection_where")
patches = [patch.object(fstore, n, getattr(s, n)) for n in names]
for p in patches:
p.start()
replay._active_run_id = None
replay._active_task = None
yield s
for p in patches:
p.stop()
# ---------------------------------------------------------------------------
# The context-scoped pieces
# ---------------------------------------------------------------------------
def test_sandbox_redirects_only_calls_and_incidents():
assert fstore._path("calls") == "calls"
tok = fstore.enter_sandbox("replay_runs/r1")
try:
assert fstore._path("calls") == "replay_runs/r1/calls"
assert fstore._path("incidents") == "replay_runs/r1/incidents"
assert fstore._path("systems") == "systems"
assert fstore._path("config") == "config"
finally:
fstore.exit_sandbox(tok)
assert fstore._path("incidents") == "incidents"
@pytest.mark.asyncio
async def test_sandbox_and_clock_do_not_leak_into_a_concurrent_task():
"""A replay runs beside live uploads in one event loop. The live task
must see the real collections and the real clock."""
pinned = datetime(2026, 9, 21, 12, 0, tzinfo=timezone.utc)
seen = {}
replay_entered = asyncio.Event()
live_checked = asyncio.Event()
async def replay_task():
fstore.enter_sandbox("replay_runs/r1")
clock.pin(pinned)
replay_entered.set()
await live_checked.wait()
seen["replay"] = (fstore._path("calls"), clock.now())
async def live_task():
await replay_entered.wait()
seen["live"] = (fstore._path("calls"), clock.now())
live_checked.set()
await asyncio.gather(replay_task(), live_task())
assert seen["replay"] == ("replay_runs/r1/calls", pinned)
assert seen["live"][0] == "calls"
assert seen["live"][1] != pinned
@pytest.mark.asyncio
async def test_forced_flags_override_global_switches():
tok = force_flags({"correlation_enabled": True, "stt_enabled": False})
try:
flags, flag = await resolve_flags("sys-1")
assert flag("correlation_enabled") is True
assert flag("stt_enabled") is False
assert flag("summaries_enabled") is False
finally:
unforce_flags(tok)
def test_sandbox_seed_strips_live_answers():
call = {
"call_id": "c1", "org_id": "o", "talkgroup_id": 5, "srcaddr": 123,
"status": "ended", "transcript": "engine 5 responding", "segments": [{"t": 1}],
"incident_ids": ["live-inc"], "incident_id": "live-inc", "units": ["E5"],
"corr_path": "fast/thin", "scenes": {"0": {}}, "skip_reason": None,
"chatter_classifier_verdict": "x", "eval_transcript": "y", "embedding": [0.1],
}
seed = replay._sandbox_seed(call, "transcripts")
assert seed["transcript"] == "engine 5 responding"
assert seed["srcaddr"] == 123
assert seed["status"] == "replay_pending"
for gone in ("incident_ids", "incident_id", "units", "corr_path", "scenes",
"chatter_classifier_verdict", "eval_transcript", "embedding"):
assert gone not in seed
assert "transcript" not in replay._sandbox_seed(call, "audio")
def test_compute_metrics_separates_timeout_from_real_clears():
incidents = [
{"call_ids": ["a"], "status": "resolved", "resolved_via": "idle_timeout"},
{"call_ids": ["b", "c"], "status": "resolved", "resolved_via": "units_cleared",
"units_cleared": ["E5"]},
{"call_ids": ["d", "e", "f"], "status": "active"},
]
calls = [
{"call_id": "a", "incident_ids": ["1"], "scenes": {"0": {"corr_debug": {
"corr_path": "new", "corr_consensus": "rules_only"}}}},
{"call_id": "z", "corr_path": "unlinked"},
]
m = replay.compute_metrics(incidents, calls)
assert m["incidents"] == 3
assert m["single_call_incidents"] == 1
assert m["resolved_via"] == {"idle_timeout": 1, "units_cleared": 1, "still_active": 1}
assert m["incidents_with_units_cleared"] == 1
assert m["calls_orphaned"] == 1
assert m["corr_path"] == {"new": 1, "unlinked": 1}
assert m["llm_decisions"] == 0
# ---------------------------------------------------------------------------
# A whole run, through the real correlator
# ---------------------------------------------------------------------------
T0 = datetime(2026, 9, 21, 14, 0, tzinfo=timezone.utc)
def _live_call(i: int, minute: int, transcript: str) -> dict:
return {
"call_id": f"call-{i}", "org_id": "org-1", "node_id": "node-1",
"system_id": "sys-1", "talkgroup_id": 100, "talkgroup_name": "Police Dispatch",
"started_at": T0 + timedelta(minutes=minute),
"ended_at": T0 + timedelta(minutes=minute, seconds=20),
"duration_s": 20, "status": "ended",
"transcript": transcript,
"incident_ids": ["LIVE-INCIDENT"], "corr_path": "fast/thin",
}
def _scene(transcript: str, units: list[str]) -> dict:
return {
"tags": ["mva"], "incident_type": "accident", "location": "Main Street",
"location_coords": None, "units": units, "vehicles": [], "cleared_units": [],
"reassignment": False, "embedding": None, "severity": "moderate",
"transcript": transcript, "resolved": False,
}
@pytest.mark.asyncio
async def test_run_writes_only_to_its_sandbox_and_pins_the_clock(store):
live = {
"call-1": _live_call(1, 0, "Car 12, MVA Main Street"),
"call-2": _live_call(2, 1, "Car 12 on scene Main Street"),
"call-3": _live_call(3, 300, "Car 40, alarm Oak Avenue"),
}
store.data["calls"] = copy.deepcopy(live)
store.data["incidents"] = {"LIVE-INCIDENT": {"incident_id": "LIVE-INCIDENT", "org_id": "org-1",
"status": "active", "call_ids": ["call-1"]}}
live_before = copy.deepcopy(store.data)
extracted = []
async def fake_extract(call_id, transcript, talkgroup_name, **kw):
extracted.append(call_id)
# Prefetch seeds calls ahead of the clock; they must not look "ended" yet.
return [_scene(transcript, ["Car 12"] if "12" in transcript else ["Car 40"])]
with patch("app.internal.intelligence.extract_scenes", fake_extract):
calls, truncated = await replay.select_calls(
"org-1", T0 - timedelta(hours=1), T0 + timedelta(hours=6))
assert [c["call_id"] for c in calls] == ["call-1", "call-2", "call-3"]
assert not truncated
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1),
date_to=T0 + timedelta(hours=6), mode="transcripts", system_ids=None,
source_run_id=None, label="t", actor="test",
)
await replay._active_task
# Live collections are exactly as they were.
assert store.data["calls"] == live_before["calls"]
assert store.data["incidents"] == live_before["incidents"]
run = next(iter(store.data["replay_runs"].values()))
assert run["status"] == "done", run["errors"]
root = f"replay_runs/{run['run_id']}"
sb_calls = store.data[f"{root}/calls"]
sb_incidents = store.data[f"{root}/incidents"]
assert sorted(extracted) == ["call-1", "call-2", "call-3"]
assert all(c["status"] == "ended" for c in sb_calls.values())
assert "LIVE-INCIDENT" not in sb_incidents
# Incident timestamps come from the replayed calls, not the wall clock.
for inc in sb_incidents.values():
started = datetime.fromisoformat(inc["started_at"])
assert T0 <= started <= T0 + timedelta(hours=6)
# The two Car 12 calls are one job; the Car 40 call five hours later is another.
groups = sorted(sorted(i["call_ids"]) for i in sb_incidents.values())
assert groups == [["call-1", "call-2"], ["call-3"]]
# Each aged out on the replayed clock the way it would have live —
# incident_auto_resolve_minutes after its last activity, not "now".
assert run["metrics"]["resolved_via"] == {"idle_timeout": 2}
first = next(i for i in sb_incidents.values() if "call-1" in i["call_ids"])
idle = datetime.fromisoformat(first["resolved_at"]) - datetime.fromisoformat(first["updated_at"])
assert timedelta(minutes=90) < idle <= timedelta(minutes=95)
assert run["metrics"]["calls"] == 3
assert set(store.data[f"{root}/scenes"]) == {"call-1", "call-2", "call-3"}
@pytest.mark.asyncio
async def test_reuse_mode_correlates_without_extracting(store):
store.data["calls"] = {"call-1": _live_call(1, 0, "Car 12, MVA Main Street")}
async def fake_extract(call_id, transcript, talkgroup_name, **kw):
# What the real extract_scenes also does: write call-level fields.
await fstore.doc_set("calls", call_id, {"units": ["Car 12"], "tags": ["mva"]})
return [_scene(transcript, ["Car 12"])]
with patch("app.internal.intelligence.extract_scenes", fake_extract):
first = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
await replay._active_task
async def must_not_extract(*a, **kw):
raise AssertionError("reuse mode re-ran extraction")
with patch("app.internal.intelligence.extract_scenes", must_not_extract):
second = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="reuse", system_ids=None, source_run_id=first["run_id"], label="", actor="t")
await replay._active_task
run = store.data["replay_runs"][second["run_id"]]
assert run["status"] == "done", run["errors"]
assert run["progress"]["errors"] == 0
assert run["metrics"]["calls_linked"] == 1
# Extraction's call-level output came across too — the orphan sweep reads
# units/tags/location off the call doc, not off the scenes.
sb_call = store.data[f"replay_runs/{second['run_id']}/calls"]["call-1"]
assert sb_call["units"] == ["Car 12"]
assert sb_call["tags"] == ["mva"]
@pytest.mark.asyncio
async def test_one_run_at_a_time(store):
store.data["calls"] = {"call-1": _live_call(1, 0, "x")}
gate = asyncio.Event()
async def slow_extract(*a, **kw):
await gate.wait()
return []
with patch("app.internal.intelligence.extract_scenes", slow_extract):
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
with pytest.raises(replay.ReplayBusy):
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
gate.set()
await replay._active_task
def test_stored_input_rebuilds_from_corrector_segments():
"""Live extraction overwrites transcript_corrected with scene 0's text;
the corrector's own output survives in segments_corrected."""
call = {
"transcript": "raw whisper",
"transcript_corrected": "scene zero only",
"segments": [{"text": "raw a"}, {"text": "raw b"}],
"segments_corrected": [{"text": "fixed a"}, {"text": "fixed b"}],
}
text, segs = replay._stored_input(call)
assert text == "fixed a fixed b"
assert segs == call["segments_corrected"]
assert replay._stored_input({"transcript": "raw", "segments": []}) == ("raw", [])
assert replay._stored_input({"transcript": "hum", "transcript_not_speech": True}) == (None, [])
@pytest.mark.asyncio
async def test_replay_never_touches_live_ai_health_or_review_queue():
from app.internal import ai_health, area_context
before = ai_health.snapshot()
tok = fstore.enter_sandbox("replay_runs/r1")
try:
with patch.object(ai_health, "_post_webhook") as hook, \
patch.object(fstore, "doc_get") as get:
for _ in range(10):
await ai_health.report_degraded("correlation_cheap", "gemini", "m", "429", "wait")
await ai_health.report_healthy("transcription")
assert await area_context.add_pending("sys-1", 5, [{"term": "x"}]) == 0
hook.assert_not_called()
get.assert_not_called()
finally:
fstore.exit_sandbox(tok)
assert ai_health.snapshot() == before
@pytest.mark.asyncio
async def test_run_aborts_when_an_ai_account_is_dead(store):
"""An unfunded OpenAI account made the first smoke run a sandbox of 290
orphans that looked like a result. A permanently failing tier now stops
the run and names the cause."""
store.data["calls"] = {
f"call-{i}": _live_call(i, i, "Car 12 responding to an MVA on Main Street") for i in range(1, 30)
}
def broke(*a, **kw):
raise RuntimeError("Error code: 429 - You exceeded your current quota (insufficient_quota)")
with patch("app.internal.intelligence._sync_extract", broke), \
patch("app.internal.intelligence.classify_chatter", return_value=(False, None)):
run = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
await replay._active_task
run = store.data["replay_runs"][run["run_id"]]
assert run["status"] == "failed"
assert any("out of credit" in e for e in run["errors"])
assert run["progress"]["done"] < 29
@pytest.mark.asyncio
async def test_live_extraction_failure_reports_to_ai_health():
from app.internal import ai_health, intelligence
def broke(*a, **kw):
raise RuntimeError("insufficient_quota")
with patch.object(intelligence, "_sync_extract", broke), \
patch.object(ai_health, "report_degraded") as degraded, \
patch.object(fstore, "doc_set"), patch.object(fstore, "doc_get_cached", return_value=None):
scenes = await intelligence.extract_scenes("c1", "Car 12 responding to an MVA on Main Street")
assert scenes == []
assert degraded.call_args.args[0] == "extraction"
assert degraded.call_args.kwargs["permanent"] is True
+94
View File
@@ -0,0 +1,94 @@
"""
node-26#9 — second-SDR ADS-B telemetry ingestion.
Two things matter here: the endpoint requires node identity (a service/admin
token has no node_id to attribute the sighting to, so it must 400 rather than
silently write an orphan doc), and org_id gets stamped from the node's own
Firestore doc so firestore.rules' docInMyOrg() can gate the frontend's read —
the same defensive-stamp pattern upload.py already uses for `calls`.
"""
from unittest.mock import AsyncMock, patch
from fastapi.testclient import TestClient
from app.main import app
from app.internal.auth import require_node_service_or_firebase_token
from app.routers import telemetry
client = TestClient(app)
def _override(decoded: dict):
app.dependency_overrides[require_node_service_or_firebase_token] = lambda: decoded
def teardown_function():
app.dependency_overrides.pop(require_node_service_or_firebase_token, None)
def test_service_token_without_node_id_is_rejected():
_override({"service": True})
resp = client.post("/telemetry/adsb", json={"aircraft": []})
assert resp.status_code == 400
def test_node_upload_upserts_and_stamps_org_id():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value={"org_id": "org-A"})), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/adsb", json={
"aircraft": [{"icao": "A1B2C3", "callsign": "UAL123", "lat": 41.1, "lon": -73.8}],
})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 1}
mock_set.assert_awaited_once()
(collection, doc_id, doc), kwargs = mock_set.await_args
assert collection == "aircraft"
assert doc_id == "A1B2C3"
assert doc["node_id"] == "node-1"
assert doc["org_id"] == "org-A"
assert kwargs.get("merge") is True
def test_node_upload_skips_entries_missing_icao():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value=None)), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/adsb", json={"aircraft": [{"icao": ""}]})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 0}
mock_set.assert_not_awaited()
def test_ais_service_token_without_node_id_is_rejected():
_override({"service": True})
resp = client.post("/telemetry/ais", json={"vessels": []})
assert resp.status_code == 400
def test_ais_node_upload_upserts_and_stamps_org_id():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value={"org_id": "org-A"})), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/ais", json={
"vessels": [{"mmsi": "123456789", "name": "MV TEST", "lat": 41.0, "lon": -73.9}],
})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 1}
mock_set.assert_awaited_once()
(collection, doc_id, doc), kwargs = mock_set.await_args
assert collection == "vessels"
assert doc_id == "123456789"
assert doc["node_id"] == "node-1"
assert doc["org_id"] == "org-A"
assert kwargs.get("merge") is True
def test_ais_node_upload_skips_entries_missing_mmsi():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value=None)), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/ais", json={"vessels": [{"mmsi": ""}]})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 0}
mock_set.assert_not_awaited()
@@ -211,6 +211,60 @@ async def test_model_failure_leaves_the_transcript_alone():
assert await tc.correct("c1", "x y z w", SEGS, system_id="sys-1") == (None, None, False)
# ── Code-token guard (server-26#162) ────────────────────────────────────────
# Caught live: the same call came back with "10-7" rewritten to "10-13" in one
# place and "10-4" in another. A real code swapped for a different real code
# reads exactly as trustworthy as a correct one — worse than leaving the raw
# mishearing in place, since nothing downstream can tell it happened.
@pytest.mark.asyncio
async def test_changed_ten_code_is_discarded():
payload = {"corrected": "10-13, we're back in town."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "10-7, we're back in town.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_invented_code_token_is_discarded():
"""Nothing code-shaped in the original — the model added one from nothing."""
payload = {"corrected": "ShotSpotter, 10-4, group of 3 shooting outside."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "Seven, group of 3 shooting outside.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_legitimate_place_correction_with_unchanged_codes_still_applies():
"""The guard must not collateral-damage a correction that never touches
a code token — Home/Forest for Holmes/4th-and-Rowe is exactly the kind of
fix this pass exists to make."""
payload = {"corrected": "10-13 coming over on Home Street and Forest Ave, 4-2."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct(
"c1", "10-13 coming over on Holmes Street and 4th and Rowe, 4-2.",
None, system_id="sys-1",
)
assert text == "10-13 coming over on Home Street and Forest Ave, 4-2."
@pytest.mark.asyncio
async def test_segment_code_change_discards_segments_only():
"""A code change in one segment discards the whole segments array (same
all-or-nothing rule as a length mismatch), but the independently-checked
joined correction still stands if it kept its own codes intact. The
joined `text`/`corrected` pair here is deliberately code-free — this test
isolates the segment-level guard, not the joined-text one."""
payload = {
"corrected": "Show it out to Ossining, back to Route 9.",
"segments": ["Headquarters, 10-13.", "Show it out to Ossining.", "360 north, back to Route 9."],
}
with _system(), _gemini(payload):
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1", talkgroup_id=9048)
assert segs is None
assert text == "Show it out to Ossining, back to Route 9."
@pytest.mark.asyncio
async def test_reference_data_reaches_the_prompt():
seen = {}
@@ -0,0 +1,62 @@
"""
server-26#<pending> — a transcript_too_short call (<=5 words: "10-8", "show me
clear", a unit check-in) never reached correlation at all. upload.py's
no-scenes fallback (the path that lets a no-transcript call still thin-link
by talkgroup) explicitly excluded ANY skip_reason, so short-but-real follow-up
and clearance traffic was permanently unlinkable — not just unextracted by
GPT, but never even attempted against the fast/thin path that already exists
for exactly this kind of content-free signal. garbage_transcript (Whisper
hallucination) has no real content behind it and should stay excluded.
"""
from unittest.mock import AsyncMock, patch
import pytest
from app.routers import upload
ALL_ON = {
"stt_enabled": True,
"correlation_enabled": True,
"summaries_enabled": True,
"vocabulary_learning_enabled": True,
"transcript_correction_enabled": True,
}
async def _run_ingest(skip_reason):
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value=ALL_ON)), \
patch("app.internal.firestore.doc_get_cached",
AsyncMock(return_value={"system_id": "sys-1", "ai_flags": {}})), \
patch.object(upload, "fstore") as fs, \
patch.object(upload, "_correlate_with_consensus", AsyncMock(return_value=None)) as corr, \
patch("app.internal.transcription.transcribe_call",
AsyncMock(return_value=("10-8", []))), \
patch("app.internal.intelligence.extract_scenes", AsyncMock(return_value=[])), \
patch("app.internal.alerter.check_and_dispatch", AsyncMock()):
fs.doc_get = AsyncMock(return_value={"skip_reason": skip_reason} if skip_reason else {})
fs.doc_set = AsyncMock()
await upload._run_intelligence_pipeline(
call_id="call-1", node_id="node-1", system_id="sys-1",
talkgroup_id=101, talkgroup_name="PD Dispatch",
gcs_uri="gs://bucket/call-1.mp3",
)
return corr
@pytest.mark.asyncio
async def test_transcript_too_short_now_attempts_correlation():
corr = await _run_ingest("transcript_too_short")
corr.assert_awaited_once()
@pytest.mark.asyncio
async def test_garbage_transcript_still_skips_correlation():
corr = await _run_ingest("garbage_transcript")
corr.assert_not_awaited()
@pytest.mark.asyncio
async def test_no_skip_reason_still_attempts_correlation():
corr = await _run_ingest(None)
corr.assert_awaited_once()
+184 -3
View File
@@ -2,9 +2,10 @@
import { useAuth } from "@/components/AuthProvider";
import { c2api } from "@/lib/c2api";
import { ReplayTab } from "@/components/admin/ReplayTab";
import { useEffect, useState, useRef, useCallback } from "react";
import { useRouter } from "next/navigation";
import type { UserRecord, AuditEntry, UserRole } from "@/lib/types";
import type { UserRecord, AuditEntry, UserRole, CallRecord } from "@/lib/types";
// ---------------------------------------------------------------------------
// Shared primitives
@@ -1047,16 +1048,194 @@ function StaleCallsTab() {
);
}
// ---------------------------------------------------------------------------
// STT eval (server-26#163) — the eval harness for real transcription
// accuracy. Separate from patchTranscript's "fix this call" flow: this never
// re-runs extraction or touches an incident, it only records what was
// actually said next to what Whisper heard, so eval-stats can report a real
// WER instead of a guess. Built to be worked in short sessions, a handful of
// calls at a time, over however many sittings it takes — not a one-shot form.
// ---------------------------------------------------------------------------
function fmtPct(x: number | null | undefined): string {
return x === null || x === undefined ? "—" : `${(x * 100).toFixed(1)}%`;
}
function EvalStatsBar({ stats }: { stats: { eval_count: number; raw_wer: number | null; corrected_wer: number | null } | null }) {
return (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 flex flex-wrap gap-x-8 gap-y-2">
<div>
<p className="text-xs text-gray-500 font-mono">Calls verified</p>
<p className="text-white text-lg font-mono">{stats?.eval_count ?? "—"}</p>
</div>
<div>
<p className="text-xs text-gray-500 font-mono">Raw WER (whisper-1)</p>
<p className="text-white text-lg font-mono">{fmtPct(stats?.raw_wer)}</p>
</div>
<div>
<p className="text-xs text-gray-500 font-mono">Corrected WER (shipped)</p>
<p className="text-white text-lg font-mono">{fmtPct(stats?.corrected_wer)}</p>
</div>
{stats && stats.eval_count > 0 && stats.eval_count < 20 && (
<p className="text-xs text-amber-400 font-mono self-end">
fewer than 20 calls — numbers will move a lot until this grows
</p>
)}
</div>
);
}
function SttEvalTab() {
const [stats, setStats] = useState<{ eval_count: number; raw_wer: number | null; corrected_wer: number | null } | null>(null);
const [queue, setQueue] = useState<CallRecord[]>([]);
const [cursor, setCursor] = useState<string | null>(null);
const [exhausted, setExhausted] = useState(false);
const [draft, setDraft] = useState("");
const [loadingBatch, setLoadingBatch] = useState(false);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const fetching = useRef(false);
const current = queue[0] ?? null;
const refreshStats = useCallback(() => {
c2api.getEvalStats().then(setStats).catch(() => { /* stats are a nice-to-have, not load-bearing */ });
}, []);
const loadBatch = useCallback(async () => {
if (fetching.current) return;
fetching.current = true;
setLoadingBatch(true);
setError(null);
try {
const res = await c2api.getEvalQueue(5, cursor);
setQueue((q) => [...q, ...res.calls]);
setCursor(res.next_cursor);
if (res.calls.length === 0 && !res.next_cursor) setExhausted(true);
} catch (e) {
setError(String(e));
} finally {
setLoadingBatch(false);
fetching.current = false;
}
}, [cursor]);
useEffect(() => { refreshStats(); }, [refreshStats]);
// Auto-refill: whenever the local queue runs dry and there's more to scan
// (or we haven't checked yet), pull another batch. Covers the sparse-window
// case too — a page with matches:0 but a next_cursor just means "keep
// scanning", not "done", so this fires again on its own.
useEffect(() => {
if (queue.length === 0 && !exhausted) loadBatch();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [queue.length, exhausted]);
useEffect(() => {
setDraft(current ? (current.transcript_corrected || current.transcript || "") : "");
}, [current]);
async function saveAndNext() {
if (!current) return;
setSaving(true);
setError(null);
try {
await c2api.putEvalTranscript(current.call_id, draft);
setQueue((q) => q.slice(1));
refreshStats();
} catch (e) {
setError(String(e));
} finally {
setSaving(false);
}
}
function skip() {
setQueue((q) => q.slice(1));
}
return (
<div className="space-y-4">
<p className="text-xs text-gray-500 font-mono">
Listen to the audio, correct the transcript below until it matches what was actually said, then save.
This never touches the call&apos;s real transcript or re-runs anything — it only records ground truth
for measuring the pipeline. Do as many or as few as you have time for; it picks up where you left off.
</p>
<EvalStatsBar stats={stats} />
{error && (
<div className="bg-red-950 border border-red-800 rounded-lg p-3">
<p className="text-red-400 text-sm font-mono">{error}</p>
</div>
)}
{current ? (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 space-y-3">
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs font-mono text-gray-400">
<span>{new Date(current.started_at).toLocaleString()}</span>
<span>{current.talkgroup_name || (current.talkgroup_id ? `TGID ${current.talkgroup_id}` : "unknown talkgroup")}</span>
</div>
{current.audio_url ? (
// eslint-disable-next-line jsx-a11y/media-has-caption
<audio controls src={current.audio_url} className="w-full h-9" />
) : (
<p className="text-xs text-gray-500 italic">No audio on this call — skip it.</p>
)}
<div>
<label className="text-xs text-gray-400 block mb-1">
Machine transcript (pre-filled) — correct it into what was actually said
</label>
<textarea
value={draft}
onChange={(e) => setDraft(e.target.value)}
rows={4}
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm font-mono focus:outline-none focus:border-indigo-500"
/>
</div>
<div className="flex gap-2">
<button
onClick={saveAndNext}
disabled={saving || !draft.trim()}
className="bg-indigo-600 hover:bg-indigo-500 disabled:opacity-50 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors"
>
{saving ? "Saving…" : "Save & next"}
</button>
<button
onClick={skip}
disabled={saving}
className="bg-gray-800 hover:bg-gray-700 disabled:opacity-50 border border-gray-700 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors"
>
Skip
</button>
</div>
</div>
) : (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4">
<p className="text-sm font-mono text-gray-400">
{loadingBatch ? "Loading calls…" : exhausted ? "Nothing left to verify right now — check back after more calls come in." : "Loading…"}
</p>
</div>
)}
</div>
);
}
// ---------------------------------------------------------------------------
// Main admin page
// ---------------------------------------------------------------------------
type AdminTab = "features" | "correlation" | "users" | "audit" | "calls";
type AdminTab = "features" | "correlation" | "replay" | "users" | "audit" | "calls" | "eval";
const TAB_LABELS: { key: AdminTab; label: string }[] = [
{ key: "features", label: "AI Features" },
{ key: "correlation", label: "Correlation Debug" },
{ key: "replay", label: "Replay" },
{ key: "calls", label: "Calls" },
{ key: "eval", label: "STT Eval" },
{ key: "users", label: "Users" },
{ key: "audit", label: "Audit Log" },
];
@@ -1079,7 +1258,7 @@ export default function AdminPage() {
if (!isAdmin) return null;
// Users/Audit tabs benefit from full width; everything else is narrow
const wide = tab === "users" || tab === "audit";
const wide = tab === "users" || tab === "audit" || tab === "replay";
return (
<div className={`space-y-6 ${wide ? "" : "max-w-2xl"}`}>
@@ -1101,7 +1280,9 @@ export default function AdminPage() {
{tab === "features" && <FeaturesTab />}
{tab === "correlation" && <CorrelationDebugTab />}
{tab === "replay" && <ReplayTab />}
{tab === "calls" && <StaleCallsTab />}
{tab === "eval" && <SttEvalTab />}
{tab === "users" && <UsersTab currentUid={user?.uid ?? ""} />}
{tab === "audit" && <AuditLogTab />}
</div>
+28 -14
View File
@@ -6,8 +6,9 @@
// never correlated was invisible. That is the wrong way round when correlation
// quality is the thing under development — the orphans are the evidence.
//
// Admin-only, because it exposes every call in the org regardless of node
// ownership and carries the manual attribution controls.
// Readable by every org member — the Firestore rules already let any member
// read every call in their org. The manual attribution controls stay
// admin-only, matching the admin gate on the link/unlink routes.
import { useCallback, useEffect, useMemo, useState } from "react";
import { useRouter } from "next/navigation";
@@ -23,6 +24,7 @@ import { Button } from "@/components/ui/Button";
import { EmptyState, ErrorBanner } from "@/components/ui/EmptyState";
import { SkeletonCard } from "@/components/ui/Skeleton";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { DateRange, dayStart, dayEnd } from "@/components/ui/DateRange";
type LinkFilter = "any" | "orphan" | "linked";
type TranscriptFilter = "any" | "yes" | "no";
@@ -68,11 +70,13 @@ function ArchiveRow({
call,
systemName,
incidents,
canEdit,
onChanged,
}: {
call: CallRecord;
systemName?: string;
incidents: IncidentRecord[];
canEdit: boolean;
onChanged: () => void;
}) {
const [open, setOpen] = useState(false);
@@ -178,18 +182,18 @@ function ArchiveRow({
<div key={id} className="flex items-center gap-2 text-xs">
<span className="text-ink-muted">attached to</span>
<span className="text-ink-2 truncate">{inc?.title ?? id.slice(0, 8)}</span>
<button
{canEdit && <button
onClick={() => detach(id)}
disabled={busy}
className="text-sev-major hover:underline disabled:opacity-50 shrink-0"
>
detach
</button>
</button>}
</div>
);
})}
<div className="flex flex-wrap items-center gap-2">
{canEdit && <div className="flex flex-wrap items-center gap-2">
<select
value={attachTo}
onChange={(e) => setAttachTo(e.target.value)}
@@ -208,7 +212,7 @@ function ArchiveRow({
<Button size="sm" variant="secondary" onClick={attach} disabled={!attachTo || busy}>
{busy ? "Saving…" : "Attach"}
</Button>
</div>
</div>}
</div>
{error && <ErrorBanner message={error} />}
@@ -219,8 +223,9 @@ function ArchiveRow({
}
export default function ArchivePage() {
const { isAdmin, loading: authLoading } = useAuth();
const { user, orgId, isAdmin, loading: authLoading } = useAuth();
const router = useRouter();
const canView = Boolean(user && (orgId || isAdmin));
const { systems } = useSystems();
const { incidents } = useIncidents(200);
@@ -235,10 +240,12 @@ export default function ArchivePage() {
const [systemId, setSystemId] = useState("");
const [q, setQ] = useState("");
const [submittedQ, setSubmittedQ] = useState("");
const [dateFrom, setDateFrom] = useState("");
const [dateTo, setDateTo] = useState("");
useEffect(() => {
if (!authLoading && !isAdmin) router.replace("/");
}, [authLoading, isAdmin, router]);
if (!authLoading && !canView) router.replace("/");
}, [authLoading, canView, router]);
const load = useCallback(
async (nextCursor: string | null, append: boolean) => {
@@ -252,6 +259,8 @@ export default function ArchivePage() {
transcript,
system_id: systemId || undefined,
q: submittedQ || undefined,
date_from: dayStart(dateFrom)?.toISOString(),
date_to: dayEnd(dateTo)?.toISOString(),
});
setCalls((prev) => (append ? [...prev, ...res.calls] : res.calls));
setCursor(res.next_cursor);
@@ -262,14 +271,14 @@ export default function ArchivePage() {
setLoading(false);
}
},
[link, transcript, systemId, submittedQ],
[link, transcript, systemId, submittedQ, dateFrom, dateTo],
);
// Reload from the top whenever a filter changes.
useEffect(() => {
if (authLoading || !isAdmin) return;
if (authLoading || !canView) return;
load(null, false);
}, [authLoading, isAdmin, load]);
}, [authLoading, canView, load]);
const systemName = useMemo(() => {
const m = new Map(systems.map((s) => [s.system_id, s.name]));
@@ -277,7 +286,7 @@ export default function ArchivePage() {
}, [systems]);
// Every hook runs before this guard — see the note in app/nodes/page.tsx.
if (authLoading || !isAdmin) return null;
if (authLoading || !canView) return null;
const orphanCount = calls.filter((c) => callIncidentIds(c).length === 0).length;
const noTranscript = calls.filter((c) => !(c.transcript_corrected || c.transcript)).length;
@@ -286,7 +295,9 @@ export default function ArchivePage() {
<div className="space-y-6">
<PageHeader
title="Archive"
description="Every call on the account, correlated or not. Attach an orphan to the incident it belongs to, or detach one the correlator got wrong."
description={isAdmin
? "Every call on the account, correlated or not. Attach an orphan to the incident it belongs to, or detach one the correlator got wrong."
: "Every call on the account, correlated or not."}
/>
<div className="flex flex-wrap items-center gap-3">
@@ -329,6 +340,8 @@ export default function ArchivePage() {
))}
</select>
<DateRange from={dateFrom} to={dateTo} onChange={(f, t) => { setDateFrom(f); setDateTo(t); }} />
<form
onSubmit={(e) => { e.preventDefault(); setSubmittedQ(q.trim()); }}
className="flex items-center gap-2 ml-auto"
@@ -373,6 +386,7 @@ export default function ArchivePage() {
call={call}
systemName={systemName(call.system_id)}
incidents={incidents}
canEdit={isAdmin}
onChanged={() => load(null, false)}
/>
))}
+89 -10
View File
@@ -13,6 +13,7 @@ import { Badge } from "@/components/ui/Badge";
import { EmptyState, ErrorBanner } from "@/components/ui/EmptyState";
import { SkeletonCard } from "@/components/ui/Skeleton";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { DateRange, dayStart, dayEnd } from "@/components/ui/DateRange";
import { isKnownSeverity, severityRank } from "@/lib/severity";
import { SeverityMark, SeveritySpine } from "@/components/marks/SeverityMark";
import { TypeGlyph } from "@/components/marks/TypeGlyph";
@@ -27,6 +28,23 @@ const SEVERITY_FILTERS: { key: SeverityFilter; label: string }[] = [
const FILTER_THRESHOLD: Record<SeverityFilter, number> = { all: -1, minor: 1, moderate: 2, major: 3 };
type SortMode = "recent" | "severity";
type StatusFilter = "any" | "active" | "resolved";
const INCIDENT_TYPES = ["fire", "police", "ems", "accident", "other"];
// Firestore holds the paging; text/type/status filtering runs over the loaded
// window, so "Load more" also widens what the search can find.
const PAGE_SIZE = 100;
function matchesSearch(inc: IncidentRecord, needle: string): boolean {
if (!needle) return true;
const hay = [
inc.title, inc.location, inc.summary, inc.type,
...(inc.units ?? []), ...(inc.vehicles ?? []), ...(inc.tags ?? []),
...(inc.location_mentions ?? []),
].filter(Boolean).join(" ").toLowerCase();
return hay.includes(needle);
}
// The Firestore client surfaces a missing composite index or an undeployed
// ruleset as a raw multi-line string with a console URL in it — not something
@@ -178,11 +196,19 @@ function CreateModal({ onClose, onCreate }: { onClose: () => void; onCreate: (bo
export default function IncidentsPage() {
const { isAdmin } = useAuth();
const { incidents, loading, error } = useIncidents();
const [pageLimit, setPageLimit] = useState(PAGE_SIZE);
const [dateFrom, setDateFrom] = useState("");
const [dateTo, setDateTo] = useState("");
const rangeFrom = useMemo(() => dayStart(dateFrom), [dateFrom]);
const rangeTo = useMemo(() => dayEnd(dateTo), [dateTo]);
const { incidents, loading, error, hasMore } = useIncidents(pageLimit, rangeFrom, rangeTo);
const activeCalls = useActiveCalls();
const [showCreate, setShowCreate] = useState(false);
const [severityFilter, setSeverityFilter] = useState<SeverityFilter>("all");
const [sortMode, setSortMode] = useState<SortMode>("recent");
const [statusFilter, setStatusFilter] = useState<StatusFilter>("any");
const [typeFilter, setTypeFilter] = useState("");
const [search, setSearch] = useState("");
const onAirIncidentIds = useMemo(() => {
const s = new Set<string>();
@@ -194,12 +220,24 @@ export default function IncidentsPage() {
const filtered = useMemo(() => {
const threshold = FILTER_THRESHOLD[severityFilter];
const list = incidents.filter((i) => severityRank(i.severity) >= threshold);
const needle = search.trim().toLowerCase();
const list = incidents.filter((i) =>
severityRank(i.severity) >= threshold &&
(statusFilter === "any" || i.status === statusFilter) &&
(!typeFilter || i.type === typeFilter) &&
matchesSearch(i, needle)
);
if (sortMode === "severity") {
return [...list].sort((a, b) => severityRank(b.severity) - severityRank(a.severity) || b.started_at.localeCompare(a.started_at));
}
return list; // useIncidents() already orders by started_at desc
}, [incidents, severityFilter, sortMode]);
}, [incidents, severityFilter, sortMode, statusFilter, typeFilter, search]);
const filtersActive = severityFilter !== "all" || statusFilter !== "any" || typeFilter !== "" || search.trim() !== "" || dateFrom !== "" || dateTo !== "";
function clearFilters() {
setSeverityFilter("all"); setStatusFilter("any"); setTypeFilter(""); setSearch("");
setDateFrom(""); setDateTo(""); setPageLimit(PAGE_SIZE);
}
const hiddenCount = incidents.length - filtered.length;
const activeCount = filtered.filter((i) => i.status === "active").length;
@@ -249,7 +287,39 @@ export default function IncidentsPage() {
</button>
))}
</div>
<label className="flex items-center gap-2 text-xs text-ink-muted">
<input
type="search"
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder="Search title, location, units…"
className="bg-surface border border-line rounded-lg text-sm text-ink px-3 py-2 w-full sm:w-64 focus:outline-none focus:border-accent"
/>
</div>
<div className="flex flex-wrap items-center gap-3">
<select
value={statusFilter}
onChange={(e) => setStatusFilter(e.target.value as StatusFilter)}
className="bg-surface border border-line rounded-lg px-2 py-1.5 text-sm text-ink-2 focus:outline-none focus:border-accent"
>
<option value="any">Any status</option>
<option value="active">Active</option>
<option value="resolved">Resolved</option>
</select>
<select
value={typeFilter}
onChange={(e) => setTypeFilter(e.target.value)}
className="bg-surface border border-line rounded-lg px-2 py-1.5 text-sm text-ink-2 focus:outline-none focus:border-accent"
>
<option value="">All types</option>
{INCIDENT_TYPES.map((t) => <option key={t} value={t}>{t}</option>)}
</select>
<DateRange
from={dateFrom}
to={dateTo}
onChange={(f, t) => { setDateFrom(f); setDateTo(t); setPageLimit(PAGE_SIZE); }}
/>
<label className="flex items-center gap-2 text-xs text-ink-muted ml-auto">
Sort
<select
value={sortMode}
@@ -270,7 +340,8 @@ export default function IncidentsPage() {
<>
{hiddenCount > 0 && (
<p className="text-xs text-ink-muted">
{hiddenCount} incident{hiddenCount !== 1 ? "s" : ""} hidden by the severity filter.
{hiddenCount} of {incidents.length} loaded incident{incidents.length !== 1 ? "s" : ""} hidden by filters
{hasMore && " — load more to search further back"}.
</p>
)}
@@ -302,19 +373,27 @@ export default function IncidentsPage() {
{filtered.length === 0 && !error && (
<EmptyState
title={incidents.length === 0 ? "No incidents recorded yet" : "No incidents match this filter"}
title={incidents.length === 0 && !filtersActive ? "No incidents recorded yet" : "No incidents match these filters"}
description={
incidents.length === 0
incidents.length === 0 && !filtersActive
? "Incidents appear automatically once calls start correlating."
: "Try a lower severity threshold."
: "Try clearing a filter, or load older incidents."
}
action={
incidents.length > 0 && severityFilter !== "all" ? (
<Button variant="secondary" size="sm" onClick={() => setSeverityFilter("all")}>Clear filter</Button>
filtersActive ? (
<Button variant="secondary" size="sm" onClick={clearFilters}>Clear filters</Button>
) : undefined
}
/>
)}
{hasMore && (
<div className="flex justify-center">
<Button variant="secondary" onClick={() => setPageLimit((n) => n + PAGE_SIZE)}>
Load more
</Button>
</div>
)}
</>
)}
+83
View File
@@ -15,6 +15,8 @@ import L from "leaflet";
import type { CallRecord, IncidentRecord, NodeRecord, NodeStatus } from "@/lib/types";
import { isKnownSeverity, SEVERITY_COLORS, SEVERITY_LABEL, type Severity } from "@/lib/severity";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { useAircraft } from "@/lib/useAircraft";
import { useVessels } from "@/lib/useVessels";
// ── Leaflet icon fix ──────────────────────────────────────────────────────────
delete (L.Icon.Default.prototype as unknown as Record<string, unknown>)._getIconUrl;
@@ -90,6 +92,73 @@ function nodeIcon(status: NodeStatus): L.DivIcon {
});
}
// ── Aircraft icon — node-26#9 second-SDR ADS-B overlay ────────────────────────
function aircraftIcon(trackDeg: number | null): L.DivIcon {
const size = 16;
const rotation = trackDeg ?? 0;
return L.divIcon({
className: "",
html: `<div style="width:${size}px;height:${size}px;transform:rotate(${rotation}deg)"><svg width="${size}" height="${size}" viewBox="0 0 24 24" fill="var(--accent)" stroke="var(--surface)" stroke-width="1"><path d="M12 2 L15 11 L22 15 L15 15.5 L14 21 L17 22.5 L12 21.5 L7 22.5 L10 21 L9 15.5 L2 15 L9 11 Z"/></svg></div>`,
iconSize: [size, size],
iconAnchor: [size / 2, size / 2],
});
}
function AircraftLayer() {
const { aircraft } = useAircraft();
return (
<>
{aircraft
.filter((a) => a.lat != null && a.lon != null)
.map((a) => (
<Marker key={a.icao} position={[a.lat as number, a.lon as number]} icon={aircraftIcon(a.track_deg)}>
<Popup minWidth={160}>
<div className="space-y-1">
<div className="font-semibold">{a.callsign || a.icao}</div>
<div className="text-xs text-ink-muted">ICAO {a.icao}</div>
{a.altitude_ft != null && <div className="text-xs">Altitude: {Math.round(a.altitude_ft)} ft</div>}
{a.ground_speed_kt != null && <div className="text-xs">Speed: {Math.round(a.ground_speed_kt)} kt</div>}
</div>
</Popup>
</Marker>
))}
</>
);
}
// ── Vessel icon — node-26#9 second-SDR AIS overlay ─────────────────────────────
function vesselIcon(headingDeg: number | null): L.DivIcon {
const size = 14;
const rotation = headingDeg ?? 0;
return L.divIcon({
className: "",
html: `<div style="width:${size}px;height:${size}px;transform:rotate(${rotation}deg)"><svg width="${size}" height="${size}" viewBox="0 0 24 24" fill="var(--accent)" stroke="var(--surface)" stroke-width="1"><path d="M12 2 L18 14 L18 20 L6 20 L6 14 Z"/></svg></div>`,
iconSize: [size, size],
iconAnchor: [size / 2, size / 2],
});
}
function VesselLayer() {
const { vessels } = useVessels();
return (
<>
{vessels
.filter((v) => v.lat != null && v.lon != null)
.map((v) => (
<Marker key={v.mmsi} position={[v.lat as number, v.lon as number]} icon={vesselIcon(v.heading_deg)}>
<Popup minWidth={160}>
<div className="space-y-1">
<div className="font-semibold">{v.name || v.mmsi}</div>
<div className="text-xs text-ink-muted">MMSI {v.mmsi}</div>
{v.speed_kt != null && <div className="text-xs">Speed: {Math.round(v.speed_kt)} kt</div>}
</div>
</Popup>
</Marker>
))}
</>
);
}
function nodeFanIcon(members: NodeRecord[]): L.DivIcon {
const n = members.length;
const CARD = 13;
@@ -577,6 +646,20 @@ export default function MapView({ nodes, activeCalls, incidents = [], calls = []
</FeatureGroup>
</LayersControl.Overlay>
{/* Overlay: Aircraft — node-26#9 second-SDR ADS-B live snapshot, opt-in */}
<LayersControl.Overlay name="Aircraft">
<FeatureGroup>
<AircraftLayer />
</FeatureGroup>
</LayersControl.Overlay>
{/* Overlay: Vessels — node-26#9 second-SDR AIS live snapshot, opt-in */}
<LayersControl.Overlay name="Vessels">
<FeatureGroup>
<VesselLayer />
</FeatureGroup>
</LayersControl.Overlay>
{/* Overlay: Weather Radar — NEXRAD via Iowa Env Mesonet; key forces remount on refresh */}
<LayersControl.Overlay name="Weather Radar">
<TileLayer
+449
View File
@@ -0,0 +1,449 @@
"use client";
// ---------------------------------------------------------------------------
// Replay — re-run the intelligence pipeline over a past time range into a
// sandbox, then compare runs. Backend: drb-c2-core/app/internal/replay.py.
// Nothing here touches a live call or incident; every run spends real AI
// credits, so the form estimates first and only then offers Start.
// ---------------------------------------------------------------------------
import { useCallback, useEffect, useState } from "react";
import { c2api } from "@/lib/c2api";
import { useSystems } from "@/lib/useSystems";
import type {
ReplayEstimate, ReplayIncident, ReplayIncidents, ReplayMode, ReplayRun, ReplayCallRow,
} from "@/lib/types";
const MODES: { key: ReplayMode; label: string; help: string }[] = [
{ key: "transcripts", label: "Saved transcripts", help: "Reuse each call's transcript; re-run extraction + correlation." },
{ key: "audio", label: "Re-transcribe audio", help: "Whisper the saved audio again, then extract + correlate. For ranges where AI was off." },
{ key: "reuse", label: "Correlation only", help: "Reuse an earlier run's extraction; re-run correlation only. Isolates a correlator change." },
];
// Clears that came from the radio traffic itself, vs the 90-minute idle timer.
const SIGNAL_RESOLVES = ["units_cleared", "llm_closure", "reassignment", "children_resolved"];
const input =
"bg-gray-800 border border-gray-700 rounded-lg px-3 py-1.5 text-white text-sm font-mono focus:outline-none focus:border-indigo-500";
const btn =
"bg-gray-800 hover:bg-gray-700 disabled:opacity-50 border border-gray-700 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors";
function toLocalInput(d: Date): string {
const pad = (n: number) => String(n).padStart(2, "0");
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}T${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
function fmtTime(iso: string | null | undefined): string {
if (!iso) return "—";
return new Date(iso).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit" });
}
function sum(rec: Record<string, number> | undefined, keys: string[]): number {
return keys.reduce((n, k) => n + (rec?.[k] ?? 0), 0);
}
// ---------------------------------------------------------------------------
function NewRunForm({ runs, busy, onStarted }: { runs: ReplayRun[]; busy: boolean; onStarted: () => void }) {
const { systems } = useSystems();
const [from, setFrom] = useState(() => toLocalInput(new Date(Date.now() - 6 * 3600_000)));
const [to, setTo] = useState(() => toLocalInput(new Date()));
const [mode, setMode] = useState<ReplayMode>("transcripts");
const [systemIds, setSystemIds] = useState<string[]>([]);
const [sourceRun, setSourceRun] = useState("");
const [label, setLabel] = useState("");
const [est, setEst] = useState<ReplayEstimate | null>(null);
const [working, setWorking] = useState(false);
const [error, setError] = useState<string | null>(null);
// Any change to what would be replayed invalidates the estimate.
useEffect(() => { setEst(null); }, [from, to, mode, systemIds]);
const sources = runs.filter((r) => r.status === "done" && r.mode !== "reuse");
const range = () => ({ date_from: new Date(from).toISOString(), date_to: new Date(to).toISOString() });
async function estimate() {
setWorking(true); setError(null);
try {
setEst(await c2api.estimateReplay({ ...range(), mode, system_ids: systemIds }));
} catch (e) { setError(String(e)); } finally { setWorking(false); }
}
async function start() {
setWorking(true); setError(null);
try {
await c2api.startReplay({
...range(), mode, system_ids: systemIds, label,
source_run_id: mode === "reuse" ? sourceRun : null,
});
setEst(null); setLabel("");
onStarted();
} catch (e) { setError(String(e)); } finally { setWorking(false); }
}
function toggleSystem(id: string) {
setSystemIds((s) => (s.includes(id) ? s.filter((x) => x !== id) : [...s, id]));
}
const canStart = est && !est.truncated && est.calls > 0 && !busy && (mode !== "reuse" || sourceRun);
return (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 space-y-4">
<div className="flex flex-wrap items-end gap-4">
<div>
<label className="text-xs text-gray-400 block mb-1">From</label>
<input type="datetime-local" value={from} max={to} onChange={(e) => setFrom(e.target.value)} className={input} />
</div>
<div>
<label className="text-xs text-gray-400 block mb-1">To</label>
<input type="datetime-local" value={to} min={from} onChange={(e) => setTo(e.target.value)} className={input} />
</div>
<div>
<label className="text-xs text-gray-400 block mb-1">Label</label>
<input value={label} onChange={(e) => setLabel(e.target.value)} placeholder="what changed?" className={`${input} w-56`} />
</div>
</div>
<div className="space-y-1">
{MODES.map((m) => (
<label key={m.key} className="flex items-start gap-2 text-sm font-mono cursor-pointer">
<input type="radio" checked={mode === m.key} onChange={() => setMode(m.key)} className="mt-1" />
<span className="text-white">{m.label}</span>
<span className="text-gray-500 text-xs mt-0.5">{m.help}</span>
</label>
))}
</div>
{mode === "reuse" && (
<div>
<label className="text-xs text-gray-400 block mb-1">Reuse extraction from</label>
<select value={sourceRun} onChange={(e) => setSourceRun(e.target.value)} className={input}>
<option value="">— pick a finished run —</option>
{sources.map((r) => (
<option key={r.run_id} value={r.run_id}>
{r.label || r.run_id} · {fmtTime(r.date_from)} → {fmtTime(r.date_to)}
</option>
))}
</select>
<p className="text-xs text-gray-500 mt-1">Set the range to match that run; calls outside it are skipped.</p>
</div>
)}
{systems.length > 1 && (
<div className="flex flex-wrap gap-3">
<span className="text-xs text-gray-400">Systems (none = all):</span>
{systems.map((s) => (
<label key={s.system_id} className="flex items-center gap-1 text-xs font-mono text-gray-300 cursor-pointer">
<input type="checkbox" checked={systemIds.includes(s.system_id)} onChange={() => toggleSystem(s.system_id)} />
{s.name}
</label>
))}
</div>
)}
<div className="flex flex-wrap items-center gap-3">
<button onClick={estimate} disabled={working} className={btn}>{working && !est ? "Counting…" : "Estimate"}</button>
<button
onClick={start}
disabled={!canStart || working}
className="bg-indigo-600 hover:bg-indigo-500 disabled:opacity-50 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors"
>
Start run
</button>
{busy && <span className="text-xs text-amber-400 font-mono">a run is in progress</span>}
</div>
{est && (
<p className="text-sm font-mono text-gray-300">
{est.truncated ? (
<span className="text-red-400">More than {est.max_calls} calls — narrow the range.</span>
) : (
<>
<span className="text-white">{est.calls}</span> calls ·{" "}
{est.calls_with_transcript} with transcripts · {est.audio_minutes} audio min ·{" "}
<span className="text-amber-400">~${est.est_cost_usd.toFixed(2)}</span> est.
{mode === "transcripts" && est.calls_with_transcript < est.calls / 2 && (
<span className="text-amber-400"> — most calls have no transcript; consider Re-transcribe audio.</span>
)}
</>
)}
</p>
)}
{error && <p className="text-red-400 text-sm font-mono">{error}</p>}
</div>
);
}
// ---------------------------------------------------------------------------
function RunsTable({
runs, activeId, selected, onSelect, onChanged,
}: {
runs: ReplayRun[]; activeId: string | null; selected: string | null;
onSelect: (id: string) => void; onChanged: () => void;
}) {
async function cancel(id: string) {
await c2api.cancelReplay(id).catch(() => undefined);
onChanged();
}
async function remove(id: string) {
if (!window.confirm("Delete this run and its sandbox incidents?")) return;
await c2api.deleteReplay(id).catch(() => undefined);
onChanged();
}
if (!runs.length) return <p className="text-sm text-gray-500 font-mono">No runs yet.</p>;
const th = "text-left text-xs text-gray-500 font-normal px-2 py-1.5 whitespace-nowrap";
const td = "px-2 py-1.5 whitespace-nowrap";
return (
<div className="overflow-x-auto">
<table className="w-full text-sm font-mono">
<thead>
<tr className="border-b border-gray-800">
<th className={th}>Run</th>
<th className={th}>Range</th>
<th className={th}>Status</th>
<th className={th} title="Incidents created">Inc</th>
<th className={th} title="Share of incidents that are a single call">1-call</th>
<th className={th} title="Calls that never linked to an incident">Orphans</th>
<th className={th} title="Closed by radio traffic (units cleared, closure, reassignment) vs the idle timer">Real clears / timeouts</th>
<th className={th} title="Correlation decisions the LLM took part in">LLM</th>
<th className={th}>Cost</th>
<th className={th}></th>
</tr>
</thead>
<tbody>
{runs.map((r) => {
const m = r.metrics;
const running = r.run_id === activeId;
return (
<tr
key={r.run_id}
onClick={() => onSelect(r.run_id)}
className={`border-b border-gray-800/60 cursor-pointer ${selected === r.run_id ? "bg-gray-800/60" : "hover:bg-gray-900"}`}
>
<td className={td}>
<div className="text-white">{r.label || r.run_id}</div>
<div className="text-xs text-gray-500">{r.mode} · {r.git_sha?.slice(0, 7)} · {fmtTime(r.created_at)}</div>
</td>
<td className={`${td} text-xs text-gray-400`}>{fmtTime(r.date_from)} → {fmtTime(r.date_to)}</td>
<td className={td}>
{running ? (
<span className="text-amber-400">{r.progress.done}/{r.progress.total}</span>
) : (
<span className={r.status === "done" ? "text-green-400" : "text-red-400"}>{r.status}</span>
)}
{r.progress.errors > 0 && <span className="text-red-400 text-xs"> · {r.progress.errors} err</span>}
</td>
<td className={td}>{m?.incidents ?? "—"}</td>
<td className={td}>{m?.single_call_pct != null ? `${m.single_call_pct}%` : "—"}</td>
<td className={td}>{m ? `${m.calls_orphaned}/${m.calls}` : "—"}</td>
<td className={td}>
{m ? (
<>
<span className="text-green-400">{sum(m.resolved_via, SIGNAL_RESOLVES)}</span>
{" / "}
<span className="text-gray-400">{m.resolved_via.idle_timeout ?? 0}</span>
</>
) : "—"}
</td>
<td className={td}>{m?.llm_decisions ?? "—"}</td>
<td className={td}>{m ? `$${m.est_cost_usd.toFixed(2)}` : `~$${r.estimate.est_cost_usd.toFixed(2)}`}</td>
<td className={td} onClick={(e) => e.stopPropagation()}>
{running ? (
<button onClick={() => cancel(r.run_id)} className="text-xs text-amber-400 hover:text-amber-300">cancel</button>
) : (
<button onClick={() => remove(r.run_id)} className="text-xs text-gray-500 hover:text-red-400">delete</button>
)}
</td>
</tr>
);
})}
</tbody>
</table>
</div>
);
}
// ---------------------------------------------------------------------------
function CallLine({ call }: { call: ReplayCallRow }) {
const [audio, setAudio] = useState<string | null>(null);
async function play() {
try {
const c = await c2api.getCall(call.call_id);
setAudio(c.audio_url);
} catch { /* audio is a convenience */ }
}
return (
<div className="py-1.5 border-t border-gray-800/60 text-xs font-mono">
<div className="flex flex-wrap gap-2 text-gray-500">
<span>{fmtTime(call.started_at)}</span>
<span>{call.talkgroup_name}</span>
{call.corr_path.map((p, i) => <span key={i} className="text-indigo-400">{p}</span>)}
{call.units?.length ? <span>units {call.units.join(", ")}</span> : null}
{call.cleared_units?.length ? <span className="text-green-400">cleared {call.cleared_units.join(", ")}</span> : null}
{call.skip_reason && <span className="text-gray-600">{call.skip_reason}</span>}
{audio ? (
<audio src={audio} controls autoPlay className="h-6" />
) : (
<button onClick={play} className="text-gray-400 hover:text-white">▶ audio</button>
)}
</div>
<div className="text-gray-200 mt-0.5">{call.transcript || <span className="text-gray-600">(no transcript)</span>}</div>
</div>
);
}
function IncidentCard({ inc }: { inc: ReplayIncident }) {
const [open, setOpen] = useState(false);
const signal = inc.resolved_via && SIGNAL_RESOLVES.includes(inc.resolved_via);
return (
<div className="bg-gray-900 border border-gray-800 rounded-lg px-3 py-2">
<button onClick={() => setOpen(!open)} className="w-full text-left flex flex-wrap items-center gap-x-3 gap-y-1 text-sm font-mono">
<span className="text-gray-500">{open ? "▾" : "▸"}</span>
<span className="text-white">{inc.title || inc.incident_id}</span>
<span className="text-gray-500 text-xs">{inc.calls.length} call{inc.calls.length !== 1 ? "s" : ""}</span>
<span className="text-gray-500 text-xs">{fmtTime(inc.started_at)} → {fmtTime(inc.resolved_at)}</span>
<span className={`text-xs ${signal ? "text-green-400" : "text-gray-500"}`}>{inc.resolved_via ?? inc.status}</span>
{inc.location_coords && <span className="text-xs text-indigo-400">📍 {inc.location}</span>}
</button>
{open && <div className="mt-2">{inc.calls.map((c) => <CallLine key={c.call_id} call={c} />)}</div>}
</div>
);
}
function RunDetail({ run }: { run: ReplayRun }) {
const [data, setData] = useState<ReplayIncidents | null>(null);
const [error, setError] = useState<string | null>(null);
const [filter, setFilter] = useState<"all" | "multi" | "single">("all");
const [showOrphans, setShowOrphans] = useState(false);
useEffect(() => {
setData(null); setError(null);
if (run.status === "running") return;
c2api.getReplayIncidents(run.run_id).then((d) => {
setData(d);
// Exposed for in-page analysis (console / automation) of a run's
// sandbox — the same data this tab renders, nothing more.
(window as unknown as { __drbReplay?: unknown }).__drbReplay = { run, ...d };
}).catch((e) => setError(String(e)));
}, [run.run_id, run.status]);
const m = run.metrics;
const shown = (data?.incidents ?? []).filter((i) =>
filter === "all" ? true : filter === "multi" ? i.calls.length > 1 : i.calls.length === 1,
);
return (
<div className="space-y-3">
{m && (
<div className="grid grid-cols-2 sm:grid-cols-4 gap-2 text-xs font-mono">
{[
["Linked calls", `${m.calls_linked}/${m.calls}`],
["Median calls / incident", m.median_calls_per_incident ?? "—"],
["With units cleared", m.incidents_with_units_cleared],
["With map pin", m.incidents_with_coords],
].map(([k, v]) => (
<div key={k as string} className="bg-gray-900 border border-gray-800 rounded-lg p-2">
<div className="text-gray-500">{k}</div>
<div className="text-white text-base">{v}</div>
</div>
))}
</div>
)}
{m && (
<p className="text-xs font-mono text-gray-500">
resolved: {Object.entries(m.resolved_via).map(([k, v]) => `${k} ${v}`).join(" · ")}
<br />
paths: {Object.entries(m.corr_path).map(([k, v]) => `${k} ${v}`).join(" · ")}
</p>
)}
{m?.ai_failures && Object.keys(m.ai_failures).length > 0 && (
<p className="text-xs font-mono text-amber-400">
AI failures: {Object.entries(m.ai_failures).map(([k, v]) => `${k} ×${v}`).join(" · ")}
</p>
)}
{run.errors?.length > 0 && (
<details className="text-xs font-mono text-red-400">
<summary>{run.errors.length} error(s)</summary>
{run.errors.map((e, i) => <div key={i}>{e}</div>)}
</details>
)}
{run.status === "running" && <p className="text-sm text-gray-500 font-mono">Incidents appear when the run finishes.</p>}
{error && <p className="text-red-400 text-sm font-mono">{error}</p>}
{data && (
<>
<div className="flex gap-1 text-xs font-mono">
{(["all", "multi", "single"] as const).map((f) => (
<button
key={f}
onClick={() => setFilter(f)}
className={`px-3 py-1 rounded-md ${filter === f ? "bg-gray-800 text-white" : "text-gray-500 hover:text-gray-300"}`}
>
{f === "all" ? `All (${data.incidents.length})` : f === "multi" ? "Multi-call" : "Single-call"}
</button>
))}
<button
onClick={() => setShowOrphans(!showOrphans)}
className={`px-3 py-1 rounded-md ${showOrphans ? "bg-gray-800 text-white" : "text-gray-500 hover:text-gray-300"}`}
>
Orphans ({data.orphans.length})
</button>
</div>
{showOrphans ? (
<div className="bg-gray-900 border border-gray-800 rounded-lg px-3 py-2">
{data.orphans.map((c) => <CallLine key={c.call_id} call={c} />)}
</div>
) : (
<div className="space-y-1.5">{shown.map((i) => <IncidentCard key={i.incident_id} inc={i} />)}</div>
)}
</>
)}
</div>
);
}
// ---------------------------------------------------------------------------
export function ReplayTab() {
const [runs, setRuns] = useState<ReplayRun[]>([]);
const [activeId, setActiveId] = useState<string | null>(null);
const [selected, setSelected] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
const load = useCallback(async () => {
try {
const res = await c2api.listReplays();
setRuns(res.runs);
setActiveId(res.active_run_id);
setError(null);
} catch (e) { setError(String(e)); }
}, []);
useEffect(() => { load(); }, [load]);
// Poll only while something is running.
useEffect(() => {
if (!activeId) return;
const t = setInterval(load, 4000);
return () => clearInterval(t);
}, [activeId, load]);
const run = runs.find((r) => r.run_id === selected) ?? null;
return (
<div className="space-y-5">
<p className="text-xs text-gray-500 font-mono">
Re-run the pipeline over a past time range, in the calls&apos; original order, into a sandbox — live incidents are never
touched. Replay the same range after each change and compare the rows below. A run starts with no incidents
open, so its first ~90 minutes split more than live did — compare runs over the same range, not a run against live.
</p>
<NewRunForm runs={runs} busy={!!activeId} onStarted={load} />
{error && <p className="text-red-400 text-sm font-mono">{error}</p>}
<RunsTable runs={runs} activeId={activeId} selected={selected} onSelect={setSelected} onChanged={load} />
{run && <RunDetail run={run} />}
</div>
);
}
+57
View File
@@ -0,0 +1,57 @@
"use client";
// A from/to pair of native date inputs. Values are the inputs' own
// "YYYY-MM-DD" strings; dayStart/dayEnd turn them into the local-midnight
// bounds a started_at range query needs, so "to" includes the whole day.
export function dayStart(ymd: string): Date | undefined {
if (!ymd) return undefined;
const [y, m, d] = ymd.split("-").map(Number);
return new Date(y, m - 1, d, 0, 0, 0, 0);
}
export function dayEnd(ymd: string): Date | undefined {
if (!ymd) return undefined;
const [y, m, d] = ymd.split("-").map(Number);
return new Date(y, m - 1, d, 23, 59, 59, 999);
}
const inputClass =
"bg-surface border border-line rounded-lg px-2 py-1.5 text-sm text-ink-2 focus:outline-none focus:border-accent";
export function DateRange({
from,
to,
onChange,
}: {
from: string;
to: string;
onChange: (from: string, to: string) => void;
}) {
return (
<div className="flex items-center gap-2 text-xs text-ink-muted">
<input
type="date"
aria-label="From date"
value={from}
max={to || undefined}
onChange={(e) => onChange(e.target.value, to)}
className={inputClass}
/>
<span>to</span>
<input
type="date"
aria-label="To date"
value={to}
min={from || undefined}
onChange={(e) => onChange(from, e.target.value)}
className={inputClass}
/>
{(from || to) && (
<button onClick={() => onChange("", "")} className="text-ink-muted hover:text-ink-2">
clear
</button>
)}
</div>
);
}
+48
View File
@@ -82,6 +82,8 @@ export const c2api = {
link?: "any" | "orphan" | "linked";
transcript?: "any" | "yes" | "no";
q?: string;
date_from?: string;
date_to?: string;
}) => {
const qs = new URLSearchParams();
for (const [k, v] of Object.entries(params)) {
@@ -100,6 +102,30 @@ export const c2api = {
closeStallCalls: (olderThanMinutes: number, dryRun: boolean) =>
request<{ dry_run: boolean; older_than_minutes: number; count: number; call_ids: string[] }>(`/calls/close-stale?older_than_minutes=${olderThanMinutes}&dry_run=${dryRun}`, { method: "POST" }),
// STT eval harness (server-26#163) — separate from patchTranscript above,
// which is a production correction with real side effects (re-extraction,
// incident unlinking, vocabulary learning). This is pure measurement.
getEvalQueue: (limit: number, cursor?: string | null) => {
const qs = new URLSearchParams({ limit: String(limit) });
if (cursor) qs.set("cursor", cursor);
return request<{
calls: import("@/lib/types").CallRecord[];
next_cursor: string | null;
scanned: number;
matched: number;
window_exhausted: boolean;
}>(`/calls/eval-queue?${qs.toString()}`);
},
getEvalStats: () =>
request<{ eval_count: number; raw_wer: number | null; corrected_wer: number | null }>(
"/calls/eval-stats"
),
putEvalTranscript: (callId: string, text: string) =>
request<{ ok: boolean; call_id: string }>(`/calls/${callId}/eval-transcript`, {
method: "PUT",
body: JSON.stringify({ text }),
}),
// Incidents
getIncidents: (params?: { status?: string; type?: string }) => {
const qs = params ? "?" + new URLSearchParams(params as Record<string, string>).toString() : "";
@@ -207,6 +233,28 @@ export const c2api = {
getCorrelationDebug: (limit: number, orphanHours: number) =>
request<unknown>(`/admin/debug/correlation?limit=${limit}&orphan_hours=${orphanHours}`),
// Replay (admin) — re-run the pipeline over past calls into a sandbox.
// See drb-c2-core/app/internal/replay.py.
estimateReplay: (p: { date_from: string; date_to: string; mode: import("@/lib/types").ReplayMode; system_ids?: string[] }) => {
const qs = new URLSearchParams({ date_from: p.date_from, date_to: p.date_to, mode: p.mode });
if (p.system_ids?.length) qs.set("system_ids", p.system_ids.join(","));
return request<import("@/lib/types").ReplayEstimate>(`/admin/replay/estimate?${qs}`);
},
startReplay: (body: {
date_from: string; date_to: string; mode: import("@/lib/types").ReplayMode;
system_ids?: string[]; source_run_id?: string | null; label?: string;
}) =>
request<import("@/lib/types").ReplayRun>("/admin/replay", { method: "POST", body: JSON.stringify(body) }),
listReplays: () =>
request<{ runs: import("@/lib/types").ReplayRun[]; active_run_id: string | null }>("/admin/replay"),
getReplay: (runId: string) => request<import("@/lib/types").ReplayRun>(`/admin/replay/${runId}`),
cancelReplay: (runId: string) =>
request<{ ok: boolean }>(`/admin/replay/${runId}/cancel`, { method: "POST" }),
deleteReplay: (runId: string) =>
request<{ ok: boolean }>(`/admin/replay/${runId}`, { method: "DELETE" }),
getReplayIncidents: (runId: string) =>
request<import("@/lib/types").ReplayIncidents>(`/admin/replay/${runId}/incidents`),
// Preferred bot token per system
setPreferredToken: (tokenId: string, systemId: string) =>
request<{ ok: boolean; preferred_for_system_id: string | null }>(`/tokens/${tokenId}/prefer/${systemId}`, { method: "PUT" }),
+122
View File
@@ -53,12 +53,39 @@ export interface NodeRecord {
hardware_preset?: string;
ppm_override?: number | null;
node_type?: string;
secondary_sdr_mode?: string;
sdr_count?: number;
enforce_override_timeout?: boolean;
is_overridden?: boolean;
override_system_id?: string | null;
override_timeout_at?: string | null;
}
export interface AircraftTrack {
icao: string;
org_id?: string;
node_id: string;
callsign: string | null;
lat: number | null;
lon: number | null;
altitude_ft: number | null;
ground_speed_kt: number | null;
track_deg: number | null;
last_seen: string;
}
export interface VesselTrack {
mmsi: string;
org_id?: string;
node_id: string;
name: string | null;
lat: number | null;
lon: number | null;
speed_kt: number | null;
heading_deg: number | null;
last_seen: string;
}
export interface VocabularyPendingTerm {
term: string;
source: "induction" | "correction";
@@ -131,6 +158,10 @@ export interface CallRecord {
corr_incident_idle_min?: number | null;
corr_shared_units?: number | null;
corr_candidates?: number | null;
/** Human-verified reference transcript for the STT eval harness (server-26#163) — never read by anything downstream. */
eval_transcript?: string | null;
eval_transcript_by?: string | null;
eval_transcript_at?: string | null;
}
export interface IncidentRecord {
@@ -275,3 +306,94 @@ export interface TalkgroupPending {
talkgroup_name?: string;
pending: PendingLocalTerm[];
}
// ---------------------------------------------------------------------------
// Replay (admin) — drb-c2-core/app/internal/replay.py
// ---------------------------------------------------------------------------
export type ReplayMode = "audio" | "transcripts" | "reuse";
export interface ReplayEstimate {
calls: number;
calls_with_transcript: number;
calls_with_audio: number;
audio_minutes: number;
est_cost_usd: number;
truncated: boolean;
max_calls: number;
}
export interface ReplayMetrics {
calls: number;
calls_linked: number;
calls_orphaned: number;
incidents: number;
single_call_incidents: number;
single_call_pct: number | null;
median_calls_per_incident: number | null;
max_calls_in_incident: number | null;
incidents_with_units_cleared: number;
incidents_with_coords: number;
resolved_via: Record<string, number>;
corr_path: Record<string, number>;
corr_consensus: Record<string, number>;
llm_decisions: number;
est_cost_usd: number;
ai_failures?: Record<string, number>;
}
export interface ReplayRun {
run_id: string;
label: string;
mode: ReplayMode;
source_run_id: string | null;
date_from: string;
date_to: string;
system_ids: string[];
git_sha: string;
created_by: string;
created_at: string;
finished_at?: string;
status: "running" | "done" | "cancelled" | "failed" | "interrupted";
estimate: Omit<ReplayEstimate, "truncated" | "max_calls">;
progress: { total: number; done: number; errors: number; skipped?: number };
metrics: ReplayMetrics | null;
errors: string[];
}
export interface ReplayCallRow {
call_id: string;
started_at: string;
talkgroup_name: string | null;
transcript: string | null;
units: string[] | null;
cleared_units: string[] | null;
location: string | null;
skip_reason: string | null;
corr_path: string[];
incident_ids: string[];
}
export interface ReplayIncident {
incident_id: string;
title: string | null;
type: string | null;
severity: string | null;
status: string;
resolved_via: string | null;
started_at: string;
updated_at: string | null;
resolved_at: string | null;
location: string | null;
location_coords: { lat: number; lng: number } | null;
units: string[] | null;
units_active: string[] | null;
units_cleared: string[] | null;
talkgroup_ids: number[] | null;
calls: ReplayCallRow[];
}
export interface ReplayIncidents {
incidents: ReplayIncident[];
orphans: ReplayCallRow[];
}
+52
View File
@@ -0,0 +1,52 @@
"use client";
import { useEffect, useState } from "react";
import { collection, onSnapshot, query, where, FirestoreError } from "firebase/firestore";
import { onAuthStateChanged } from "firebase/auth";
import { db, auth } from "@/lib/firebase";
import { useAuth } from "@/components/AuthProvider";
import type { AircraftTrack } from "@/lib/types";
// `aircraft` docs are a live snapshot (one per icao, overwritten on every
// sighting, node-26#9) — nothing prunes a doc when a plane leaves range, so
// staleness is filtered client-side rather than assuming the collection only
// ever holds current traffic.
const STALE_AFTER_MS = 2 * 60 * 1000;
export function useAircraft() {
const [aircraft, setAircraft] = useState<AircraftTrack[]>([]);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const { orgId } = useAuth();
useEffect(() => {
let unsubFirestore: (() => void) | undefined;
const unsubAuth = onAuthStateChanged(auth, (user) => {
if (unsubFirestore) { unsubFirestore(); unsubFirestore = undefined; }
if (!user || !orgId) {
setAircraft([]);
setLoading(false);
return;
}
const q = query(collection(db, "aircraft"), where("org_id", "==", orgId));
unsubFirestore = onSnapshot(q, (snap) => {
const now = Date.now();
const fresh = snap.docs
.map((d) => d.data() as AircraftTrack)
.filter((a) => now - new Date(a.last_seen).getTime() < STALE_AFTER_MS);
setAircraft(fresh);
setLoading(false);
}, (err: FirestoreError) => { console.error("useAircraft:", err); setError(err.message); setLoading(false); });
});
return () => {
unsubAuth();
if (unsubFirestore) unsubFirestore();
};
}, [orgId]);
return { aircraft, loading, error };
}
+20 -3
View File
@@ -11,12 +11,19 @@ const toISO = (v: unknown): string =>
(v as { toDate?: () => Date })?.toDate?.()?.toISOString?.() ??
(typeof v === "string" ? v : new Date().toISOString());
export function useIncidents(limitCount = 100) {
export function useIncidents(limitCount = 100, dateFrom?: Date, dateTo?: Date) {
const [incidents, setIncidents] = useState<IncidentRecord[]>([]);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
// A full page means there may be older incidents past the limit; a short
// page means the query reached the end of the collection.
const [hasMore, setHasMore] = useState(false);
const { orgId } = useAuth();
// Stable ms values so the effect dependency doesn't fire on every render
const dateFromMs = dateFrom?.getTime();
const dateToMs = dateTo?.getTime();
useEffect(() => {
let unsubFirestore: (() => void) | undefined;
@@ -34,9 +41,18 @@ export function useIncidents(limitCount = 100) {
return;
}
// A range on the ordered field rides the existing org_id/started_at index.
// Incident started_at is stored as a Python isoformat() STRING
// ("2026-09-20T12:00:00.123456+00:00", incident_correlator.py), not a
// Firestore timestamp — unlike calls. A Date bound compares by type and
// matches nothing, so the bounds go in as UTC ISO strings in the same
// shape, which then compare lexicographically in time order.
const isoBound = (ms: number) => new Date(ms).toISOString().replace("Z", "+00:00");
const q = query(
collection(db, "incidents"),
where("org_id", "==", orgId),
...(dateFromMs != null ? [where("started_at", ">=", isoBound(dateFromMs))] : []),
...(dateToMs != null ? [where("started_at", "<=", isoBound(dateToMs))] : []),
orderBy("started_at", "desc"),
limit(limitCount)
);
@@ -49,6 +65,7 @@ export function useIncidents(limitCount = 100) {
updated_at: toISO(data.updated_at),
} as IncidentRecord;
}));
setHasMore(snap.size >= limitCount);
setLoading(false);
}, (err: FirestoreError) => {
console.error("useIncidents:", err);
@@ -61,9 +78,9 @@ export function useIncidents(limitCount = 100) {
unsubAuth();
if (unsubFirestore) unsubFirestore();
};
}, [limitCount, orgId]);
}, [limitCount, dateFromMs, dateToMs, orgId]);
return { incidents, loading, error };
return { incidents, loading, error, hasMore };
}
export function useIncident(incidentId: string | null) {
+53
View File
@@ -0,0 +1,53 @@
"use client";
import { useEffect, useState } from "react";
import { collection, onSnapshot, query, where, FirestoreError } from "firebase/firestore";
import { onAuthStateChanged } from "firebase/auth";
import { db, auth } from "@/lib/firebase";
import { useAuth } from "@/components/AuthProvider";
import type { VesselTrack } from "@/lib/types";
// Same shape as useAircraft — `vessels` is a live snapshot (one per mmsi,
// overwritten on every sighting, node-26#9), nothing prunes a doc when a
// vessel goes out of range, so staleness is filtered client-side. AIS
// position reports are much less frequent than ADS-B (minutes, not
// seconds), so this window is longer than useAircraft's.
const STALE_AFTER_MS = 10 * 60 * 1000;
export function useVessels() {
const [vessels, setVessels] = useState<VesselTrack[]>([]);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const { orgId } = useAuth();
useEffect(() => {
let unsubFirestore: (() => void) | undefined;
const unsubAuth = onAuthStateChanged(auth, (user) => {
if (unsubFirestore) { unsubFirestore(); unsubFirestore = undefined; }
if (!user || !orgId) {
setVessels([]);
setLoading(false);
return;
}
const q = query(collection(db, "vessels"), where("org_id", "==", orgId));
unsubFirestore = onSnapshot(q, (snap) => {
const now = Date.now();
const fresh = snap.docs
.map((d) => d.data() as VesselTrack)
.filter((v) => now - new Date(v.last_seen).getTime() < STALE_AFTER_MS);
setVessels(fresh);
setLoading(false);
}, (err: FirestoreError) => { console.error("useVessels:", err); setError(err.message); setLoading(false); });
});
return () => {
unsubAuth();
if (unsubFirestore) unsubFirestore();
};
}, [orgId]);
return { vessels, loading, error };
}
+12
View File
@@ -95,6 +95,18 @@ service cloud.firestore {
allow write: if false;
}
// Live map overlays fed by a node's second SDR (node-26#9). Snapshot
// docs, one per icao/mmsi, last-seen-wins — not a history collection.
match /aircraft/{icao} {
allow read: if docInMyOrg();
allow write: if false;
}
match /vessels/{mmsi} {
allow read: if docInMyOrg();
allow write: if false;
}
match /alert_events/{alertId} {
allow read: if docInMyOrg();
allow write: if false;