Compare commits

..
Author SHA1 Message Date
Logan CusanoandClaude Opus 5.5 9f19750ea6 stops review + provisional LLM closure
Replay of 433b35d (server-26#170): traffic stops now open incidents
("45 Adam" stop, "CM2" stop), but the bridge MVA split 33/56 — one
transmission ("transport complete") was read as scene-resolved at 14:44
and an LLM closure was final, so the rest of the MVA opened a new one.

- LLM closure is now provisional (reopenable), like a timer close: it is
  inferred from a single transmission.
- review of 433b35d: backstop only matches a unit's own "on a stop"
  self-report (bare "car stop" mentions and "pull over" dropped), never on
  MTA/rail/bridge/fire/EMS/DPW talkgroups ("Train 4 holding on the stop"),
  negation looks 5 words back, and <=5-word reports ("Adam 3 on a stop")
  get a minimal scene instead of being skipped before the backstop.

c2-core: 471 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 20:52:31 -04:00
logan 433b35d2ba Merge pull request 'intelligence: traffic stops and self-initiated activity open incidents' (#178) from feat/traffic-stops into main
Build & Deploy / Build & push images (push) Successful in 4m43s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 6s
Build & Deploy / Deploy to VM (push) Successful in 1m49s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 20:22:04 -04:00
Logan CusanoandClaude Opus 5.5 badfe28823 intelligence: traffic stops and self-initiated activity open incidents
Owner: traffic stops should show on the portal — otherwise they are only
visible in the archive. In the 09-22 replay (server-26#170) every Ch 1
stop ("45 Adam on a stop, Eastbound Central Express", "CM2 on the stop,
southbound") came back from extraction untyped, untagged and routine —
read as status traffic after #138 — so the creation gate never opened one.

- prompt: a unit reporting its own activity (on a stop, out with a vehicle
  or pedestrian) is a real event: police, tagged, at least minor; the plate
  lookups for it belong to it.
- deterministic backstop after extraction: stop / "put me out with"
  phrasing adds a "traffic-stop" / "self-initiated" tag (the substance the
  creation gate counts), police type if none, minor if routine. Negated
  phrasing ("not pull the car over") is left alone; nothing is downgraded.

c2-core: 469 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 20:22:01 -04:00
logan 737bdf0576 Merge pull request 'reopen: act on drb-correlation-review of e972cac' (#177) from fix/reopen-review into main
Build & Deploy / Build & push images (push) Successful in 4m26s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m45s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 19:43:38 -04:00
Logan CusanoandClaude Opus 5.5 b9e7524817 reopen: act on drb-correlation-review of e972cac
- only a substantive call after the close reopens a timer-closed incident;
  a thin "10-4" (doesn't refresh updated_at) or a sweep link of a call from
  before the close rides along without reopening — otherwise the next
  sweep closed it again and the portal flickered.
- substantive_call_count counts a call once, not once per scene.
- a timer-closed incident is never adopted as a cross-system parent.

Replay e972cac (correlation-only on 731b54b's extraction) vs the 09-22
answer key: pairwise F1 0.548 -> 0.807 (precision 0.839 -> 0.956, recall
0.407 -> 0.698); the bridge MVA is one 76-call incident instead of two
40-call halves. server-26#170.

c2-core: 468 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 19:43:30 -04:00
logan e972cace4a Merge pull request 'incidents: severity-scaled quiet timer, reopen-on-link, thin calls do not fill the cap' (#176) from feat/provisional-close into main
Build & Deploy / Build & push images (push) Successful in 4m26s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m8s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 19:14:38 -04:00
Logan CusanoandClaude Opus 5.5 969d175a67 incidents: severity-scaled quiet timer, reopen-on-link, thin calls don't fill the cap
Hand-labelling the 09-22 10:00-12:00 ET replay window (server-26#170,
answer key replay_groundtruth_0922.json) found ~25 real incidents, of which
only ~5 had an audible clear — most jobs clear by MDT, so the quiet timer is
the close for most incidents and a flat 90 minutes left a lockout or a plate
check "active" on the portal an hour after it ended.

- summarizer: timer close after 30 min quiet for routine/minor, 60 moderate,
  90 major/unknown. A timer close is provisional: reopenable=True.
- correlator: reopenable incidents inside incident_reopen_window_minutes
  (90, since last substantive call) stay candidates; linking a call to one
  reopens it (status active, reopened_count++). The sweep expires the flag
  so the reopenable pool stays bounded. Real clears (units_cleared,
  llm_closure) are never reopenable.
- cap: incident_max_calls counts substantive calls only
  (substantive_call_count). The bridge MVA hit 40 in 32 min with ~40% thin
  replies, split in half, and the second half took another job's title.

c2-core: 467 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 19:14:35 -04:00
logan 731b54bed9 Merge pull request 'clearance: act on drb-correlation-review of f0a88d4' (#175) from fix/clearance-review into main
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 4s
Build & Deploy / Deploy to VM (push) Successful in 1m46s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 17:26:41 -04:00
Logan CusanoandClaude Opus 5.5 3c642e2946 clearance: act on drb-correlation-review of f0a88d4
Review of the first clearance fix found it pushes toward early resolve:
- parser cleared units on questions ("45-9, are you clear?"), negations
  ("not clear yet"), orders ("clear the scene", "clear to transport"),
  places/times ("Room 2 clear", "1400 hours, clear") and split "45 9" into
  unit 45. Now rejects "?", not/is/are/you, anything after the status word
  but sign-offs, place/time words; joins "45 9" -> "45-9".
- a clear from a unit never active on an incident was recorded in
  units_cleared and could pass the all-clear gate. Only units actually
  active there can clear there now.
- clearance-only calls skip the LLM tier (same as thin calls): only the
  rules engine's unit match can say which incident a 10-8 belongs to.
- replay incident view carries srcaddr/srcaddrs for the radio-ID clearance
  investigation.

Replay 09-22 10:00-12:00 ET with f0a88d4: real clears 0 -> 2 (both LLM
closure), unit clears still 0 — the parsed clears are right but those
units were never recorded as assigned (Whisper mangles unit IDs at
dispatch), which this commit does not fix.

c2-core: 465 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 17:26:38 -04:00
logan f0a88d401c Merge pull request 'correlator/intelligence: let 10-8s actually close incidents' (#174) from fix/clearance-signal into main
Build & Deploy / Build & push images (push) Successful in 4m11s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m36s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 16:49:33 -04:00
Logan CusanoandClaude Opus 5.5 8eac32caf5 correlator/intelligence: let 10-8s actually close incidents
Replay of 09-22 10:00-12:00 ET (server-26#170): 0 of 19 incidents resolved
on a clear, 19 on the idle timer, although 25 transmissions said 10-8/clear.
Three independent breaks:

1. Short clears never reached extraction. "45-9, I'm clear." is <=5 words,
   so extract_scenes skipped it before GPT and cleared_units stayed empty.
   A rule parser now names the unit when it precedes the status word
   (never guesses: "10-8, 10-8." / "CMT clear." clear nobody) and returns a
   minimal scene that links by unit overlap but cannot open an incident.
2. Clearance compared unit strings exactly, so "11-Adam" clearing never
   removed "11 Adam". Now by _normalize_unit key.
3. units_active collected "Desk", "Central", "Division", "unknown", plate
   phonetics — none of which ever clear, so all-clear could never pass.
   Only units carrying a number (and not a ten-code) are tracked now; the
   rest stay in `units` for matching.

c2-core: 463 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 16:49:30 -04:00
logan 6e82ee8579 Merge pull request 'correlation: smart tiebreak model gemini-2.5-pro is gone; use gemini-3.8-flash' (#173) from fix/smart-model into main
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m50s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 16:07:56 -04:00
Logan CusanoandClaude Opus 5.5 cdc61dcc9d correlation: smart tiebreak model gemini-2.5-pro is gone; use gemini-3.8-flash
The first replay (server-26#170) aborted on 5x 'model is unavailable'
from the tiebreak tier. Google's model list shows 2.5 closed to new
projects and no stable Pro model; 3.8-flash is the newest stable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 16:07:53 -04:00
logan 032e9bd653 Merge pull request 'Replay: fail fast on a dead AI account; extraction reports to ai_health' (#172) from fix/replay-visibility into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m35s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 15:39:04 -04:00
Logan CusanoandClaude Opus 5.5 ec91a9175f Replay: fail fast on a dead AI account; extraction reports to ai_health
First replay (290 calls, 09-22 10:00-12:00 ET) produced 0 incidents and
no errors: every gpt-4o-mini extraction failed and _sync_extract
swallowed it as "no scenes". Same shape as #169 — and the live extraction
tier in /health/ai had no reporter at all, so this has been invisible in
production too.

- intelligence: API failures propagate out of _sync_extract; extract_scenes
  reports them to ai_health ("extraction" tier, billing/dead-model
  classified) and still returns [] so the pipeline degrades as before.
- ai_health: inside a replay sandbox, failures go to the run's own sink
  instead of being dropped.
- replay: aborts after 5 permanent failures on a tier, naming the cause;
  run metrics carry ai_failures; UI shows them.
- replay estimate: audio minutes from started_at/ended_at (no duration
  field exists on call docs).
- ReplayTab exposes the loaded run on window.__drbReplay for in-page
  analysis.

c2-core: 458 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 15:39:01 -04:00
logan 8dadbdd977 Merge pull request 'Admin Replay: re-run the pipeline over past calls in a sandbox' (#171) from feat/replay into main
Build & Deploy / Build & push images (push) Successful in 4m15s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m31s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-26 15:24:37 -04:00
Logan CusanoandClaude Opus 5.5 aff3f16d32 Admin Replay: re-run the pipeline over past calls in a sandbox (#170)
Correlation has only ever been measured through live AI windows: days of
wall time per change, and the 09-20→22 window was invalidated outright by
unfunded AI accounts (#169). Recordings are kept regardless of AI, so the
traffic to measure against already exists.

- internal/replay.py: runs a time range of real calls through the live
  pipeline code in original order, clock pinned per call, into
  replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe),
  transcripts (re-extract), reuse (correlation only from a prior run's
  scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on
  virtual time. No alerts, summaries, vocab, AI-health alerts or pending
  terms. One run at a time, <=5000 calls, <=7 days.
- firestore.py: ContextVar sandbox redirect for calls/incidents.
- clock.py: ContextVar-pinnable now(), used on the correlation path.
- feature_flags.py: ContextVar flag override so replay runs with live AI off.
- upload.py: scene loop extracted to _extract_and_correlate, shared by the
  live pipeline and replay so replay measures the code that runs live.
- resolved_via on every incident resolve, so a real clear can be told
  from the idle timeout — live and in replay.
- routers/replay.py + /admin Replay tab: estimate, start, compare runs,
  drill into incidents with audio.

Reviewed by drb-correlation-review; its leak and fidelity findings are
fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run
(no Node on the authoring box).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 15:23:57 -04:00
logan e79b8bc37d Merge pull request 'Incident date filter matched nothing' (#168) from fix/incident-date-filter into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m27s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:15:31 -04:00
Logan CusanoandClaude Opus 5.5 c72c28f5dc frontend: incident date filter matched nothing
Incident started_at is an isoformat() string (incident_correlator.py,
routers/incidents.py), not a Firestore timestamp like calls. The range
bounds were Dates, which Firestore compares by type, so any date range
returned zero incidents. Bounds are now UTC ISO strings in the same
"+00:00" shape, which order lexicographically by time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:15:23 -04:00
logan 02b5b7b5a5 Merge pull request 'Archive Load more skipped 150 of every 200 calls' (#166) from fix/archive-paging-skip into main
Build & Deploy / Build & push images (push) Successful in 4m17s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 3m45s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:05:37 -04:00
Logan CusanoandClaude Opus 5.5 40014a47a3 c2-core: Archive "Load more" skipped 150 of every 200 calls
/calls/search scans a 200-row window and returns 50, but the next cursor
was always the last SCANNED row — so with an unfiltered list each page
jumped past the 150 matches it had already read and not shown. Resume
after the last RETURNED row when matches overflow the page; keep the
last-scanned cursor only when the page holds every match (the sparse-
filter case that cursor exists for). Same fix for /calls/eval-queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:05:31 -04:00
logan 6c0e7a4f8e Merge pull request 'Date range picker on Incidents and Archive; fix Archive Load more' (#165) from feat/date-range into main
Build & Deploy / Build & push images (push) Successful in 4m22s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m35s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-24 01:04:06 -04:00
Logan CusanoandClaude Opus 5.5 6479174022 frontend: date range picker on Incidents and Archive; fix Archive paging
Incidents and Archive get a from/to date range (native date inputs,
local-day bounds). Incidents filters in the Firestore query; Archive
passes date_from/date_to to GET /calls/search, which applies them as a
started_at range — both ride the existing org_id/started_at index.

Also fixes /calls/search and /calls/eval-queue paging: the cursor went to
Firestore as a raw ISO string against a timestamp field, which compares
by type rather than time, so "Load more" re-read the first page. Cursor
and range bounds are now parsed to datetimes (400 on garbage).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:03:40 -04:00
logan c043298902 Merge pull request 'Incidents search/filter/load-more; Archive viewable by viewers' (#164) from feat/archive-search-viewers into main
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 4s
Build & Deploy / Deploy to VM (push) Successful in 2m26s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-23 23:54:35 -04:00
Logan CusanoandClaude Opus 5.5 fa194e0f0a frontend: search, filters and load-more on Incidents; open Archive to viewers
Incidents page: text search (title, location, summary, units, vehicles,
tags, location mentions), status and type filters, and a Load more button
that pages the Firestore query 100 at a time. Filtering runs over the
loaded window, and the page says so when older incidents exist.

Archive (/calls): readable by every org member, not just admins.
GET /calls/search now takes any Firebase token scoped to the caller's org
— the Firestore rules already let members read every call in their org,
so this widens nothing. Attach/detach stays admin-only (UI and routes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 23:53:46 -04:00
Logan CusanoandClaude Sonnet 5 5f85a878fa admin: STT eval harness — record human-verified transcripts, measure real WER (#163)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 1m56s
Build & Deploy / Report a failed deploy (push) Successful in 1s
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.

  GET  /calls/eval-queue        -- calls with a transcript but no
                                    eval_transcript yet, paged (same bounded-
                                    window-plus-cursor shape as /search)
  PUT  /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
                                      never touches transcript/transcript_corrected
  GET  /calls/eval-stats        -- eval_count + average word error rate of
                                    the raw and corrected machine transcripts
                                    against the human-verified ones

internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.

Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.

Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-21 00:04:30 -04:00
Logan CusanoandClaude Sonnet 5 241a15b8da transcript_correction: reject a correction that changes a ten-code (#162)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m1s
Build & Deploy / Report a failed deploy (push) Successful in 1s
correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add
information", but nothing checked the model's output against its own
rules -- raw["corrected"] was accepted verbatim past a non-empty/changed
check, and the "changed" field it returns was logged for audit and never
validated.

Caught live: the same call's raw vs corrected transcript showed "10-7"
rewritten to "10-13" in one place and "10-4" in another, plus a bare "7"
expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes
Street and 4th and Rowe -> Home Street and Forest Ave, from this system's
own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly
as trustworthy as a correct transcript, which is worse than leaving the
raw mishearing in place.

_code_tokens() extracts every ten-code/signal-shaped token from the
original and corrected text/segments; any change to that set discards
the correction and falls back to raw. Checked independently for the
joined text and for segments, consistent with the existing
all-or-nothing segment-alignment rule.

Does not catch a wrong word swapped for another equally plausible
non-code word -- that class still depends entirely on the model
following its own prompt. Filed server-26#161 for the broader STT/audio
quality initiative this belongs alongside.

Verified: 426 pass, 0 fail (4 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 21:51:41 -04:00
Logan CusanoandClaude Sonnet 5 f91d4559f3 deploy: treat an empty .last_good_tag the same as a missing one (#156)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m0s
Build & Deploy / Report a failed deploy (push) Successful in 1s
cat file 2>/dev/null || echo latest only falls back when cat itself fails
(nonzero exit, i.e. the file is missing) -- a file that EXISTS but is EMPTY
makes cat succeed with empty output, so PREV_TAG became "" instead of
"latest". That "" failed the emptiness check further down and exited 1 --
AFTER git pull + docker compose up -d had already succeeded. Worse: exiting
there skips the Health check step (Gitea Actions doesn't run later steps
after a failure), and Health check is the ONLY step that ever writes a real
value to .last_good_tag. Self-perpetuating: once the file went empty, every
future deploy failed the same way forever, with the app itself deploying
fine underneath it every time (confirmed against run 611: deploy log shows
all three images pulled/recreated/started at 66bbf5b, and /health
self-reports that exact GIT_SHA, baked into the image at build time --
two independent signals, same commit).

${VAR:-default} covers empty and unset in one expansion, matching the
fallback behavior the surrounding comment already documented as intended.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:36:17 -04:00
Logan CusanoandClaude Sonnet 5 66bbf5b473 intelligence: don't reject a geocode just because it is far from the node (#159)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 2s
Build & Deploy / Deploy to VM (push) Failing after 2m4s
Build & Deploy / Report a failed deploy (push) Successful in 1s
_geocode_location's node-distance fallback (geocode_max_km, 40km) is a
proxy for "is this plausible" that only makes sense when the node's own
position is the best guess we have at the area. It doesn't hold for a
citywide/patched feed: node-002 sits in Westchester but relays "New York
City - NYPD Citywide 2 Patch", ~56km from the addresses on it. Every real,
correctly-geocoded address on that talkgroup was rejected by this check,
every time -- location_coords stayed permanently null for the whole
system, which killed the location_proximity correlation signal and let
duplicate incidents form for the same event reported at two nearby
addresses two minutes apart ("Shots Fired at Jackson Avenue" /
"Shots Fired at 1108 Jackson Avenue", 2026-09-20 ~23:00 UTC, merged by
hand via the Archive page's attach/detach while this fix went in).

trust_named_region skips the node-distance rejection exactly when the
query already carries a place name that isn't the node's own position --
operator-set area_context, or a municipality parsed from the talkgroup's
own name. The anchor path is untouched; an anchor's own radius is always
authoritative when one has been resolved.

Also fixes a compounding defect found while verifying: the query was
grafting the node's own COUNTY onto an already-self-named region
("...New York City..., Westchester, New York"), which is self-contradictory
and could degrade the geocode independent of the distance check. The
node's county is now used only when nothing else names the place; state
stays in both branches since it's coarse enough to be correct either way.
Extracted the query-assembly logic into a pure, unit-tested helper
(_location_query_parts) rather than testing it only through the full
extraction pipeline.

Filed #160 as a follow-up: place_verifier.py's verify() has the identical
no-anchor gap for transcript place-name correction, not fixed here.

Verified: 422 pass, 0 fail (9 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:22:42 -04:00
Logan CusanoandClaude Sonnet 5 6c095083fc correlator: use srcaddr for thin-call disambiguation instead of pure recency (#158)
The fast/thin path (short acks like "10-4", enabled to attempt linking at
all by 1ffff25) had no identity signal available -- transcript_too_short
skips GPT extraction entirely, so call_units is always empty for this
population. It fell back to "most recently updated incident on this
talkgroup", which silently misattaches a short ack to the wrong incident
whenever two are live on the same busy dispatch channel at once.

metadata_watcher.py already captures the P25 source radio ID (srcaddr) on
every call independent of transcript content, and it already reaches the
call doc (models.py, mqtt_handler.py:245) -- it was just never read by the
correlator. Thread it through _build_context, check it against the
srcaddrs already seen on each TG-matched incident before falling back to
recency, and accumulate it on the incident (_update_incident/_create_incident)
so later calls from the same radio can match.

Verified: 422 pass, 0 fail (4 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 20:22:06 -04:00
Logan CusanoandClaude Sonnet 5 2e67d1bad6 ci: deploy Firestore rules/indexes from the runner, not the VM (server-26#51)
The rules/indexes deploy step already existed here, gated on `command -v
firebase` over SSH on the deploy VM. It never found one (no node on the
VM), so it silently warned-and-skipped on every single deploy for weeks —
PR #124 even auto-closed #13/#51 as if this were fixed, when it wasn't.

New standalone deploy-firestore-rules job runs on the Gitea runner itself
(always has node), authenticated via a new FIREBASE_TOKEN secret (from
`firebase login:ci`) instead of anything pre-installed on the VM. It's
independent of the deploy job's health-check/rollback chain on purpose —
a rules deploy failure has nothing to roll back and must not trigger that
logic. notify-failure now distinguishes which job actually failed so the
Discord alert doesn't misreport "production is unchanged" when the app
deployed fine and only the rules push failed.

Needs FIREBASE_TOKEN added as a Gitea Actions secret before this actually
runs — it will fail loudly (by design) until then, which is the whole
point: a loud failure beats a silent skip.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 19:08:58 -04:00
Logan CusanoandClaude Sonnet 5 1ffff25cd2 upload: let short transcripts (<=5 words) attempt correlation instead of never linking at all
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 2m21s
Build & Deploy / Report a failed deploy (push) Successful in 1s
intelligence.py skips GPT extraction for transcripts <=5 words ("10-8", "show
me clear", a unit check-in) -- real cost/hallucination guard, kept as-is. But
upload.py's no-scenes correlation fallback (the path that thin-links a call
by talkgroup even with zero extracted content) excluded ANY skip_reason,
including transcript_too_short -- so this exact population, brief but real
follow-up and clearance traffic, never even attempted to attach to anything.
Found live: "Live, Ossining." sitting an orphan 6 seconds before a real
incident's founding call, on the same talkgroup.

Now only garbage_transcript (Whisper hallucination, no real content) stays
excluded; transcript_too_short reaches the same thin/fast-path fallback
already trusted for no-transcript calls, gated the same way -- same-talkgroup,
recently-active incident required before anything attaches. No GPT re-invoked,
no new cost.

Verified: 410 pass, 0 fail.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:39:46 -04:00
Logan CusanoandClaude Sonnet 5 3d2b722c64 Wire AIS end to end: telemetry ingestion + live map overlay
node-26#9, same shape as the ADS-B commit. Adds POST /telemetry/ais
(same node-key auth, same org_id-stamped upsert-by-key pattern, this time
by mmsi into a new `vessels` collection) and its docInMyOrg() firestore
rule. Frontend gets useVessels() (mirrors useAircraft(), longer staleness
window since AIS position reports are minutes apart, not seconds) and an
opt-in "Vessels" map overlay.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:12:56 -04:00
Logan CusanoandClaude Sonnet 5 5537b095df Wire ADS-B end to end: telemetry ingestion + live map overlay
node-26#9. Adds POST /telemetry/adsb (node-key authed via
require_node_service_or_firebase_token) that upserts one Firestore doc per
icao into a new `aircraft` collection, org_id stamped from the reporting
node the same way upload.py defensively stamps `calls`. firestore.rules
gets a matching docInMyOrg()-gated read rule.

Frontend: useAircraft() mirrors useNodes()'s onSnapshot pattern, filtering
docs older than 2 minutes client-side since nothing prunes a stale aircraft
doc server-side yet. MapView gets an opt-in "Aircraft" overlay (unchecked
by default, like the weather radar layer) rendering a rotated plane glyph
per sighting.

Unverified via typecheck — no Node.js/npm on this authoring box yet (see
CLAUDE.md Testing reality). Server side is pytest-covered (test_telemetry.py).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 16:08:29 -04:00
Logan CusanoandClaude Sonnet 5 8892e824fc Add second-SDR fields: secondary_sdr_mode + sdr_count on NodeRecord
Server-side half of node-26#9. NodeRecord gains secondary_sdr_mode
(none|adsb|ais|op25_2) and sdr_count; checkin ingestion stores both,
and PATCH /nodes/{id} accepts and re-pushes secondary_sdr_mode the same
way hardware_preset/ppm_override already work, so it isn't wiped by a
system reassignment (see server-26#111 for the pre-existing bug that
pattern avoids repeating).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 15:56:33 -04:00
Logan CusanoandClaude Sonnet 5 3f69879437 intelligence: stop the police-channel default from typing content-free chatter (#138)
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 1m12s
Build & Deploy / Report a failed deploy (push) Successful in 1s
EXTRACTION_PROMPT's incident_type rule always returned "police" for anything
on a police channel unless contradicted, so pure administrative chatter
(post check-ins, roll call, bare acknowledgements) got a truthy incident_type
and defeated the creation gate's "type" veto (_call_is_substanceless) ~82%
of the time (measured server-26#138/CORRELATION_REVIEW_0914.md, confirmed
against live 9-14 data: 9/9 type-veto examples checked all had severity
"routine" — same population the severity rubric already correctly
identifies as content-free, incident_type just wasn't using that signal).

Rule now checks for actual event content FIRST, on every channel, before
applying the channel-inference defaults; content-free traffic returns
"unknown" (already normalizes to None) instead of a channel default,
letting the existing gate correctly veto it. No correlator/gate code
touched — CORRELATION_REVIEW_0914.md's own recommendation was to fix this
in the prompt, not _call_is_substanceless, to avoid risking a real event
getting gated out.

Verified: 401 pass, 0 fail. Live effect to be confirmed against fresh
traffic (AI features just re-enabled this session after being off since
9/14) — tracking in SESSION_STATE.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 15:42:48 -04:00
Logan CusanoandClaude Sonnet 5 fb0bb15c22 correlator/intelligence: close the incident-clearance gap (dispatch-to-10-8 lifecycle)
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Failing after 3m25s
Build & Deploy / Report a failed deploy (push) Successful in 1s
Two independent fixes, found by tracing why incidents never actually close
(only 2/281 incidents in 5 correlation debug windows ever got a non-empty
units_cleared; 183 resolved via the 90-min idle sweep instead of a real clear):

1. Pattern B (re-dispatch accept, no explicit 10-8): reassignment=True already
   fired correctly and suppressed the unit from re-linking to its prior
   incident, but nothing ever released the unit FROM that incident — it just
   sat "active" until the idle sweep timed it out. _release_reassigned_units
   now scans other active incidents for unit overlap on a reassignment and
   clears the unit there, reusing the same units_active/units_cleared merge
   (factored out as _apply_unit_clearance) that explicit 10-8 extraction uses.

2. Pattern A (self-clear) extraction was inconsistent for two reasons: no
   per-system unit ID format awareness anywhere in the pipeline (formats vary
   by department with zero shared convention), and the cleared_units prompt
   rule only accepted a unit self-reporting, missing dispatch confirming a
   unit's status back to them. Added system.unit_format_hint (owner-authored
   free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed
   into the extraction prompt, and broadened the cleared_units rule while
   still requiring an identifiable unit ID (guards against bare "10-8"/"clear"
   noise, including Whisper hallucination runs already caught upstream by
   _is_garbage_transcript).

Verified: 401 pass, 0 fail (local Linux venv ~/venvs/drb-5c — see CLAUDE.md
testing-reality note).

server-26#pending — not yet filed, Gitea unreachable from this sandbox.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 14:25:01 -04:00
logan a9197709f8 Merge pull request 'correlator: stop the re-correlation sweep racing an in-flight upload (#131)' (#154) from fix/131-sweep-race-duplicate-link into main
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 1m9s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-14 00:31:14 -04:00
Logan CusanoandClaude Sonnet 5 8dd636af8f correlator: stop the re-correlation sweep racing an in-flight upload (#131)
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.

Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.

15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.

recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.

Sandboxed pytest: 381 -> 387.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:30:24 -04:00
logan e27f8f6636 Merge pull request 'correlator/admin: capture per-scene incident_type + severity (#139)' (#152) from fix/139-scene-incident-type into main
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 1m40s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-14 00:14:53 -04:00
Logan CusanoandClaude Sonnet 5 f23026b9ab correlator: address drb-correlation-review notes on #139
- Document call_severity's routine-coercion asymmetry with incident_type
  (extraction-said-routine vs extraction-said-nothing look identical).
- Test docstring no longer overclaims the ctx-linkage it doesn't cover;
  points to the tests that do (test_consensus_gate.py, test_incident_identity.py).
- scene1 now uses a distinct severity so the test actually exercises both
  fields symmetrically; the 'no flat top-level clobber' claim is now
  asserted, not just commented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:14:32 -04:00
Logan CusanoandClaude Sonnet 5 7717fcccdd correlator/admin: capture per-scene incident_type + severity (#139)
_call_is_substanceless's "type" veto reads ctx["incident_type"] at decision
time, but that value was never persisted per-scene — only the last-scene-wins
flat field, which #138's window-4 dump analysis couldn't tell apart from
cross-scene contamination without re-guessing from a live dump. Adds
incident_type/severity to _apply_and_log's per-scene write and to admin.py's
_scene_summary allowlist (the debug-dump reader has its own field allowlist,
separate from the write side — silently would not have surfaced otherwise).

Sandboxed pytest: 380 -> 381.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-14 00:09:26 -04:00
logan 454fe7e81c Merge pull request 'correlator: remove is_dispatch and the tactical fit path entirely (#134)' (#135) from fix/134-drop-tactical-fit-path into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Failing after 1m56s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 14:55:04 -04:00
Logan CusanoandClaude Sonnet 5 422e9a4dc8 correlator: fix two comments left describing the removed tactical path (#134)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:54:54 -04:00
Logan CusanoandClaude Sonnet 5 b1884852d5 correlator: remove is_dispatch and the tactical fit path entirely (#134)
Full removal, not a hardcoded flag: _is_dispatch_channel, _DISPATCH_TG_RE, the is_dispatch parameter, and _call_fits_incident's tactical branch are gone. One evaluation path for every channel.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:49:55 -04:00
Logan CusanoandClaude Sonnet 5 0473e6a583 correlator: always run the dispatch-strict fit test, not name-guessed (#134)
is_dispatch was computed from _is_dispatch_channel(talkgroup_name) and picked between two _call_fits_incident evaluation orders: dispatch (requires a positive signal, runs location-conflict/content-divergence vetoes on unit overlap) vs tactical (skips both vetoes, defaults to True on no signal at all within 20 min). Per #133's reasoning, a name not literally containing dispatch/patched/primary got the unvetoed, default-True path solely because of its label. Hardcoded is_dispatch=True at its one real call site; the tactical branch and its own tests stay in place, unreached.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:38:57 -04:00
logan 4df801c5e0 Merge pull request 'correlator: drop the dispatch/tactical name-guess from the escape hatch (#115)' (#133) from fix/115-drop-dispatch-tactical-split into main
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 1m55s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 14:07:36 -04:00
Logan CusanoandClaude Sonnet 5 c50bfda8db correlator: drop the dispatch/tactical name-guess from the escape hatch, use one window (#115)
Owner correction from direct scanning experience: a talkgroup named tac/tactical only sees materially different traffic during a real incident, and that's rare -- the bulk of traffic on any monitored channel, including high-risk stops and pursuits, runs on the main channel regardless of what it's named. _is_dispatch_channel's string match on the talkgroup name is a naming-convention guess, not a detector of actual channel behavior; trusting it here meant a busy single-channel department not literally named 'dispatch' would silently get the more permissive 15-minute window and could reproduce #115's original bug (the gate never firing on the channels it targets).

Always use tg_dispatch_thin_idle_minutes (5 min) in the escape hatch, regardless of talkgroup name. Does NOT touch incident_correlator.py's own fast/thin idle-window selection, which uses the same dichotomy for a different, decision-changing purpose -- bigger blast radius, left for its own review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 14:07:00 -04:00
logan 833cfade4e Merge pull request 'correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix' (#132) from fix/96-114-per-scene-call-doc into main
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Failing after 1m28s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 13:34:19 -04:00
Logan CusanoandClaude Sonnet 5 0fe6d3b567 correlator: delete stale scenes on re-extraction instead of leaving them to rot (#96/#114)
Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident.

Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 13:33:52 -04:00
Logan CusanoandClaude Sonnet 5 fae84a45c3 correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix
Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).

Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
  - the existing flat corr_* fields, unchanged (last-scene-wins, the safe
    backward-compatible default for any reader that doesn't know about
    `scenes` yet)
  - a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
    corr_debug}, via doc_set(..., merge=True). Firestore's
    DocumentReference.set(data, merge=True) recursively merges nested map
    fields by key (documented SDK behaviour, not assumed) — a write to
    scenes.1 merges alongside an existing scenes.0 instead of replacing the
    whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.

admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.

summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.

Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.

Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.

Fixes server-26#96, server-26#114

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 13:25:37 -04:00
logan b7701b6d49 Merge pull request 'intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127)' (#128) from feat/115-chatter-classifier-shadow-mode into main
Build & Deploy / Build & push images (push) Successful in 4m2s
Build & Deploy / Deploy to VM (push) Failing after 1m48s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-13 12:43:18 -04:00
Logan CusanoandClaude Sonnet 5 3ae0bb2d5b intelligence: run the chatter classifier before the too-short skip, not after (#127)
82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 12:42:44 -04:00
logan 76db41adf7 Merge pull request 'ci: prune docker images before AND after deploy, not only on pull failure (#129)' (#130) from fix/129-deploy-disk-prune into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Failing after 1m12s
Build & Deploy / Report a failed deploy (push) Successful in 2s
2026-09-13 12:28:08 -04:00
Logan CusanoandClaude Sonnet 5 e97dab22ce ci: prune docker images before AND after deploy, not only on pull failure (#129)
The 2026-09-12 deploy of #126 failed instantly -- git pull on the VM hit 'No space left on device' before the deploy script could even capture a rollback target. Every deploy leaves 3 freshly SHA-tagged images that only got cleaned up by a prune gated on a failed compose pull; a failure earlier than that (like this one) never reached it. 96 of 100 local images were unreferenced, 23.76GB reclaimable, disk at 100%.

Move an unconditional docker image prune -af to the top of the deploy, before git pull, and upgrade the post-up -d prune from -f (dangling only) to -af (all unused) so stale tagged images stop re-accumulating. Both are safe: prune -a never touches an image a running container references.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-13 12:27:31 -04:00
Logan CusanoandClaude Sonnet 5 05ddec8284 intelligence: shadow-mode upstream dispatch-vs-chatter classifier (server-26#127)
Three live measurement windows and two consensus-layer fixes (#125, #126)
converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop
iterating the correlator's consensus layer, the actual lever is upstream —
a classifier in scene extraction that recognizes radio housekeeping (roll
call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever
becomes a scene for the correlator to judge.

Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript)
function recognizing two shapes drawn from hand-labeled examples in the
review docs, cross-referenced against the real dumps — not invented regexes.
Deliberately conservative: anything that doesn't cleanly reduce to a known
shape returns (False, None) and the existing pipeline runs unchanged.

SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to
the existing _is_garbage_transcript / transcript_too_short gates and writes
chatter_classifier_verdict / chatter_classifier_reason onto the call doc,
but does not skip extraction. admin.py's correlation-debug _call_summary
surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the
next live window can measure the real-world false-positive rate before
anything is wired to actually skip extraction. TODO(server-26#127) marks
the call site.

Backtest against all three existing dumps (1002 calls): 154 flagged, 0
false positives (no flagged call carries tags, coords, non-routine severity,
or matches any review-doc-named dangerous-to-drop transcript — the major
extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station
subject check, the property-retrieval call, all individually verified).

tests/test_chatter_classifier.py: real transcripts from the dumps/review
docs in both directions. Sandboxed pytest 332 -> 364, green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 23:59:53 -04:00
logan 11c98daed0 Merge pull request 'correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)' (#126) from fix/115-escape-hatch-window into main
Build & Deploy / Build & push images (push) Successful in 4m17s
Build & Deploy / Deploy to VM (push) Failing after 1m24s
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-12 04:47:49 -04:00
Logan CusanoandClaude Sonnet 5 83beb2bf35 admin: surface corr_gate_veto on the correlation-debug endpoint (#115)
corr_gate_veto was written to corr_debug but the admin endpoint's whitelist (_call_summary + the summary tally) never surfaced it, so the last commit's whole point -- measuring window #4 instead of guessing -- would have produced nothing to read. Add it to both. Also softened the docstring's remaining overclaim: whether the active-only ctx[recent] limitation explains the 2/24 window-3 misses is unanswered, not confirmed -- read corr_gate_veto next window instead of asserting a guess again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:47:15 -04:00
Logan CusanoandClaude Sonnet 5 598054746a correlator: mirror the dispatch/tactical idle split in the escape hatch, record the gate-veto reason (#115)
Review of #126 found: (1) the escape hatch applied tg_dispatch_thin_idle_minutes (5 min) unconditionally, but incident_correlator's own fast/thin path only uses that on dispatch channels and 15 min on tactical ones via _is_dispatch_channel -- mirrored the same selection here, plus a config.py note flagging the second consumer. (2) the docstring claimed a 'confirmed explanation' for 2 window-3 gate misses that was actually wrong (self-contradictory in its own text); replaced the guess with corr_gate_veto, written into corr_debug on every llm=orphan/rules=new disagreement that escalates, so window #4 can see *why* each one escaped instead of reconstructing it from the raw dump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:42:14 -04:00
Logan CusanoandClaude Sonnet 5 400b74b519 correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)
_recent_incident_on_same_talkgroup previously treated ANY same-talkgroup
incident within the 2-hour correlation_window_hours lookback as 'recent',
which disabled the whole LLM-orphan consensus gate on busy dispatch
channels: window #3 (CORRELATION_REVIEW_0912.md) measured 0/24 gate fires
against the exact target shape (rules=new, llm=orphan, tiebreak=new), with
22/24 explained by a same-talkgroup incident existing somewhere in the
prior 2h — nearly guaranteed on channels producing 3-13 incidents/2h.

Now the escape hatch only counts an incident as recent within
settings.tg_dispatch_thin_idle_minutes (5 min), reusing the same recency
bound the fast/thin path already uses for the 'dispatch, thin ack 10-30s
later' case this hatch exists for, instead of inventing a new constant.

Investigated the 2 unexplained misses (no same-tg incident found even by
a naive full-collection timestamp scan): confirmed ctx["recent"] is built
from status=="active" incidents with over-capacity incidents dropped
(_build_context / _drop_capped), not a full collection scan — an incident
that has auto-resolved or hit incident_max_calls/incident_max_duration
within the window is invisible to this check even though it is
chronologically recent. This does not explain the 2 misses (a same-tg
incident was absent by both checks there, so some other
_call_is_substanceless condition must be responsible), but it is a real
gap in the check as written. Documented in the docstring with a
TODO(server-26#115); fixing it needs a new, non-active-filtered Firestore
query, out of scope for this pass.

Tests: added a regression test proving an incident inside the old 2h
window but outside the new 5-minute window now correctly gates (fails on
main, passes here), plus a test proving a truly recent (<5min) same-tg
incident still escapes the gate as intended. Sandboxed pytest: 327 -> 329
passed (2 new tests), all green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-12 04:28:07 -04:00
logan 07ff9ba193 Merge pull request 'correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)' (#125) from fix/115-consensus-orphan-gate into main
Build & Deploy / Build & push images (push) Successful in 4m11s
Build & Deploy / Deploy to VM (push) Successful in 2m11s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-11 23:18:12 -04:00
Logan CusanoandClaude Sonnet 5 15a9d10666 correlator: keep the tiebreak for typed / reassignment calls in the orphan gate (#115)
_call_is_substanceless mirrored has_event_substance but not the creation gate's type-resolved short-circuit, so a routine-severity fire/medical call with no coords/tags/vehicles — or a reassignment (unit pulled to a new job) — could be gated to orphan where rules would open an incident. Bail out of the gate on incident_type or reassignment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:57:20 -04:00
Logan CusanoandClaude Sonnet 5 dd426572fc correlator: fix consensus orphan-gate to test call substance, not empty corr_debug (#115)
The gate added in ca1d8fb checked rules_decision["corr_debug"] for a positive
signal, but that dict is empty at preview time for action=="new" (corr_path is
written at apply time). The check was always False, so the gate fired on real
events — replayed against corr_dump_9-7_pm.json it dropped ~36 linked calls
including a major "extinguishing fire", a moderate fire-alarm, geocoded calls
and pursuit updates.

Gate now runs against ctx (fully populated at preview time). It fires ONLY when
the call is substanceless: routine severity, no vehicle/geocode/tag, and no
incident already running on the same talkgroup. Any of those escalates to the
tiebreak instead. The substance predicate (has_event_substance) is factored out
of incident_correlator's creation gate and shared, so the two cannot diverge.

recorrelation_sweep: a call the gate parked gets a longer link-only retry budget
(10 vs 3) — the gate fires before any incident for the job exists, so the
substantive call that justifies linking can land after the standard ~6 min.
Still create_if_new=False.

incident_correlator location path: evaluate every in-radius candidate and link
the nearest that carries corroboration, instead of the first in an unsorted
`recent`. A unit-overlap location link is now tagged "location_unit_overlap" so
it stops merging into the fast path's bucket in the admin fit-signal histogram.

tests/test_consensus_gate.py: replaced the corr_debug-signal cases with ctx
substance cases (severity, coords, tags, vehicles, same-tg incident); added a
nearest-wins location test; the two location guard tests now assert they reach
the new guard. Full drb-c2-core suite 322 -> 325.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:50:41 -04:00
Logan CusanoandClaude Sonnet 5 ca1d8fbdae correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)
CORRELATION_REVIEW_0907b.md measured that radio housekeeping (unit
check-ins, roll call, 10-8/10-98 clearings) is being promoted to
incidents. Every case reads corr_llm_action=orphan, corr_rules_action=new,
corr_consensus=tiebreak -> new: the cheap LLM correctly reads "not an
incident", the rules engine says `new` only because there is no incident to
link to, and the smart tiebreaker then sides with rules ~21/21. Reframing
the tiebreaker prompt (#116) did nothing. The fix is a consensus-logic gate,
not another prompt.

Fix 1 (routers/upload.py) - LLM-orphan gate in _correlate_with_consensus:
when the cheap LLM says `orphan` and the rules engine says `new` with NO
positive event signal, resolve to `orphan` and skip the tiebreak call
entirely. "No positive signal" = the rules corr_debug carries neither a
positive corr_path (unit-continuity / location / fast/disambig / fast/single)
nor a positive corr_fit_signal (unit_overlap / location_proximity). When it
does carry one, the existing escalation-to-tiebreak is kept so a genuine
event the LLM misreads as orphan still gets the second look. The resolved
outcome records corr_consensus="llm_orphan_gate" (greppable, distinct from
"tiebreak") and keeps corr_llm_reasoning / corr_rules_action /
corr_llm_action populated.

Fix 2 (incident_correlator.py) - tighten corr_path=location: the location
path linked on a bare sub-location_proximity_km (0.5 km) distance with no
unit or content check, which stitched a vehicle lockout to a station-restroom
slip and merged two different churches an hour apart. A location link now
requires unit overlap with the candidate OR a distance under a tighter bar
(_LOCATION_TIGHT_PROXIMITY_KM = 0.2 km). Pursuit incidents keep their
movement-speed-validated wide radius. A surviving location link now also
writes corr_fit_signal (unit_overlap | location_proximity), consistent with
Fix 1's positive-signal set.

Tests: new tests/test_consensus_gate.py (13 cases) - the gate resolves to
orphan without calling tiebreak on a no-signal disagreement; a unit_overlap /
location_proximity / unit-continuity / fast-disambig rules signal still
escalates; llm=link vs rules=new still escalates; the location path drops a
shared-area candidate with neither unit overlap nor tight proximity, links on
unit overlap, and links on tight proximity alone. Full c2-core suite
309 -> 322 passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:32:52 -04:00
logan 7f4d684966 Merge pull request 'ci: deploy Firestore rules + indexes on every push to main (#51)' (#124) from fix/51-ci-firestore-deploy into main
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 3m49s
Build & Deploy / Report a failed deploy (push) Skipped
Reviewed-on: #124
2026-09-07 23:22:38 -04:00
Logan CusanoandClaude Sonnet 5 bd04bdbd69 firestore: declare DESC indexes for the orderBy(desc) queries (#33/#51)
The old //direction note ('ASC serves orderBy desc') was wrong for these query shapes and left useCalls/useIncidents/useAlerts and search_calls throwing FAILED_PRECONDITION. Declare calls/incidents/alert_events (…, DESC) to match the live DB (indexes created via gcloud 2026-09-08). Drop the misleading 'delete these duplicates' drift note.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 23:20:48 -04:00
Logan CusanoandClaude Sonnet 5 775244bbde ci: deploy Firestore rules + indexes on every push to main (#51)
The deploy job SSHes to the VM (which runs as the project service account) but never touched Firestore, so rules and composite indexes regressed silently after every fix. Add a firebase-tools deploy right after `git pull`, additive for indexes, warn-not-fail on error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 19:44:57 -04:00
logan bc3251e8df Merge pull request 'frontend: #109 punch-list P2 — RulesTab effect, node recent-calls window' (#123) from fix/109-punchlist-p2 into main
Build & Deploy / Build & push images (push) Successful in 5m45s
Build & Deploy / Deploy to VM (push) Successful in 2m23s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 19:06:34 -04:00
logan 7a5bd5dbbb Merge pull request 'map: remove stray clock, unstack incident rail, OSM tile fallback (#118)' (#122) from fix/118-map-overlays into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m31s
2026-09-07 19:06:30 -04:00
logan 629bd1c340 Merge pull request 'firestore: declare the alert_events composite index (#51)' (#121) from fix/51-alert-events-index into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m28s
2026-09-07 19:06:28 -04:00
logan cea094d66b Merge pull request 'c2-core: fix CORS so the browser can call the REST API (#110)' (#120) from fix/110-c2-core-cors into main
Build & Deploy / Deploy to VM (push) Canceled after 0s
Build & Deploy / Report a failed deploy (push) Canceled after 0s
Build & Deploy / Build & push images (push) Canceled after 1m31s
2026-09-07 19:06:25 -04:00
logan 01c146e21e Merge pull request 'ci: bake NEXT_PUBLIC_MAP_TILE_URL into the frontend build (#117)' (#119) from fix/117-map-tile-build-arg into main
Build & Deploy / Build & push images (push) Failing after 14s
Build & Deploy / Deploy to VM (push) Skipped
Build & Deploy / Report a failed deploy (push) Successful in 1s
2026-09-07 19:06:21 -04:00
Logan CusanoandClaude Sonnet 5 8a0412b529 firestore: add the alert_events composite index (#51)
collectionGroup alert_events (acknowledged, org_id, triggered_at desc) — the index the /watch Triggered Alerts tab and the site-wide useUnacknowledgedAlerts hook require. Still needs a manual deploy; no automation exists (#51).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:54:10 -04:00
Logan CusanoandClaude Sonnet 5 52edbf105c map: remove stray clock, unstack the incident card from the zoom controls, OSM tile fallback (#118)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:53:14 -04:00
Logan CusanoandClaude Sonnet 5 77f1d2f93f frontend: #109 punch-list P2 — effect-guard RulesTab, pending node card modal, org save, node recent-calls filter
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:52:39 -04:00
Logan CusanoandClaude Sonnet 5 d60fef67ad c2-core: add CORS middleware so the browser can call the REST API (#110)
The Archive page's GET /calls/search failed its CORS preflight (OPTIONS -> 405, no Access-Control-* headers). Allow the app origin(s) explicitly for the standard methods and the authorization/content-type headers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:52:38 -04:00
Logan CusanoandClaude Sonnet 5 fe643924c7 ci: bake NEXT_PUBLIC_MAP_TILE_URL into the frontend build (#117)
The map override var was added to MapView.tsx but never passed as a build-arg, so prod still shipped the dead Carto tile URL. Point it at OSM raster tiles.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
2026-09-07 18:48:42 -04:00
logan bccb3e0316 correlator: give the LLM tier what it needs to link, stop it defaulting to "new" (#116)
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 2m25s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 16:59:46 -04:00
Logan CusanoandClaude Sonnet 5 1a631d65d0 correlator: address #116 review — call talkgroup id in the prompt, sort candidates
drb-correlation-review: ship, with two bounds the low-bar link rule needs.

1. _call_block emitted only the talkgroup NAME while _inc_summary emits
   numeric tg ids, so the "same talkgroup" precondition in _RULES was
   unevaluable and the low link bar applied unconditionally. _call_block now
   prints "Talkgroup: <name> (id <n>)".
2. ctx["recent"] is an unordered Firestore slice with no order_by; a busy 2h
   window (~40 active incidents) showed the model an arbitrary half of the
   candidates. _prompt_incidents() sorts by updated_at desc before the [:20]
   cap — also makes each row's idle: field monotonic.

+2 tests. Full c2-core suite green (sandboxed venv).

Review follow-ups (not blockers): _parse_response demotes an unresolvable
link to orphan (drops the call) rather than falling back to rules — now on
rising link volume; the 45% tiebreak escalation rate / smart-model cost is
untouched; _ROAD_RE swallows leading tokens so "10 Parker Street" still
won't road-overlap "Parker St".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 16:59:29 -04:00
Logan CusanoandClaude Sonnet 5 3a944f35c1 correlator: give the LLM tier what it needs to link, stop it defaulting to "new" (server-26#115)
The 2026-09-07 measurement window (CORRELATION_REVIEW_0907.md) showed the
consensus tiebreaker was the dominant over-split driver: it ran on 45% of
calls and resolved link/orphan disagreements as "new" ~24/25 of the time,
shattering one Mohegan Park car-alarm job into 9 incidents and opening ~7
incidents from radio checks / roll calls.

Two causes, two fixes:

1. `_inc_summary` gave the model `id|type|loc|units|tags|idle` — no title,
   no talkgroup. It literally could not see that two "car alarms, Mohegan
   Park Ave/Avenue" incidents on TG 9560 were the same. Now includes the
   incident title (the strongest same-event signal) and talkgroup.

2. `_RULES` told the model "orphan when in doubt — conservative is always
   correct". For a system that over-splits, that is backwards: a wrong link
   is cheap, a duplicate incident is the failure. Rewritten to: prefer link
   for a plausible same-talkgroup continuation (low bar), reserve "new" for a
   genuinely different event, and explicitly "orphan" non-incidents (radio
   checks, roll call, 10-8/10-98, mileage logs).

Plus `_extract_road_ids` now canonicalises street-type synonyms
(Avenue→ave, Street→st, Road→rd, ...), so "Mohegan Park Avenue" and
"Mohegan Park Ave" share a road id — that one difference was splitting the
car-alarm incident.

+tests/test_correlator_115.py. Full c2-core suite green (sandboxed venv).
Bigger levers deferred to follow-ups: the consensus escalation itself (should
a cheap-LLM "orphan" ever reach a tiebreak?), a first-class road-overlap fit
signal in _call_fits_incident, geocode coverage.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 16:53:03 -04:00
logan 0712e7a437 correlator: LLM tier reads the scene transcript, not the whole call (#112)
Build & Deploy / Build & push images (push) Successful in 4m1s
Build & Deploy / Deploy to VM (push) Successful in 2m2s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 04:40:34 -04:00
logan a739fa64f0 frontend: safe fixes from the #109 punch-list (#113)
Build & Deploy / Build & push images (push) Successful in 4m5s
Build & Deploy / Deploy to VM (push) Successful in 2m33s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-07 00:13:35 -04:00
Logan CusanoandClaude Sonnet 5 7189ba03e4 correlator: address #102 review — 0-based segment labels, never-empty slice
drb-correlation-review on the prior commit flagged two ways the per-scene
transcript could silently fall back to the whole-call text:

1. _build_transcript_block numbered transmissions "1." while the prompt says
   "0-based indices" — a model echoing the labels it saw returned 1-based
   indices, shifting every scene's slice by one. Labels are now "0." to match
   the documented contract (also fixes the same latent skew in
   _build_scene_embed_text / #80).
2. An empty join (bad / out-of-range / non-int indices) hit
   `transcript or call_doc.get(...)` in _build_context and fell back to the
   whole-call transcript — re-opening the leak exactly when indices are wrong.
   The slice now falls back to this call's own whole transcript *before*
   _build_context sees it, so it is never "". Non-int and negative indices
   are rejected rather than raising.

Slice logic extracted to `_scene_transcript_text` with a dedicated test file
(4 cases: subset, corrected-wins, no-indices fallback, bad-indices fallback).
Call-doc fallback kept (sweep / no-scene path) per the review. Also restored
the `-> ` spacing lost in the prior commit's kwarg edit.

Full c2-core suite green: 300 passed (sandboxed venv). Still DO NOT MERGE
until the measurement window closes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 00:12:56 -04:00
Logan CusanoandClaude Sonnet 5 d67b2057e6 frontend: safe fixes from the #109 punch-list
- CallSpineEntry.tsx: drop the dead `hasAudio` prop + the early `return null`
  that sat between hooks in InlinePlayer (React #310 risk). Parent already
  gates the mount on audio presence.
- NodeCard.tsx + nodes/page.tsx: pending-node card no longer double-fires.
  NodeCard gains `linkToDetail` (default true); the pending branch passes
  false so the wrapping onClick (open config modal) isn't swallowed by the
  inner <Link> navigation. List view unchanged.
- trips/page.tsx: TripCard badge now buckets on end_date >= today, matching
  the list's own upcoming/past split — an in-progress trip no longer shows a
  "Past" badge under "Upcoming".
- trips/page.tsx, NodeConfigModal.tsx, nodes/[id]/page.tsx: tall modals get
  `p-4` on the overlay + `max-h-[90vh] overflow-y-auto` on the panel so they
  don't clip on short viewports (incidents' CreateModal pattern).
- lib/types.ts: IncidentRecord.units / vehicles are optional now, matching
  Firestore (older docs omit them); incidents/[id] gains a `?? []` guard.

Untypechecked (no node/npm locally). next build in deploy.yml gates it.
Full list of remaining items in server-26 #109.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 00:07:54 -04:00
Logan CusanoandClaude Sonnet 5 ef1e3d7f9d correlator: LLM tier reads the scene's transcript, not the whole call (server-26#102)
The last leg of the #80/#95 scene-context leak. llm_correlator._call_block
read call_doc's whole-call transcript for every scene, so on a multi-scene
call every scene's cheap-tier and tiebreaker decision was made against text
that also contained the other scenes.

- intelligence.py: each processed[] scene now carries its own "transcript" —
  transcript_corrected, else this scene's segments joined, else (single scene)
  the whole transcript.
- _build_context / preview_correlation / correlate_call: take a `transcript`
  param; _build_context resolves ctx["scene_transcript"] from it, falling
  back to the call doc (sweep, single-scene, tests) — the fallback is kept
  here, unlike embedding/severity, because a scene always has real text.
- upload.py: both scene loops pass scene["transcript"].
- llm_correlator._call_block: reads ctx["scene_transcript"] (call-doc
  fallback retained for test-built ctx).
- recorrelation_sweep: passes the call doc's text explicitly.
- +1 regression test. Full c2-core suite green (296 passed, sandboxed venv).

NOT for merge until the running correlation measurement window closes and its
dump is analysed — deploying a correlator change mid-window would mix old and
new behaviour in the sample.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-07 00:06:37 -04:00
logan c1c3e89e1d frontend: fix map stacking + honest infra error states (#108)
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 2m3s
Build & Deploy / Report a failed deploy (push) Skipped
2026-09-06 23:49:19 -04:00
Logan CusanoandClaude Sonnet 5 968134f8ee frontend: fix map stacking + honest infra error states
From a live review of drb.cusano.net.

MapView.tsx / globals.css:
- The Leaflet map painted above the sticky Nav (z-40) and modal overlays, so
  on Live the account dropdown opened *behind* the map. Pin .leaflet-container
  to its own stacking context (position:relative; z-index:0) — keeps Leaflet's
  internal pane order, drops the whole map below app chrome. The map's own
  overlay UI (legend, rail, clock, fit-all) is outside .leaflet-container and
  unaffected. Chosen over raising Nav's z-index, which would float the sticky
  header over modal backdrops on ~7 pages.
- Basemap: the "Dark" tile URL is already CARTO's keyless dark raster (so a
  prod "API KEY REQUIRED" watermark is a stale build or CARTO rate-limiting
  the origin, not this code). Add NEXT_PUBLIC_MAP_TILE_URL as a build-time
  override so a keyed style drops in without a code change; add the OSM
  attribution the keyless CARTO tiles require.

incidents/page.tsx, alerts/page.tsx:
- Both dumped raw Firestore "requires an index / PERMISSION_DENIED" strings
  (with a console.firebase URL) straight into the UI when the composite
  indexes aren't deployed (server-26 #13/#51). Collapse those known infra
  failures to a plain sentence; any other error passes through verbatim so a
  real bug still shows. alerts also now surfaces the events-query error at
  all — it was swallowed, showing a false "No alerts triggered yet." on a
  public-safety screen.

onboarding/page.tsx: stale comment (/dashboard -> "/").

Untypechecked (no node/npm locally); presentational only — one string
helper, one added error branch, a CSS rule, two tile-URL constants, a
comment. next build in deploy.yml gates it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-06 23:43:22 -04:00
logan b430cf32f2 Merge pull request 'frontend: install command uses the node id from the mint form (node-26#4)' (#107) from feat/mint-panel-nodeid into main
Build & Deploy / Build & push images (push) Successful in 5m51s
Build & Deploy / Deploy to VM (push) Successful in 1m42s
Build & Deploy / Report a failed deploy (push) Skipped
Reviewed-on: #107
2026-09-06 20:15:10 -04:00
Logan CusanoandClaude Sonnet 5 93fa3a6054 frontend: install command uses the node id from the mint form (node-26#4)
The mint panel's copy command hard-coded --node-id node-XXX. Now the label
just entered (the operator types the node id there — placeholder relabeled
"Node ID, e.g. node-003") is captured on mint and interpolated into the
command: spaces → dashes, non [A-Za-z0-9_-] stripped (install.sh's rule),
falling back to node-XXX only if that yields nothing. The "edit node-XXX"
hint now only shows in the fallback case.

Not typechecked (no node/npm here); one useState<string|null>, one derived
string, a JSX conditional. `next build` in deploy.yml gates it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-06 20:13:23 -04:00
logan de03f5bcaf Merge pull request 'frontend: mint panel shows the full one-shot install command (node-26#4)' (#106) from feat/mint-panel-install-command into main
Build & Deploy / Build & push images (push) Successful in 5m6s
Build & Deploy / Deploy to VM (push) Successful in 1m40s
Build & Deploy / Report a failed deploy (push) Skipped
Reviewed-on: #106
2026-09-06 19:31:34 -04:00
Logan CusanoandClaude Sonnet 5 0651bfe07a frontend: mint panel shows the full one-shot install command (node-26#4)
After a node enrollment token is minted, the panel now renders the
paste-ready `curl -fsSL .../install.sh | sudo bash -s -- --token <minted>
--node-id node-XXX --c2-url <derived> --mqtt-broker <derived>` line with a
Copy button, alongside the bare token (also kept, also now copyable).

- c2-url from NEXT_PUBLIC_C2_URL (same var lib/c2api.ts reads), fallback
  https://api.example.net
- mqtt-broker derived as mqtt.<api-host minus leading api.> — a DNS
  assumption; the panel text tells the operator to check it
- node id is a node-XXX placeholder; the panel collects none

Pairs with node-26's install.sh (feat/one-shot-install). The raw/tag/v1/
URL resolves once v1 is re-cut at that PR's merge.

NOT typechecked here (no node/npm in this environment); plain React, two
useState booleans + one computed string, reviewed by eye. `next build` in
the deploy workflow will catch a real type error before it ships.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-06 19:15:55 -04:00
logan c4656a9607 Merge pull request 'correlator: judge each scene on its own embedding + severity (#80, #95)' (#105) from fix/scene-context-leak-80-95 into main
Build & Deploy / Build & push images (push) Successful in 5m1s
Build & Deploy / Deploy to VM (push) Successful in 2m8s
Build & Deploy / Report a failed deploy (push) Skipped
Reviewed-on: #105
2026-09-06 17:49:23 -04:00
Logan CusanoandClaude Sonnet 5 a9d1d2475a correlator: judge each scene on its own embedding + severity (server-26#80, #95)
intelligence.py writes only the primary scene's embedding and severity to
calls/{id}. _build_context read them back off the call doc, so every
non-primary scene of a multi-scene call was correlated against scene 1's
semantic vector and severity rung: a scene about a different event scored
on the embedding path against the wrong incident, and could inherit a
minor/moderate/major severity it never had, clearing the creation gate on
borrowed weight. Same defect and same fix as the #87 coords leak.

- _build_context / preview_correlation / correlate_call: take embedding and
  severity as params; drop the call_doc.get() fallbacks. A scene that
  passes none has none, and is judged thin on its own signal.
- upload.py: both scene loops pass scene["embedding"] / scene["severity"];
  _correlate_with_consensus forwards them. The no-scene unclassified branch
  passes neither (correct: no scene, judged thin).
- recorrelation_sweep: passes the call doc's stored values explicitly
  (whole-call re-link, link-only, so a borrowed severity cannot create).
- intelligence.py: SCENE DETECTION prompt tightened toward one scene
  (server-26#5, partial) - MULTIPLE only for genuinely separate events,
  "when unsure, one scene", plus a not-a-new-scene list.
- test_incident_identity.py: +2 regression tests mirroring the #87 test.

Full c2-core suite green (295 passed). #5 prompt change is unmeasured -
needs a scoped correlation-only window. Known remaining legs, tracked
separately: llm_correlator._call_block still reads the whole-call
transcript per scene; content-divergence veto skips on a None embedding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-06 13:01:59 -04:00
Logan CusanoandClaude Opus 5 85393bdb26 ci: retrigger deploy after registry token expired mid-build
Build & Deploy / Build & push images (push) Successful in 6m20s
Build & Deploy / Deploy to VM (push) Successful in 1m59s
Build & Deploy / Report a failed deploy (push) Skipped
Run 570 (8b6c170) pushed c2-core successfully, then failed on
discord-bot with "failed to authorize: failed to fetch oauth token:
unauthorized" ~20s later, using the same credential. That is a
short-lived registry token expiring mid-run, not an invalid one.

Build job failure skipped "Deploy to VM", so 8b6c170 -- which closes
the viewer-triggerable OpenAI spend on incident summarize
(server-26#81) -- never reached production. Prod stayed on b722223
with the spend leak open.

No code change. This commit exists only to re-run the pipeline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 02:49:05 -04:00
Logan CusanoandClaude Opus 5 8b6c170265 Close viewer-triggerable OpenAI spend on incident summarize (server-26#81)
Build & Deploy / Build & push images (push) Failing after 1m58s
Build & Deploy / Deploy to VM (push) Skipped
Build & Deploy / Report a failed deploy (push) Successful in 2s
POST /incidents/{id}/summarize was gated by require_service_or_firebase_token,
which accepts any authenticated Firebase user including role "viewer". That
route spends OpenAI credits via the background summarizer. The call-side
equivalent was already moved to require_admin_token; this brings the incident
side in line with it.

The frontend's two "summarize now" buttons on the incident detail page are
already gated behind isAdmin, so this backend change matches existing UI
behavior exactly and does not break any viewer/operator surface — it only
closes direct-API access for non-admins.

Swept every other route in incidents.py: list/get are reads with no spend and
correctly stay open to any signed-in user; create/update/delete/link/unlink
were already require_admin_token. No other sibling route needed changing.

Adds test_incident_summarize_auth.py pinning the dependency wiring directly
(the convention used in test_admin_feature_flags.py), so a future revert back
to the weak dependency fails a test immediately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 02:45:08 -04:00
Logan CusanoandClaude Opus 5 b7222230bd frontend: label machine-generated output and unbuilt entitlements (Gate A)
Build & Deploy / Build & push images (push) Successful in 4m25s
Build & Deploy / Deploy to VM (push) Successful in 1m55s
Build & Deploy / Report a failed deploy (push) Skipped
Gate A (BUSINESS_MODEL.md, board minutes #42, dated to today by minutes #79
decision 14) blocks putting a price or an unbuilt entitlement claim on a
surface a reader can see, and requires that unverified machine assertions be
labelled as such on the same screen as the assertion.

The pricing leg was already met — /pricing and both homepage CTAs stopped
quoting the invented catalog. Condition A2 was not: a search of the whole
frontend for a "machine-generated" or "unverified" qualifier returned zero
hits. Every transcript, summary, title, location, unit list and vehicle list
is pipeline output that no human reviews, and entity-name accuracy in those
transcripts has never been measured (server-26#48) — yet all of it was
rendered to the reader as plain fact. Unqualified machine assertions about
real incidents and real people is the exposure Gate A exists to stop.

A2 — one reusable element, components/ui/MachineOutputNotice.tsx, rendered on
the same screen as the output (a footnote elsewhere does not satisfy A1's
"same screen" standard). Three variants for three shapes of surface, all
saying the same thing; the "popup" variant uses fixed grays because a Leaflet
popup is stock-white in both themes. Covered:

  - incident detail: under the summary (covers summary, title, location,
    units on scene/cleared, vehicles, tags) and above the call spine
  - incident list: above the timeline groups
  - Archive (/calls): above the transcript rows
  - node detail: above the Recent Calls table
  - Watch//alerts: above the events table, whose Snippet column is transcript
    text and whose keyword match was made against it
  - Live map: the desktop incident rail, pinned above the scroll area so it
    cannot be scrolled off the screen it qualifies; the mobile drawer; the
    incident marker popup; the incident-path stop popup
  - /systems: the source-call transcript preview
  - /features: the two marketing sections that describe the AI pipeline

A1 — components/ui/UnbuiltMarker.tsx marks a claim unbuilt inline:

  - /faq: the retention answer promised 7/90/365-day windows. There is no TTL
    and no deletion sweep anywhere in the product (server-26#44), so the
    answer now states plainly that nothing is deleted automatically and marks
    per-plan retention as not yet available.
  - /settings/billing: the plan cards' claims — custom retention, SSO/SAML,
    uptime SLA, data residency — are marked not-yet-available next to the plan
    that makes them.

Labelling only. No retention, SSO, SLA or residency was built; no billing,
Stripe or checkout code was touched (Gate B still bars charging anyone); no
price was added anywhere; no Python was touched. Both themes verified against
the light-mode !important overrides in globals.css, which are untouched.

tsc --noEmit clean.

Refs: server-26#46, server-26#44, server-26#48

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-31 02:47:40 -04:00
Logan Cusano bdb57ae75a correlator: stop non-primary scenes inheriting the call doc's pin (#87)
Build & Deploy / Build & push images (push) Successful in 4m30s
Build & Deploy / Deploy to VM (push) Successful in 1m47s
Build & Deploy / Report a failed deploy (push) Skipped
_build_context fell back to call_doc.get("location_coords") whenever a
scene passed no coordinates of its own. One radio call can be split
into several scenes, but only the primary scene's geocode is ever
written to the call doc — so every non-primary scene silently
inherited the primary scene's pin. That fabricated location_proximity,
the strongest accept signal the correlator has, for a scene that had
no location at all, and drove it into the primary scene's incident on
a pin it never had.

Drop the fallback: coords = location_coords. A scene with no location
is now correctly judged thin, cannot win the location path, cannot
supply call_coords to _call_fits_incident, and cannot seed
_find_cross_system_parent.

recorrelation_sweep.py, the only other caller of correlate_call, was
verified to already pass both location and location_coords explicitly
from the call doc, so the fallback there was a no-op and this change
is behavior-preserving for that path.

Adds test_a_scene_with_no_location_does_not_inherit_the_call_docs_pin
to test_incident_identity.py, pinning ctx["coords"] is None and
ctx["is_thin_call"] is True when location=None but the call doc
carries a location_coords.

Ref: server-26#87
2026-08-31 02:45:31 -04:00
Logan CusanoandClaude Opus 5 29c2fb11b9 Ignore drb-telegram-bot/ — out of scope, not a deployed service
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Successful in 1m55s
Build & Deploy / Report a failed deploy (push) Skipped
Scaffolding for a service that does not run and is not in compose. It has sat
untracked across four unattended runs, each of which had to decide again
whether to commit or delete someone else's work. Declaring it out of scope
ends that. server-26#56.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-30 23:28:29 -04:00
Logan CusanoandClaude Opus 5 865b5b4317 Close the /admin/features side-door that needed a container shell to flip AI spend
Build & Deploy / Build & push images (push) Successful in 4m15s
Build & Deploy / Deploy to VM (push) Successful in 1m56s
Build & Deploy / Report a failed deploy (push) Skipped
Board minutes #62 Decision 2 (server-26#64), due 2026-08-31. CTO draft #60
finding 1 and CISO draft #61 finding 3 reached this independently.

GET/PUT /admin/features accepted only a Firebase admin token, so the unattended
runbook had no headless path and SSHed into the c2-core container to write
config/ai_features with the admin SDK. Moving a platform-wide AI cost switch
required a full container shell, and set_flags() wrote no audit entry either
way, so a flag flip was unattributable however it happened.

- New agent_service_key (AGENT_SERVICE_KEY), deliberately separate from the
  Discord bot's service_key. Sharing one key would collapse two principals into
  a single unattributable identity in every log line, and the bot has no
  business flipping AI flags regardless.
- require_agent_key_or_admin accepts the agent key or a Firebase admin, and
  rejects the Discord key. The "key is configured" guard is load-bearing:
  compare_digest("", "") is a match, so a deployment that never set the key
  would otherwise accept an empty credential.
- set_flags() writes an audit_log entry with before/after values and the actor,
  wrapped so an audit failure cannot lose the flag write or 500 the route.
- Cascade helper sets the global doc and every system carrying an ai_flags
  override in one call. A global False already beats everything, but a system
  False beats a global True, so turning AI *on* could half-apply and leave a
  radio system hot after shutoff. It scans for the override rather than
  hardcoding the two known system IDs, so a new system cannot silently defeat
  it.
- cascade defaults to False. PUT /systems/{id}/ai-flags and the AiFlagsPanel
  toggle mean a per-system override is deliberate operator intent; cascading by
  default would erase it on any unrelated global flip. The runbook opts in.

Issue items 5 and 6 (retiring the SSH path from drb-worksession.md) are NOT
done here and the runbook is untouched. The credential does not exist in
production yet, so the SSH path is still the only one that works; retiring it
now would break the next unattended run. Owner activation is recorded on #64.

Tests 273 -> 289.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-30 02:52:00 -04:00
Logan Cusano 0635de8dac Stop alert webhooks putting raw transcripts in a third-party channel
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Successful in 1m44s
Build & Deploy / Report a failed deploy (push) Skipped
Alert dispatch attached a 200-character raw transcript snippet to the
alert_events document and POSTed the same text to the org's Discord
webhook, with no redaction of any kind. Board minutes #42 ratified that
person names are suppressed on every surface until E&O is bound, and a
Discord channel is the least recoverable surface there is: once the text
lands we do not own it, cannot unsend it, and cannot audit who read it.

Raw transcript text now requires two independent gates, both closed by
default:

  1. alert_transcript_snippet_enabled -- an operator switch in config,
     set from the environment.
  2. alert_snippet_opt_in on the org document -- the customer's own
     explicit consent.

Gate 1 is not redundant. The frontend reads and writes Firestore directly
from the browser, so the org flag alone would let an org owner opt
themselves into receiving person names lifted from live public-safety
traffic. Capability is the operator's to grant; consent is the org's.

The gate fails closed on a Firestore error and on a call with no org
(a pre-tenancy node that has not been backfilled) -- a less informative
alert is cheap, an unrecallable disclosure is not. Alerting itself is
unchanged: the webhook still fires and still names the rule, the
talkgroup and the matched keywords.

This does not wait on the Gate B3 redactor (#43, 2026-09-30). The
snippet was a convenience field and needed no redactor to withhold.

Tests assert the person name in a sample transcript does not appear in
either the outbound payload or the Firestore write, in every combination
of the two gates.

Closes server-26#85. Refs #42, #43, #48.
2026-08-29 02:42:31 -04:00
Logan CusanoandClaude Opus 5 3df427f914 Scope Gate B3 so the owner can stop being blocked on it (server-26#43)
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 1m47s
Build & Deploy / Report a failed deploy (push) Skipped
Board minutes #62 decision 6d bars showing live data to a prospect until #43 is
scoped. The conversation count is 0 of 12 with a hard checkpoint on 2026-09-05,
so the scoping document is worth more this week than the implementation, which
is not due until 2026-09-30.

Corrects a premise in #43: the extraction prompt carries no person-name entity
field, so "entities are already extracted" does not hold. Redaction has to work
on raw free text, and that is most of the estimate.

Redaction is specified at write time rather than read time, because the frontend
reads Firestore directly and rules cannot mask a field -- redacting only in the
API would leave the raw document readable in the browser.

EMS exclusion needs a per-talkgroup flag. ai_flags is per-system, and real
systems carry EMS alongside police and fire.

Refs server-26#43, #42, #62, #66, #85.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 03:09:41 -04:00
Logan CusanoandClaude Opus 5 e30d594eea Stop a back-dated call from silently disabling every recency gate (server-26#74)
Build & Deploy / Build & push images (push) Successful in 4m11s
Build & Deploy / Deploy to VM (push) Successful in 1m44s
Build & Deploy / Report a failed deploy (push) Skipped
_call_fits_incident measured incident idle with the signed helper while every
other recency gate in the file uses the unsigned one. On the re-correlation
sweep, `now` is the call's own started_at, which can precede the incident's
last activity, so the value went negative.

Negative idle made `idle_min >= 15` false, which meant the content-divergence
veto never ran and unit overlap was accepted unconditionally -- on a shared
dispatch backbone that is the feedback loop that lets one incident absorb a
whole talkgroup. It also made `idle_min < 20.0` true at any back-dating, so a
tactical channel returned tactical_default for every swept orphan out to the
90-minute bound.

One variable feeds all four gates in the function, so this is a one-line change
at the source. The signed value is untouched where it belongs: callers still
compute corr_incident_idle_min themselves, so debug output keeps its meaning.

Direction is toward more splitting, on the sweep path only, which is the point
-- the bug was suppressing an over-merge veto. Forward-dated calls and anything
inside the thresholds behave exactly as before.

Two tests added alongside the existing idle-gate cases; both fail on the old
line and pass on the new one. 266 pass, 0 fail.

Refs server-26#74, #5, #80.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 02:51:07 -04:00
Logan CusanoandClaude Opus 5 187b8c1500 Declare and create the calls(system_id, started_at) index dedup needs
Dedup was failing on essentially every inbound call in production. dedup.py
queries system_id == X with a started_at range; that composite index was
neither declared in firestore.indexes.json nor present in the live c2-server
database, so the query returned FAILED_PRECONDITION, dedup swallowed it as a
warning, and every duplicate check degraded to "not a duplicate".

While AI is off that only cost duplicate call documents. With a window open it
would have paid Whisper and Gemini twice for every double-heard transmission,
and fed Gate B5's cost measurement a figure that is wrong for a reason
unrelated to the pipeline being measured. Two documents for one transmission is
also the exact input shape that produces a spurious second incident.

The index is created on c2-server and building. This declares it in source so
the file and the live database agree.

Refs server-26#84, #33, #45.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 02:43:21 -04:00
Logan Cusano d18e4f0743 Make "AI is off" true, and stop the transcript PATCH from destroying calls
Build & Deploy / Build & push images (push) Successful in 4m2s
Build & Deploy / Deploy to VM (push) Successful in 1m53s
Build & Deploy / Report a failed deploy (push) Skipped
config/ai_features was not the switch it was documented to be. Three paths
spent money with it off, and one path read it wrong, so per-system opt-outs
did not opt anything out.

- Correlation in the ingest pipeline tested the raw global flag instead of the
  per-system resolution. With a system opted out, extraction was skipped but
  the no-scenes fallback still correlated the call with empty tags, taking the
  thin/recency path and attaching it to whatever incident was most recent on
  that system. The opt-out did not disable correlation, it disabled good
  correlation and left the worst kind running. (#75)

- Transcript correction ran on every transcribed call gated only by an env var,
  spending Gemini tokens and a Places lookup per proposed location. An
  "STT-only" window was never STT-only and its cost could not be attributed.
  Now behind transcript_correction_enabled. (#76)

- _run_extraction_pipeline and the vocabulary learner, both reachable from
  PATCH /calls/{id}/transcript, checked no flags at all. (#76, #81)

The flag resolver now lives in feature_flags.resolve_flags() rather than as a
local helper in upload.py. Three copies of that logic is how #75 happened.

PATCH /calls/{id}/transcript now refuses with 409 when correlation is off.
That route wipes tags, severity, location, units, embedding and unlinks the
call from every incident before queueing re-extraction. Gating extraction
alone would have made it destructive-only in the standing flags-off
configuration: the call left blank and orphaned forever, with the route still
answering 200. The wipe and the rebuild are one transaction in intent, so it
refuses before the first write.

Also: the summarizer's stale-incident sweep is no longer behind
summaries_enabled. It is pure Firestore with no model call in it, and gating
it meant nothing auto-resolved while AI was off - so every incident stayed
active forever and the candidate set every correlation reads kept growing.

transcript_correction_enabled is documented as NOT a pure cost lever. The
corrector is also the noise gate that sets not_speech; with it off, recogniser
noise reaches extraction as a real transcript, comes back thin, and
auto-attaches. Never open an evaluation window with correction off and
correlation on.

14 tests added covering flag precedence, both pipeline paths, the 409, the
correction gate and the summarizer no-op. Suite: 264 passed.

Refs #75, #76, #81, #45.
2026-08-27 02:49:09 -04:00
Logan Cusano 5fc4e2c57b Roll back a bad deploy instead of leaving it live (server-26#65)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy to VM (push) Successful in 2m40s
Build & Deploy / Report a failed deploy (push) Skipped
deploy.yml ran `compose up -d` before the health check and never reverted
on failure. A build that passes tests, returns 200 on /health with the
right git_sha, but has a live logic bug (exactly the class of bug the
correlator instrumentation exists to catch) would stay live indefinitely
- notify-failure would even claim production was "still running the
previous build", which is false in that scenario.

Deploy step now reads /opt/drb/.last_good_tag (written only after a prior
deploy's own health check confirmed its SHA) to capture the previously-
verified tag before switching, and emits it as a step output. Health
check is unchanged in shape (bounded 20x5s retry, still requires the
polled git_sha to match) but now persists the new SHA as the rollback
target only once confirmed live. A new Rollback step runs on any failure
above, re-deploys the previous tag, and re-verifies via the same git_sha
check rather than trusting mere liveness - then fails the job loudly
either way, since the push itself was still bad. notify-failure now
reports what actually happened (rollback succeeded/failed/skipped and to
which SHA) instead of the old unconditional claim.

This unblocks #62 decision 9: autonomous pushes to incident_correlator.py,
llm_correlator.py, intelligence.py and routers/upload.py were frozen until
this rollback path landed.

Refs #65, #62, #60, #57.
2026-08-25 21:34:10 -04:00
DRB CEO agentandClaude Opus 5 a1bdccff45 Gate A: take invented prices off every public surface
Build & Deploy / Build & push images (push) Successful in 4m59s
Build & Deploy / Deploy to VM (push) Successful in 1m42s
Build & Deploy / Report a failed deploy (push) Skipped
/pricing and the homepage teaser rendered the $0/$79/Custom catalog from
lib/billing.ts with a below-the-fold disclaimer. Board minutes #42 ratified
Gate A: no price on a public surface until the model is ratified and the
entitlements exist — a false price anchor with a footnote is worse than no
price. The page has been live in breach since ratification (server-26#46).

- /pricing: no numbers, no plan cards, no interval toggle. "Pricing is in
  development", CTA to the existing /waitlist request-access page.
- homepage: pricing teaser replaced with the same message; PLANS import gone.
- homepage CTAs pointed at /login, which has no signup path — a real visitor
  could not create an account. Now /waitlist ("Request access"); the secondary
  CTA is honestly labelled "Sign in".
- lib/billing.ts: plan catalog header now states the prices are invented and
  that retention/SSO/SLA have no backend, so the next person to import PLANS
  is warned at the definition site.

Refs server-26#46, server-26#62. Authenticated /settings/billing is unchanged
and still stubbed — not a public price surface, stays with #46.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-24 23:34:46 -04:00
Logan CusanoandClaude Opus 5 cc038e6326 A unit call-sign is not a place
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy to VM (push) Successful in 2m12s
Build & Deploy / Report a failed deploy (push) Skipped
"Post 1-2" reached the geocoder, resolved against its talkgroup anchor and
produced a confident pin in the right town for an event with no known
location — while sitting in the same incident's `units` list the whole time.
A plausible wrong pin is worse than no pin: nothing downstream can tell it
is wrong.

Extraction returns `location` and `units` from one pass, so a string in both
is a misclassification, not two facts. Drop it before the geocoder sees it.

Closes server-26#52.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 23:16:48 -04:00
Logan CusanoandClaude Opus 5 964343c819 area_context v2 + Maps place verification (server-26#36, #37)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy to VM (push) Successful in 1m54s
Build & Deploy / Report a failed deploy (push) Skipped
#36 — the correction pass shipped in 58efdbd was right, its reference-data
shape was not. One shape now, at both scopes, every field nullable:

  area_context: { municipality?, county?, state?,
                  center?, radius_km?, resolved_from?, resolved_at?,
                  local_knowledge?: [{term, meaning}] }

`state` closes the ambiguity that made "Ossining" a national guess.
`local_knowledge` replaces roads[]/landmarks[], which could not hold
intersections, schools or nicknames and carried no meanings — `11-X-ray` is
useless alone, `11-X-ray — MTA PD patrol unit` is what a corrector can act on.
Pre-#36 roads[]/landmarks[] are read forward as bare terms so nothing an
operator already entered is lost.

Nullability is the mechanism: which scope gets filled is the operator's
declaration of how homogeneous the system is. One town — fill it once at system
level. Statewide — leave it blank and fill each talkgroup.

The backend owns the derived anchor. PUT /systems/{id} merges config.talkgroups[]
against what is stored instead of writing the client's blob verbatim, which
would have erased the anchor and the pending queue — the same defect as the
ten_codes wipe.

#37 — Maps as a verifier, not as prompt stuffing. The corrector emits its
location nouns; each is geocoded against the talkgroup's anchor, and on a miss
we look for a sound-alike that does resolve there, correct to it, and propose
{term, meaning} to that talkgroup. Cost scales with location nouns, not calls.

No anchor means SKIP. An area too wide to discriminate stores no anchor at all,
because a statewide radius would confirm anything inside it — verification that
passes everything is worse than none, since it reads as a check in the data.

Also re-anchors _geocode_location, which rejected results >40km from the NODE
(server-26#6). An antenna is not a jurisdiction; distance-from-node was always
a stand-in for the anchor and is now only the fallback.

The induction loop proposes at talkgroup level and never promotes. Blast
radius: a wrong term on a channel misleads that channel, the same term
system-wide misleads one 400km away on a statewide system.

38 new tests; 240 pass. Frontend typechecks clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 16:43:59 -04:00
Logan CusanoandClaude Opus 5 58efdbd6eb Correct the transcript before anything reads it
Build & Deploy / Build & push images (push) Successful in 4m0s
Build & Deploy / Deploy to VM (push) Successful in 2m28s
Build & Deploy / Report a failed deploy (push) Skipped
Correction existed, but as a line in intelligence.py's EXTRACTION_PROMPT --
which put it in the wrong place twice over. The same model call that extracted
units, location and severity emitted the correction afterwards, so extraction
reasoned over text already known to be wrong; and it sat behind
correlation_enabled, so during a cost-controlled STT-only window nothing was
ever corrected at all. That is the normal state during development.

internal/transcript_correction.py is now its own pass, between the degenerate
filter and the Firestore write. It receives an already-produced transcript plus
a reference list, so unlike a Whisper prompt it has no series to extend -- the
distinction that keeps vocabulary out of the recogniser's prompt, where an
enumerated ten-code list once made it hallucinate ten-code runs.

Reference data is merged from the talkgroup and the system, TALKGROUP FIRST. A
system spanning several counties can have a talkgroup covering one
municipality, and that municipality's streets must not be buried under a
county-wide list. A single-municipality system is the degenerate case: populate
the system level and every talkgroup inherits it. Area context is now SET --
municipality, county, roads, landmarks, on both scopes -- rather than guessed
from talkgroup names, which is what vocabulary_learner did and which is close
to useless across multiple counties.

Segments are corrected too, not just the joined text. extract_scenes builds its
prompt from numbered segments whenever there is more than one, so a correction
that only fixed the transcript would have been discarded on exactly the
multi-transmission calls carrying the most content. Alignment is enforced: an
array of the wrong length or type is dropped whole, because scenes map back to
transmissions by index and a shifted array would misattribute audio silently.

Whisper is also retried once on degenerate output. Call e49ea32c produced a
56-word ten-code counting run on one attempt and ordinary speech on the next --
same clip, same temperature=0 -- so a hallucination is a coin-flip, and
discarding on the first bad roll threw away a recoverable transcript.

Two things found on the way:

PUT /systems/{id} wiped ten_codes on every save. The systems form sends only
{name, type, config}, and model_dump() wrote every omitted field as its default
over the top. Now exclude_unset. area_context would have been the next victim,
which is why it gets its own route alongside ten-codes rather than a field on
that payload.

Closes server-26#36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 14:23:41 -04:00
Logan CusanoandClaude Opus 5 1bfa856d1b Serve audio as whatever it actually is
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Successful in 1m53s
Build & Deploy / Report a failed deploy (push) Skipped
audio/mpeg was hardcoded at both points call audio is written and served, from
back when the node produced nothing but 16 kbps MP3. It now uploads FLAC, and a
browser will not play a FLAC body labelled audio/mpeg.

storage.py grows one extension -> Content-Type map, used by the GCS upload and
by /media. Keyed off the object's real extension, so every existing .mp3
recording keeps working with no migration -- and _safe_audio_filename already
accepted .flac, so object naming needed nothing.

Also flags what this costs: /media sends the whole body with Accept-Ranges:
none, which was fine at ~60 KB per call and is not fine at ~1.3 MB/min. Noted
at the header and in DEFERRED.md, whose stated reason for deferring Range
support was the old file size.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 12:48:47 -04:00
Logan CusanoandClaude Opus 5 457e6d7e0f Make Archive a real page instead of a redirect
Build & Deploy / Build & push images (push) Successful in 4m7s
Build & Deploy / Deploy to VM (push) Successful in 3m44s
Build & Deploy / Report a failed deploy (push) Skipped
/calls was a ten-line stub that redirected to /incidents, so there was nowhere
in the app to look at a call. The nav's "Archive" link led to the incident
list, and a call that never correlated was invisible entirely -- which is
backwards when correlation quality is the thing under development, because the
orphans are the evidence. Its stated blocker (Gitea #17/#18) closed weeks ago.

The page browses the org's calls newest-first over the new /calls/search route,
filtered by link state (all / orphans / linked), transcript presence, and
system, with a transcript substring search and cursor paging. A row expands to
the full transcript, a playback link minted on demand, and the correlation path
that decided it. The counts line -- how many of the loaded calls are orphaned,
how many have no transcript at all -- is the number worth watching during an
AI window.

Attribution is the point of it: attach an orphan to the incident it belongs to,
or detach one the correlator got wrong. Both go through the routes fixed in the
previous commit, so a manual attachment now actually shows up on the incident.

Admin-only. It exposes every call in the org regardless of node ownership and
carries controls that rewrite incident membership.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 12:36:38 -04:00
Logan CusanoandClaude Opus 5 140dfbfc74 Give the archive a real read, and the debug view a verdict
Build & Deploy / Build & push images (push) Successful in 4m17s
Build & Deploy / Deploy to VM (push) Successful in 1m55s
Build & Deploy / Report a failed deploy (push) Skipped
Three backend pieces the /calls page needs, plus the fix for a debug view that
hid its data exactly when it was wanted.

GET /calls/search — paged, filterable call archive. GET /calls returns every
call in one unordered shot: fine for a node's handful of active calls, useless
as an archive. Only the org scope and the started_at ordering go to Firestore,
since that pair is the one composite index that exists; the rest filters in
Python over a bounded window, the same shape admin.py's debug route uses. The
cursor advances over the scanned window rather than the returned page, or a
sparse filter would re-scan from the same place forever.

Manual attribution. POST /incidents/{id}/calls/{id} only ever wrote the legacy
scalar incident_id, never incident_ids -- which is what the correlator writes
and what the frontend queries with array-contains. A manually attached call was
therefore invisible on the incident page it had just been attached to. It now
maintains both and marks the summary stale. DELETE is new: there was no way to
undo an attachment at all, so a wrong link was permanent.

The debug view no longer filters to AI-enabled systems by default. That filter
emptied the view the moment the flags went off, which is precisely when a
window gets reviewed -- on 2026-08-23 it fell from 100 incidents to 6 between
switching correlation off and opening the tab. ai_systems_only=true restores it.

It also returns a summary block now: corr_path / fit_signal / consensus /
llm_action tallies, transcript coverage on both linked and orphaned calls,
single-call and median-calls-per-incident for fragmentation, max span and
anything past the server-26#22 caps for merging, and the count of incidents
still carrying a fallback "— TGID" title. All of it was being recomputed by
hand from the raw payload on every review.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 12:33:52 -04:00
Logan CusanoandClaude Opus 5 7ef5704be2 Make firestore.indexes.json describe the database again
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Successful in 2m11s
Build & Deploy / Report a failed deploy (push) Skipped
The file had drifted four indexes behind c2-server, so the 2026-08-23 deploy
offered to delete four live indexes and then added ASC copies of two that
already existed as DESC. Reconciled against gcloud's actual list.

Adds the two backend indexes that were live but undeclared and are genuinely
in use -- calls(status, ended_at) for recorrelation_sweep's ended-call scan and
calls(system_id, ended_at) for vocabulary_learner. Deleting either would have
broken a background loop with no frontend symptom.

Declares every index ASCENDING. Firestore scans an index in either direction,
so org_id+started_at ASC already serves the orderBy(started_at, 'desc') that
every frontend hook actually asks for; a matched ASC/DESC pair is one index of
pure write amplification on every call document. The three duplicates now left
undeclared are named in the file header so the next deploy's interactive
delete prompt has a documented answer instead of a guess.

Refs server-26#33.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 12:15:39 -04:00
Logan CusanoandClaude Opus 5 039a06dc72 Let C2 name a talkgroup it already knows
Build & Deploy / Build & push images (push) Successful in 4m5s
Build & Deploy / Deploy to VM (push) Successful in 1m40s
Build & Deploy / Report a failed deploy (push) Skipped
84 of the 100 incidents in the 2026-08-23 dump were titled "Ems — TGID 9048"
or "Other — TGID 9600" -- the fallback, not a description. The title is the
incident's name everywhere it appears: list rows, map pins, Discord alerts.

_create_incident builds it from a content tag and a talkgroup label, and the
label was collapsing to "TGID {id}" because talkgroup_name arrived as None.
It is a plain form field on /upload, forwarded untouched into correlation, and
the node only sends it when OP25 had the name in its loaded tags file -- which
is exactly the case C2 can cover from its own systems collection, where all 125
talkgroup definitions live.

The lookup already existed, on the other path: mqtt_handler resolved it from
the system config on call_start. So the call document held the right name while
the pipeline that titles the incident ignored it. That asymmetry is the bug.

internal/talkgroups.py is now the one implementation -- caller's hint, then the
call document, then the system config -- and both paths use it.
_run_intelligence_pipeline resolves once at the funnel /upload and
/calls/{id}/reprocess share, so the dispatch-channel test, scene extraction and
the title all see a real name. When the call document was the thing missing it,
the resolved name is written back, so the archive and the orphan panel stop
showing a bare TGID too.

Also gives fast/thin a corr_fit_signal. It is 63% of all links and was the only
path writing none, so corr_fit_signal was absent on 295 of 309 calls and the
admin debug view's distribution panel read empty -- looking broken when it was
faithfully reporting that the dominant path records nothing. It now says
thin_recency, which is what actually decided it.

Closes server-26#34. Refs server-26#35 -- the tier's 3.5% invocation rate is a
cost/benefit question, not a bug, and stays open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 03:24:00 -04:00
Logan CusanoandClaude Opus 5 a278e2215a Pin the Firestore deploy target to the c2-server database
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Successful in 2m8s
Build & Deploy / Report a failed deploy (push) Skipped
firebase.json declared rules and indexes with no database key, so the CLI
deploys them to (default). This project does not use (default) -- c2-core
reads FIRESTORE_DATABASE and the frontend reads NEXT_PUBLIC_FIRESTORE_DATABASE,
both c2-server in production, and the index-required errors the browser prints
name /databases/c2-server/ outright. A deploy without this key reports success
and changes nothing the app can see, which is a bad way to find out.

Refs server-26#13.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 02:37:43 -04:00
Logan CusanoandClaude Opus 5 be79499635 Give the nav's dead links somewhere to land
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 1m54s
Build & Deploy / Report a failed deploy (push) Skipped
Three of the app's routes were referenced but never existed, so the redesign's
navigation pointed at 404s from several directions.

/dashboard was the post-login and fallback redirect target in nine places --
login, onboarding, middleware, the admin/nodes/systems/tokens/settings guards,
and the marketing header -- but app/dashboard/ was never created. Signing in
normally dropped the user on a 404. The real signed-in home is "/", which
app/page.tsx already renders as LiveView for an authed user with an org, and
which the nav labels "Live"; all nine now point there.

Nav also linked /watch and /network, neither of which existed. /watch is the
alerts screen under its redesign name, so it re-exports app/alerts/page.tsx
and /alerts stays reachable for old links. /network is new: the "my equipment"
hub the redesign moved /nodes, /systems and /tokens behind and then never
built, which had left /systems and /tokens with no entry point in the UI at
all. Its hooks all run before the admin/operator guard, per d041c86.

Separately, the admin page's guard read isAdmin without authLoading, so every
cold load of /admin -- typed URL, hard refresh, bookmark -- redirected away
while the Firebase claims were still resolving. Admin was only reachable by
clicking through from an already-mounted page. Now it waits, like every other
guarded route does.

And /incidents no longer lies about an empty list: a failed Firestore query
leaves `incidents` empty just as a quiet night does, and the page was printing
"No incidents recorded yet" over the top of a missing-composite-index error.
useIncidents already returned `error`; the page just ignored it. It now renders
an ErrorBanner instead, so the undeployed indexes in server-26#13 read as a
failure rather than as silence on the radio.

Closes server-26#30, server-26#31. server-26#13 stays open -- the rules and
indexes still have to be pushed to the live project by hand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 02:28:02 -04:00
Logan Cusano 861ea41cec Deploy the commit's own images instead of :latest
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Successful in 1m26s
Build & Deploy / Report a failed deploy (push) Skipped
The build-stamp health check added in 8fbfe7d worked on its first run, and
what it caught was not a stale container -- it was a race. Runs 544 and 545
overlapped; both deployed :latest, 545's images won, and 544's health check
correctly reported that the build serving traffic was not the one it had just
deployed.

That is a real hazard, not a false positive: with :latest, two pushes landing
close together means whichever finishes last silently wins for BOTH, and
neither run's log tells you which code is actually live. Pushes land close
together constantly here.

docker-compose.yml already resolved images as ${TAG:-latest}, so the fix is to
export TAG=<commit sha> for the deploy. Each run now pulls and starts exactly
the images it built, rollback becomes "deploy a different tag", and the health
check's assertion becomes meaningful rather than order-dependent. A manual
`docker compose up -d` on the VM with no TAG set still falls back to :latest,
which is the intended escape hatch.

Also replaces the health check's single `sleep 20` with a poll of up to 100s
that stops as soon as the expected SHA appears. A fixed sleep is either too
short -- flaky red runs -- or wastes time on every deploy, and a check that
cries wolf gets ignored, which is exactly the failure this job exists to stop.

Refs logan/server-26#21
2026-08-23 01:38:09 -04:00
Logan CusanoandClaude Opus 5 82c88379d4 Stop an incident lying about what it is and where it is
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 2m5s
Build & Deploy / Report a failed deploy (push) Skipped
An incident header had two independently last-write-wins halves, and in the
2026-08-20 dump both were wrong at once. `b9b4f392` opened on a suspect search
at 80 Grasslands Road; it was labelled "100 South Mosher" (its third call),
pinned at `Westmed` (its second), and titled after the label. Five of six
incidents were pinned somewhere other than the place they claimed to be.

Location and pin are now one value
----------------------------------
`_resolve_location_pair()` computes `location`, `location_coords` and the new
`location_coords_source` together, and `_update_incident`/`_create_incident`/
`_create_master_incident` always write all three. There is no longer a code
path that can move one and leave another behind — including the cross-system
master, which used to take its label from the parent and its pin from the call.

The pin now carries the label it was geocoded from. `_verified_pin()` returns
it only when that source still matches the incident's current label; anything
else is dropped. That includes every pre-existing incident, whose pin has no
recorded source and therefore cannot be reconciled — which is the right
outcome, since the dump says 5 in 6 of those are wrong. A missing pin reads as
missing data; a wrong pin reads as fact, and this is a map people may act on.

An incident also keeps the first place it was given rather than the latest.
Later mentions still accumulate in `location_mentions` (what the map path is
drawn from); they just don't rename the incident's own location. The one
permitted change is filling in a pin the incident never had, from a later call
naming the exact same label — geocoding needs the node position, a quota and a
response, so the same address genuinely does fail once and resolve later.

"49" is not a place
-------------------
`clean_location()` rejects any string with no two-letter word in it, applied at
extraction (intelligence.py, before the geocoder and before the call document)
and again at the correlator's context boundary. `9d376ffe` carried
`location: "49"` from "Fire received. Flames from 49." — a box number — and its
summary asserted "A fire incident was reported at location 49". Nothing
validated that field at all, so it would have recurred.

Title: the founding event, escalation only
------------------------------------------
The title was re-derived from the newest classified call, so `f5190670` was
named after the thirteenth of its thirteen events. It now names the call that
opened the incident, recorded in `title_tag`/`title_severity`, and can only be
replaced by a call of strictly higher severity.

Three candidates were considered:

  * Newest call (status quo) — rejected. The same incident has a different name
    at different times, so a user who saw it in the rail cannot find it again,
    and the name is decided by radio timing rather than by the event.
  * Highest severity alone — rejected as the sole rule. Severity has four
    levels and most traffic sits on one of them, so ties are the common case
    and the tiebreak degrades to "newest" — the defect it was meant to fix.
  * Founding event, escalated by strictly-greater severity — chosen. An
    incident's identity is the event that opened it, so that is its default
    name and it is stable for the incident's whole life. The single case where
    the header MUST change is the one where the situation got worse: a check
    condition that becomes a structure fire is a structure fire, and the
    worst-first rail, the "Major only" filter and the map colour all exist so
    that is never missed. Requiring strictly-greater makes it monotonic, the
    same contract `_max_severity` already gives the severity field: routine
    chatter can never take the name back.

A summary-level title regenerated as a whole was rejected outright: it needs an
LLM call per incident, AI flags are off in production, and every incident today
would have no title at all.

Two renames survive, because neither replaces an event name: filling in the
placeholder title of an incident that opened on a call with no content tags
("Police — Ch 1"), and re-rendering the same event once the incident learns its
address. Incidents created before this change have no `title_tag`, so their
existing title is treated as the founding one rather than handed to whichever
call links next.

Interaction with the caps from 33a247d: `incident_max_duration_minutes` /
`incident_max_calls` bound how far an incident can drift, but they don't fix
this — `b9b4f392` was renamed by its third call, 30 minutes in, well inside
both caps. What the caps do change is the cost of being wrong in the other
direction: a founding-derived title can no longer be left describing a
four-hour chain, because there are no four-hour chains any more.

Tests
-----
tests/test_incident_identity.py, 23 cases: the b9b4f392 chain replayed
end-to-end with the label/pin invariant asserted after every link; the pin not
moving without the label; the same-label pin fill-in; an unverifiable legacy
pin dropped; bare numbers, ten-codes and unit designators rejected at
`clean_location`, at `_build_context` and at incident creation; an unrelated
later call not renaming; a worse call renaming and a calmer one not taking it
back; placeholder fill-in; address learned later; legacy title not claimed.
Each was confirmed to fail against the reverted behaviour. 171 passed.

Closes logan/server-26#23
Closes logan/server-26#26

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 01:35:00 -04:00
Logan Cusano c7be6416f2 Surface LLM correlation fields in debug view; fix unit-continuity path
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Successful in 1m4s
Build & Deploy / Report a failed deploy (push) Skipped
/admin/debug/correlation stripped corr_consensus and the corr_llm_* fields
that upload.py's consensus correlator writes onto the call doc, making it
the one tool built to answer "is the LLM correlation tier alive" unable to
answer it (2026-08-19 dump had to infer LLM state from commit dates instead
of reading it off the data). admin.py's _call_summary() now includes
corr_consensus, corr_llm_reasoning, corr_llm_action, corr_rules_action.

The unit-continuity correlation path never wrote corr_matched_units, unlike
fast/single and fast/disambig, so the debug view showed null for a match
that was in fact unit-driven by construction. Now populated unconditionally
on that path (server-26#16).

Also traced the negative corr_incident_idle_min (-4.1 observed) to its root
cause: the re-correlation sweep anchors `now` to the linking call's own
started_at, and that back-dated value was being written straight into the
incident's updated_at, letting it land before the incident's own
started_at. Added _floor_at_started_at() so updated_at can never precede
started_at. (commit 33a247d already fixed the recency *gates* misreading
that negative value; this fixes the write that produced it.) Verified the
skip_reason filter in recorrelation_sweep.py:63 is already correct, no
change needed there.

Added tests for the debug endpoint's LLM field passthrough, the
unit-continuity corr_matched_units fix, and the updated_at floor — each
confirmed to fail when its fix is reverted. 148 passed, 0 failed.

Closes logan/server-26#24
Closes logan/server-26#16
2026-08-23 01:30:01 -04:00
Logan Cusano 8fbfe7d6de Make a failed deploy impossible to miss, and a wildcard CORS harmless
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Failing after 2m16s
Build & Deploy / Report a failed deploy (push) Successful in 1s
Two unrelated-looking problems with the same shape: a dangerous state that
looked fine from the outside.

DEPLOY (server-26#21). The Deploy job failed on fifteen consecutive pushes
between 2026-08-18 and 08-20 and nobody noticed for two days, because the
build job was green and a red run is only visible to someone who opens Gitea.
Production served 08-18 code the whole time -- including the entire frontend
redesign, chunks 2 through 8. Three changes:

  * The health check now asserts WHICH build answered, not just that something
    did. CI bakes the commit into the image (Dockerfile ARG/ENV GIT_SHA) and
    /health reports it, so a deploy that "succeeds" while the previous
    container keeps running now fails. Liveness alone could never have caught
    this.
  * The image pull retries once after a prune. The actual failure was
    containerd unable to extract a layer -- "failed to Lchown ... no such file
    or directory" -- a corrupted entry in the snapshot store, which a prune
    clears. A second failure after pruning is a real problem (check the VM's
    disk) and still stops the deploy.
  * A notify-failure job POSTs to DEPLOY_ALERT_WEBHOOK when anything in the
    workflow fails. Unset means skip quietly, not fail.

CORS (server-26#20). allow_origins=["*"] with allow_credentials=True is not
the permissive-but-harmless setting it reads as. Starlette does not reject the
pair -- it reflects the caller's Origin back and still sends
Access-Control-Allow-Credentials: true, so the effective policy is "any
origin, WITH credentials", the opposite of what a wildcard normally means.

Rather than trust every deployment to remember CORS_ORIGINS, the pair is now
unrepresentable: a wildcard forces allow_credentials off and logs an ERROR
naming the variable to set. Correctly configured deployments that name their
origins are unaffected and keep credentialed requests.

Severity honestly: low today. c2-core is bearer-auth, and browsers do not
attach bearer tokens cross-origin the way they attach cookies. This is a
misconfiguration waiting for the day something starts trusting a cookie.

Also adds firebase_admin.auth.UserRecord and the list/update/create/delete_user
names to the conftest stub. routers/users.py annotates with UserRecord at
import time, so without it importing app.main failed at collection -- which is
why nothing had ever tested anything wired at app level, CORS included.

Tests: 5 new in test_cors_policy.py, covering the pure policy function, the
middleware actually mounted on the app (so re-hardcoding allow_credentials=True
fails here), and the presence of the build stamp.

Closes logan/server-26#20
Closes logan/server-26#21
2026-08-23 01:26:15 -04:00
Logan CusanoandClaude Opus 5 33a247d306 Stop thin calls fusing a work shift into one incident (server-26#22)
Build & Deploy / Build & push images (push) Successful in 4m3s
Build & Deploy / Deploy to VM (push) Successful in 1m52s
The 2026-08-20 production dump had 4 of 6 sampled incidents as junk chains,
the worst being f5190670: 68 calls over 4h09m, 44 units, 12 tags, at least
13 genuinely distinct events. 58 of 133 linked calls took the fast/thin
path, which is the one path that attaches a call with no fit test at all.

Three defects combined to produce that, and all three are fixed here.

1. What counted as thin was wrong.

is_thin_call was "not units and not vehicles and not coords". A real
dispatch qualified as thin whenever no unit ID parsed and the geocode
failed - six of them did in that dump, including "All units head over to
the powerhouse, 55 Hyman Hills Road ... she's 87 years old", a brand new
job that attached to the four-hour chain and then overwrote its location
and its title. A call is now substantive if it carries tags, a location
string, a severity above routine, or is a reassignment; only genuinely
content-free housekeeping ("10-4", "Copy") stays thin. Those calls now go
through _call_fits_incident like everything else, which on a dispatch
backbone with no positive signal means they open their own incident or
orphan rather than merging.

The reassignment clause closes a self-defeating guard: upload.py blanks
units when dispatch pulls a unit onto a NEW job, specifically to stop
unit-overlap chaining - and blanking units made the call thin, routing it
to the only path with no fit check. The guard produced the merge it
existed to prevent.

2. The thin path was bounded on dispatch channels only.

Every other talkgroup fell through to "thin_pool = tg_recent": any
incident idle up to tg_fast_path_idle_minutes (90), no single-candidate
requirement, no fit test. The 30-second tier-1 / single-candidate tier-2
structure now applies to all channels. Non-dispatch gets its own window,
TG_THIN_IDLE_MINUTES=15, rather than sharing the dispatch value: a
tactical channel really is dedicated to one scene so it earns longer, but
15 sits inside the 20-minute tactical-default window already used in
_call_fits_incident, so the no-evidence path is never more permissive than
the fit-tested path on the same channel.

Recency gates now compare the magnitude of the idle, not the signed value.
The re-correlation sweep anchors "now" to the call's own started_at, so
idle goes negative routinely - incident 9d376ffe recorded
corr_incident_idle_min: -4.1 - and every "idle <= window" test in this
module reads True for a negative number. Those gates had silently stopped
bounding anything for exactly the calls the sweep re-examines.

3. Nothing capped an incident's total size.

Every fit test in the correlator is pairwise: does this call belong with
that incident. Each of f5190670's 68 links was individually arguable; the
mistake was the accumulated shape, which no pairwise rule can see. Two
hard caps now remove an incident from the candidate pool entirely, before
any path can choose it - including the LLM tier, which reads the same
ctx lists.

INCIDENT_MAX_DURATION_MINUTES=120. The one incident in that dump that was
genuinely a single event ran 63 minutes (06:15 wrong-way driver to 07:18
closeout), so the cap has to clear an hour with real headroom. The four
junk chains ran 3h41m, 3h43m, 4h05m and 4h09m, so it has to sit well under
three hours. 120 also equals correlation_window_hours: the location and
slow paths already refuse a candidate older than that, and the fast path
was the only one exempt, so this removes an inconsistency rather than
inventing a number.

INCIDENT_MAX_CALLS=40. A backstop for a burst that fills up inside the
duration cap, not the primary bound. The worst chain averaged ~16
calls/hour while absorbing an entire dispatch backbone, so 40 calls in
under two hours means one incident is eating most of the channel. Set
deliberately above any plausible single-incident call volume (a
multi-alarm fire on its own tactical channel) so this cap errs toward
keeping real incidents whole and lets the duration cap do the cutting.

Capping is not truncation: the incident keeps every call it has and still
auto-resolves on the normal idle sweep. It just stops being a candidate.

Every ambiguous call here was resolved toward a separate incident rather
than a merge. A wrongly-separate incident is visibly wrong and can be
merged later; a wrongly-merged one silently corrupts every unit, tag,
severity and map pin on the incident it joined, and poisons the AI
summary written from them. The cost is some acknowledgements orphaning
instead of riding along on an incident, which is a small, visible loss.

Deliberately NOT changed, since both push toward more merging while the
current failure mode is entirely over-merging (every incident in the dump
has exactly one "new" call; there is no over-splitting left to trade
against):
  - unit-overlap positive feedback on shared dispatch channels, which is
    now bounded by the caps rather than fixed at its root
  - the sweep retry budget expiring before the target incident exists

Tests: 31 new cases in tests/test_correlator_merge_caps.py, including a
replay of the f5190670 night - 13 unrelated jobs at their real offsets,
plus roster unit traffic and acknowledgements every two minutes. Without
the caps that traffic still builds a 125-call incident spanning 244
minutes; with the old thinness test on top, 153 calls over 247 minutes in
3 incidents. With this commit it is 13 incidents, largest 40 calls over 80
minutes. Each new case was checked to fail when the behaviour it covers is
reverted. Suite: 138 passed.

No AI feature flag was touched; correlation stays off in production.

Closes logan/server-26#22

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 03:34:48 -04:00
Logan Cusano baa9d1811f Pin *.sh to LF so Windows checkouts cannot ship a CRLF shebang
Build & Deploy / Build & push images (push) Successful in 4m19s
Build & Deploy / Deploy to VM (push) Successful in 44s
2026-08-20 03:16:28 -04:00
Logan Cusano a250c29e3c Add AI provider degradation registry and alerting (server-26#14)
Build & Deploy / Build & push images (push) Successful in 4m18s
Build & Deploy / Deploy to VM (push) Successful in 1m35s
Three AI dependency failures in one night (retired Gemini model IDs,
depleted Gemini balance, unpayable OpenAI account) each surfaced only
as a single ERROR log line that nobody was watching. Add
app/internal/ai_health.py, a shared in-memory registry that
transcription.py and llm_correlator.py report into on every call
(success and failure), distinguishing permanent conditions (dead
model, dead billing) which alert immediately from transient ones
(rate limits, network blips) which only alert after they persist.
Alerts POST once per degradation episode and once on recovery to an
optional Discord webhook (AI_ALERT_WEBHOOK_URL), reusing alerter.py's
httpx pattern. State is exposed unauthenticated at GET /health/ai
alongside the existing /health.

Closes logan/server-26#14
2026-08-20 03:14:22 -04:00
Logan Cusano 5355095c48 Compare node API keys in constant time on /upload
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Successful in 46s
/upload compared the per-node API key with a plain !=, which short-circuits on
the first differing byte and so leaks a little information about how much of a
guess was correct.

The reason to fix it is less the timing channel itself -- an HTTP round trip is
noisy -- than the inconsistency: enrollment.py and dynsec.py both went out of
their way to use secrets.compare_digest for the same class of credential, so the
codebase contradicted itself on whether this mattered. Now it does not.

Also coalesces a missing api_key field to "" so compare_digest is never handed
None, which would raise TypeError and turn a malformed node_keys document into a
500 instead of a 401.

Closes logan/server-26#12
2026-08-20 03:08:06 -04:00
Logan CusanoandClaude Opus 5 6dfa5bc66d fix: repair 10 stale tests in test_mqtt_handler.py and test_node_sweeper.py
Build & Deploy / Build & push images (push) Successful in 4m15s
Build & Deploy / Deploy to VM (push) Successful in 2m0s
All 10 failures were tests that had drifted behind the product code, not
regressions in it. Diagnosed each individually:

test_mqtt_handler.py:
- test_checkin_creates_new_node, test_checkin_new_node_defaults_lat_lon:
  unpacked 4 positional args from doc_set.call_args[0], but
  fstore.doc_set(collection, doc_id, data, merge=False) always passes
  merge as a kwarg, so only 3 positional args are ever recorded. Fixed
  the unpack to 3.
- test_call_start_creates_call_doc, test_call_start_uses_now_when_started_at_missing:
  mocked fstore.doc_get, but _on_call_start looks the node up via the
  cached fstore.doc_get_cached (added when Firestore reads were cut to
  stay in the free tier). The unmocked doc_get_cached returned a bare
  MagicMock, which isn't awaitable. Mocked doc_get_cached instead; also
  fixed the same 4-vs-3 positional-arg unpack on doc_set's merge=False call.
- test_call_end_updates_status_and_times, test_call_end_sets_audio_url_when_present:
  mocked fstore.doc_update, but _on_call_end now writes via
  fstore.doc_set(merge=True) (see the "Fix Upload 404 warning" commit —
  doc_update raised "No document to update" when call_end arrived before
  call_start). Also calls doc_get_cached to stamp org_id. Mocked
  doc_get_cached and asserted against doc_set instead of doc_update.

test_node_sweeper.py:
- test_stale_online_node_marked_offline, test_stale_recording_node_marked_offline,
  test_tz_naive_last_seen_is_handled, test_only_stale_nodes_updated_in_batch:
  _sweep() now calls app.routers.tokens.release_token(node_id) for every
  node it marks offline (added in 2a690ec, the PulseAudio/Discord-token
  work). These tests never mocked it, so the module-level
  patch("asyncio.to_thread", ...) meant for the node-query call leaked
  into release_token's own internal to_thread call, feeding it raw node
  dicts where it expected Firestore doc snapshots with .id — hence
  "AttributeError: 'dict' object has no attribute 'id'". Patched
  app.routers.tokens.release_token directly (it's imported inline inside
  _sweep, so patching the source module works); the batch test also now
  asserts release_token fires for exactly the two nodes that went offline.

No product code changed — app/internal/mqtt_handler.py, app/routers/tokens.py,
and app/internal/node_sweeper.py all behave as intended. This was pure test
drift across two unrelated feature additions (Firestore-read caching,
Discord-token release-on-offline) that landed without their tests being
updated.

93 passed, 0 failed.

Closes logan/server-26#10.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 03:06:41 -04:00
Logan Cusano 4919b02238 Frontend redesign chunk 8: incidents browse
Build & Deploy / Build & push images (push) Successful in 4m29s
Build & Deploy / Deploy to VM (push) Failing after 3m7s
Rewrite app/incidents/page.tsx per UI_REDESIGN.md chunk 8. Replaces the old
active/resolved two-table split with a single timeline-grouped list (Today
/ Yesterday / date), each row using the same rail-card anatomy as Live's
incident panel — severity spine + type glyph + severity chip + ON AIR pill
(from useActiveCalls, matching a call's incident_ids against the row) +
title + location + on-scene unit chips + age/call-count — so status is a
chip on the row instead of a section boundary, and Live/Incidents visibly
read as the same object at two densities. Severity filter and sort are
unchanged. The create-incident modal and resolve action are unchanged.

Per UI_REDESIGN.md chunk 8.
2026-08-19 23:08:36 -04:00
Logan Cusano 4b5cf1971e Frontend redesign chunk 7: incident detail rebuild
Build & Deploy / Build & push images (push) Successful in 4m32s
Build & Deploy / Deploy to VM (push) Failing after 3m47s
Rewrite app/incidents/[id]/page.tsx to UI_REDESIGN.md §5.2. Header is now
type glyph + SeverityMark + active/resolved chip + a 27px title, with
elapsed time, path length (haversine sum over geocoded calls) and call
count as a single subline. Summary is promoted out of the old tab into a
first-class prose block (16.5px/1.58) — it's the artifact the product
sells, so it gets the best position instead of competing with Units/
Details behind a click. Units/Details tabs are gone; On scene / Cleared
render directly from units_active/units_cleared (chunk 3), Vehicles below.

New components/CallSpineEntry.tsx replaces CallRow for this page (CallRow
stays for the Archive table until chunk 12): time-ordered entries with a
numbered stop marker that matches the map's path stops via the same
sort-by-started_at-over-geocoded-calls index MapView's IncidentPathLayer
uses — the "shared index" from §2.4. Includes an inline play/scrub audio
player (lazy-fetches the signed URL on first play, same pattern CallRow
already used), transcript in sans prose instead of a font-mono <pre>, unit/
cleared-unit chips, and a paginating "N earlier calls" control. Thin/
status-only calls collapse to one line.

The incident map keeps the location_coords guard and now passes `calls`
through to MapView so its path polyline (chunk 5) renders here too.

Per UI_REDESIGN.md chunk 7.
2026-08-19 23:07:28 -04:00
Logan Cusano bc636c00ce Frontend redesign chunk 6: Live view
Build & Deploy / Build & push images (push) Successful in 4m39s
Build & Deploy / Deploy to VM (push) Failing after 6m9s
New components/LiveView.tsx renders the default landing at "/": full-bleed
MapView (rail + legend from chunk 5) plus a new TimeScrubber strip below
it — real call-density bars over the selected 1h/6h/24h/7d window, tinted
by the worst severity in each bucket, playhead pinned to NOW. The playhead
doesn't scrub yet; that needs `resolved_at` on incidents, which doesn't
exist server-side (blocked chunk 13, in DEFERRED.md) — the density data
itself is live, not a fixture.

Distinguishes the two empty states UI_REDESIGN.md §4 calls out: a
configured-but-quiet org (nodes online, zero active incidents) now shows
"Listening — last check-in Xm ago" instead of rendering nothing, separate
from the zero-node case (chunk 10's Activation screen).

app/page.tsx's HomePage now renders LiveView directly for a signed-in,
provisioned user instead of the chunk-4 interim redirect to /incidents.

Per UI_REDESIGN.md chunk 6.
2026-08-19 23:05:49 -04:00
Logan Cusano 31c0b3addf Frontend redesign chunk 5: MapView rewrite — draw the incident path
Build & Deploy / Build & push images (push) Successful in 4m24s
Build & Deploy / Deploy to VM (push) Failing after 9m20s
The flagship feature: a police pursuit has never been drawn as a path.
Add an IncidentPathLayer that, for each incident, takes calls with
location_coords (now declared on CallRecord as of chunk 3), sorts them by
started_at, and draws a <Polyline> with numbered stop markers — first stop
hollow, last stop haloed, using the same index the call spine will use in
chunk 7 (UI_REDESIGN.md §2.4's "shared index"). Needs no backend; per-call
geocodes are already written by intelligence.py. MapView takes a new
optional `calls` prop (the caller's already-loaded recent calls) and
groups them by incident_id internally, so it stays a pure presentation
component.

Retheme markers onto the §2.3 encoding: incident pins are a teardrop with
the type glyph knocked out (from TypeGlyph's paths, duplicated as raw SVG
since Leaflet icons are HTML strings, not React nodes), filled by severity
colour and hollow-with-ink-stroke for minor/routine; node markers are
NodeMark-style diamonds via a shared nodeDiamondSvg() helper, deleting
statusColor() and all its green. Legend rebuilt shape-first (severity
glyphs + node diamond weights, never a bare colour swatch) and reads
correctly in both themes via the surface/ink tokens instead of the old
bg-gray-950/90 that had no light mapping. Removed the three dead
placeholder overlays (News Alerts, ADS-B, Meshtastic). Fan-cluster
grouping (computeGroups) is unchanged.

Per UI_REDESIGN.md chunk 5.
2026-08-19 23:03:53 -04:00
Logan Cusano eaae452d4e Frontend redesign chunk 4: navigation and routing
Build & Deploy / Build & push images (push) Successful in 4m21s
Build & Deploy / Deploy to VM (push) Failing after 11m50s
Rewrite Nav.tsx to the five-destination IA from UI_REDESIGN.md §3 (Live,
Incidents, Archive, Watch, Network) on tokens/sans type, with Settings,
Admin, Trips and Profile moved into the avatar dropdown instead of sitting
as nav peers. Network stays gated to admin/operator, matching the write
boundary its constituent pages (nodes/systems/tokens) already had.

Delete app/dashboard/page.tsx — its incident cards become the Live rail,
its node cards become Network, its call table becomes Archive; nothing on
it is unique. Add app/map/page.tsx -> redirect('/') and rewrite
app/calls/page.tsx -> redirect('/incidents') (Archive/search is blocked on
backend work, chunk 12).

ChromeSwitcher now gives a signed-in user at "/" the app shell instead of
marketing chrome; app/page.tsx branches the same way, sending a signed-in
provisioned user to /incidents as an honest interim until the Live screen
itself lands (chunk 6) — marketing content and behavior for signed-out
visitors is unchanged.

Left the light-mode !important overrides in globals.css in place past this
chunk (deviating from the chunk 4 acceptance criteria) — they still back
every page outside this redesign's 11-chunk scope (settings, admin,
profile, marketing). Deleting them now would break light mode on all of
those. Logged in DEFERRED.md.

Per UI_REDESIGN.md chunk 4.
2026-08-19 23:01:13 -04:00
Logan Cusano 8fdedee25b Frontend redesign chunk 3: type layer honesty and the duplicate fix
Build & Deploy / Build & push images (push) Successful in 4m18s
Build & Deploy / Deploy to VM (push) Failing after 2m7s
Declare the fields the backend already writes and the UI was discarding:
CallRecord gains location_coords, units, vehicles, cleared_units,
duplicate_of, srcaddr (intelligence.py ~315-327); IncidentRecord gains
units_active, units_cleared, location_mentions, last_thin_at
(incident_correlator.py _attach, ~1270-1300).

Filter duplicate_of client-side in useCalls.ts's three hooks (useCalls,
useCallsByIncident, useActiveCalls) so a call flagged as a second node's
recording of the same transmission no longer renders twice. Client-side
rather than a where() clause to avoid a new composite index.

Removes the two now-resolved DEFERRED.md entries (dedup/useCalls,
lib/types.ts field gaps).

Per UI_REDESIGN.md chunk 3.
2026-08-19 22:57:21 -04:00
Logan Cusano 70d63abeaa Re-evaluate incident severity on link, stamp resolved_at at every resolution site
#17: severity was written once at _create_incident and never touched again,
so an incident that opened routine and escalated to a working fire stayed
routine forever. _update_incident now merges call_severity into the incident
via _max_severity() on every link.

Severity is monotonic: it only ever rises, never falls. An incident briefly
assessed "major" genuinely was major at that moment; a later, calmer-sounding
call is evidence the situation is winding down, not that the earlier read was
wrong. status/resolved_at exist to retire an incident — severity should stay
as the high-water mark so the worst-first rail, "Major only" filter, and map
colouring never bury a call that was genuinely major. See _max_severity's
docstring in incident_correlator.py for the full argument.

#18: none of the resolution sites wrote resolved_at, so an incident's
lifespan couldn't be reconstructed for the history-scrub feature. Added
resolved_at alongside status="resolved" at all six sites that flip it:
  - incident_correlator.py _update_incident (signal-based: units all cleared)
  - incident_correlator.py maybe_resolve_parent (master auto-resolve)
  - summarizer.py _stale_sweep (90-minute auto-resolve)
  - upload.py, both scene-resolution loops (single- and multi-scene)
  - calls.py reprocess/correction path
(_update_incident's signal-resolve and maybe_resolve_parent's master-resolve
weren't named in the issue's four call sites, but they set status the same
way and were missing resolved_at too.)

No backfill: existing resolved incidents keep resolved_at = null, which
means "resolved before this field existed," not "never resolved." Backfilling
from updated_at would be a guess dressed up as data.

Tests: added to tests/test_correlator_gate.py, which needs no Firestore for
the pure _max_severity cases and patches fstore for the _update_incident/
maybe_resolve_parent writes. Covers the escalation case (routine -> major),
the no-downgrade case, and resolved_at on both the signal-resolve and
master-resolve paths. 52/52 passing in that file; 83 passed / 10
pre-existing failures for drb-c2-core overall (baseline was 69/10 — the
+14 is exactly the new tests, no regressions).

Fixes #17, #18.
2026-08-19 22:57:20 -04:00
Logan Cusano 3a786bc227 Frontend redesign chunk 2: primitives and marks
Build & Deploy / Build & push images (push) Successful in 5m11s
Build & Deploy / Deploy to VM (push) Failing after 3m33s
Rewrite components/ui/* (Button, Card, Badge, PageHeader, EmptyState,
Skeleton) against the chunk-1 tokens instead of hardcoded gray-9xx classes,
and drop the remaining font-mono from label/heading text.

Add the three colour-blindness-validated encoding components from
UI_REDESIGN.md §2.3:
- components/marks/SeverityMark.tsx — glyph (filled triangle / outline
  triangle / outline circle) + optional spine + optional label, from the
  four-level severity ladder. Colour is never the only channel.
- components/marks/TypeGlyph.tsx — five stroked SVG glyphs (fire, police,
  ems, collision, other) in currentColor. Incident type is now shape, not
  hue, since five hues can't clear an all-pairs CVD gate.
- components/marks/NodeMark.tsx — diamond at four weights (filled+ring /
  filled / hollow / hollow-dashed). Green is gone from node state entirely.

lib/severity.tsx now renders through SeverityMark; SEVERITY_COLORS reads
the validated sev-moderate/sev-major tokens with routine/minor neutral.
IncidentBadges.tsx's TypeBadge is reimplemented on TypeGlyph instead of a
coloured pill.

Per UI_REDESIGN.md chunk 2.
2026-08-19 22:56:02 -04:00
Logan Cusano c6bc712b54 Frontend redesign chunk 1: design tokens and type
Build & Deploy / Build & push images (push) Successful in 4m10s
Build & Deploy / Deploy to VM (push) Successful in 1m57s
Replace hardcoded dark-palette Tailwind classes with semantic CSS custom
properties (page/surface/raised/line/ink/accent/sev-moderate/sev-major/
map-*) defined on :root (light) and .dark (dark), wired through
tailwind.config.ts theme.extend.colors. Add IBM Plex Sans/Mono via
next/font/google: sans for everything a person reads, mono reserved for
machine identifiers only. Drop font-mono from body and Button's base
classes. Existing !important light-mode overrides kept temporarily so
nothing goes unreadable mid-migration (removed in chunk 4).

Per UI_REDESIGN.md chunk 1.
2026-08-19 22:53:20 -04:00
Logan CusanoandClaude Opus 5 d041c8648d Run every hook before the admin guard on /nodes and /systems
Build & Deploy / Build & push images (push) Successful in 4m6s
Build & Deploy / Deploy to VM (push) Successful in 2m25s
Both pages crashed to a blank "client-side exception" screen in production.
React error #310: the useState calls sat *below* `if (authLoading || (!isAdmin
&& !isOperator)) return null`, so the first render returned before reaching
them and the next render, once auth resolved, ran more hooks than the previous
one. React tracks hooks by call order and refuses.

The guard itself is fine and stays where it is -- only the hook declarations
move above it. Behaviour is unchanged for a user who passes the guard, and a
user who fails it still renders nothing before the effect redirects them.

Found by walking the deployed site: /nodes and /systems were the only two
routes that failed outright rather than merely showing empty data. The empty
data everywhere else is the org_id backfill, which is a separate problem.

npx tsc --noEmit clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 23:25:24 -04:00
Logan CusanoandClaude Opus 5 bc191fb59f Stop one malformed call document 500ing the whole debug view
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 9m48s
/admin/debug/correlation built its call lookup as {doc["call_id"]: doc}, which
raises KeyError on any stored call missing that field -- and at least one in
production is missing it. One bad document took down the entire view rather
than dropping a single call from it.

The document id is authoritative and always present; the call_id *field* is
written by the upload path and evidently has not always been. Keying off the id
we asked for removes the dependency on the field entirely.

Found while generating a correlation dump server-side, because the UI route this
serves has been unusable tonight.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:03:17 -04:00
Logan CusanoandClaude Opus 5 157be0c049 Serve Firebase's auth handler from our own domain
Build & Deploy / Build & push images (push) Successful in 4m8s
Build & Deploy / Deploy to VM (push) Failing after 23s
Google sign-in fails in production: the popup opens, flashes, closes, and the
page shows a generic failure with nothing in the console or the network tab.

The app is served from drb.cusano.net while signInWithPopup opens its handler on
the project's firebaseapp.com origin. Chrome partitions third-party storage, so
the popup cannot read back the state its opener wrote and dies immediately.
Visiting the handler directly says so: "missing initial state ... a
storage-partitioned browser environment". Nothing about authorised domains or
the build was wrong -- the shipped bundle carries the correct apiKey and
authDomain, which is exactly what made this look like a code bug.

Caddy now proxies /__/auth/* on the bare domain to the Firebase Hosting origin,
rewriting Host so Firebase recognises the request. Same-site again, which is
Google's documented fix. The vhost becomes a `route` so the handler matches
before the catch-all proxy to Next.

The upstream host is a jinja default rather than a group_vars entry because
group_vars/all.yml is gitignored; override it there if the project ever moves.

Two manual steps remain, and all three parts are required or nothing changes:
the CI secret FIREBASE_AUTH_DOMAIN must become drb.cusano.net with a frontend
rebuild, and drb.cusano.net must be an authorised domain in the Firebase
console. This template also needs an ansible run -- CI alone will not deploy it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 21:59:07 -04:00
Logan Cusano 4dc3f27ac4 Fix the no-org redirect loop and swallowed Google sign-in errors
Redirect chain traced across middleware.ts, ChromeSwitcher.tsx and
AuthProvider.tsx before touching anything, per the ask. Those three were
already correct as of c7f985d/2a1d52b/83416fe (middleware exempts
/onboarding and /signup from the drb_session cookie gate, ChromeSwitcher
sends any signed-in no-org user to /onboarding, AuthProvider only sets the
cookie once an org_id claim exists). The actual loop was one file upstream
of all three: app/login/page.tsx hardcoded `router.push("/dashboard")`
after both the email/password and Google handlers resolved. That push
races AuthProvider's async onAuthStateChanged -> getIdTokenResult ->
cookie decision. For a no-org account the cookie never gets set, so
middleware bounces the very next request back to /login with no
explanation — the ping-pong the coordinator saw live.

Fix: login page no longer navigates from the handlers. It waits on
AuthProvider's own `loading`/`orgId` and redirects once claims are
settled (/dashboard with org_id, /onboarding without). This also fixes a
second case: a user who lands on /login already signed in (e.g. bounced
there by middleware while their Firebase session was still valid) now
gets routed the same way instead of sitting inert on a login form with no
feedback. /onboarding itself (org-name form, single action) was already
adequate as the "explain the state" screen once the loop stopped
recreating it.

Also, live tonight: Google sign-in was failing outright in prod with no
console/network trace. app/login/page.tsx's Google handler did
`catch { setError("Google sign-in failed. Try again.") }` — no binding,
error discarded. Added lib/authErrors.ts: logs the raw error, and maps
Firebase codes to messages that distinguish two categories — the user's
own situation (popup blocked/closed, bad password, network) says "try
again"; deployment misconfiguration (auth/unauthorized-domain,
auth/operation-not-allowed) says so explicitly and does not suggest
retrying, since retrying can't fix a missing authorized-domain entry or a
disabled provider. Applied to both handlers in login/page.tsx and both
in signup/page.tsx (same swallowing pattern, same fix). Per the
coordinator's steer: this is diagnosis only — no popup-to-redirect
fallback, no auth method change. If production is hitting
auth/unauthorized-domain, that's a Firebase Console fix
(drb.cusano.net -> Authorized domains), not a code fix.

Nav.tsx: sign-out was only reachable from /profile. Added a profile
dropdown (desktop) and drawer entries (mobile) with Profile / Refresh
access / Sign out, so sign-out is reachable from anywhere in the app.

"Refresh access" calls AuthProvider.refreshClaims() (already existed,
already used by /onboarding after signup) so a user whose role or org
was just changed server-side can pick it up without a full logout.

Decision on unknown Google accounts (point 4): kept self-serve org
creation via /onboarding rather than a "request access" pending state.
BUSINESS_MODEL.md #2.1 already answers this for the owner: "a limited
free public tier *and* full paid access without contributing... cash is
the primary revenue line from day one." A pending-approval gate would
contradict that — it would make org creation itself the thing being
gated, when the model explicitly does not want contribution (or approval)
to be the only door. Self-serve org provisioning via POST /auth/signup
was already built for this (2a1d52b) and needed no further gating
decision, just for the loop in front of it to stop.

Reversible: no schema change, no new gating, no billing/Stripe touched.
Bench: rsync'd to the WSL-native ~/drb-frontend workspace and ran
`npx tsc --noEmit` there (per CLAUDE.md — the H: drive install path is
not viable) — exit 0, no errors. No Python touched this pass.
2026-08-18 21:57:39 -04:00
142 changed files with 17332 additions and 1877 deletions
+4
View File
@@ -0,0 +1,4 @@
# Shell scripts run inside Linux containers. A CRLF shebang there fails as
# "bad interpreter: /bin/sh^M", which surfaces only as a container that will
# not start. Windows checkouts have core.autocrlf=true, so pin these to LF.
*.sh text eol=lf
+302 -9
View File
@@ -31,6 +31,8 @@ jobs:
with:
context: ./drb-c2-core
push: true
build-args: |
GIT_SHA=${{ gitea.sha }}
tags: |
${{ env.REGISTRY }}/c2-core:latest
${{ env.REGISTRY }}/c2-core:${{ gitea.sha }}
@@ -61,11 +63,16 @@ jobs:
NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=${{ secrets.FIREBASE_MESSAGING_SENDER_ID }}
NEXT_PUBLIC_FIREBASE_APP_ID=${{ secrets.FIREBASE_APP_ID }}
NEXT_PUBLIC_FIRESTORE_DATABASE=${{ secrets.FIRESTORE_DATABASE }}
NEXT_PUBLIC_MAP_TILE_URL=https://tile.openstreetmap.org/{z}/{x}/{y}.png
deploy:
name: Deploy to VM
needs: build
runs-on: ubuntu-latest
outputs:
prev_sha: ${{ steps.deploy.outputs.prev_sha }}
rollback_status: ${{ steps.rollback.outputs.status }}
rollback_sha: ${{ steps.rollback.outputs.rolled_back_to }}
steps:
- name: Check runner outbound IP
@@ -78,27 +85,313 @@ jobs:
ssh-keygen -l -f /tmp/deploy_key
- name: Deploy
id: deploy
run: |
ssh -o StrictHostKeyChecking=no \
set -o pipefail
OUTPUT=$(ssh -o StrictHostKeyChecking=no \
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
-o ConnectTimeout=15 \
-v \
-i /tmp/deploy_key \
drb@${{ secrets.SERVER_IP }} << 'ENDSSH'
drb@${{ secrets.SERVER_IP }} << 'ENDSSH' | tee /dev/stderr
set -e
cd /opt/drb
# server-26#129: every deploy pushes 3 freshly SHA-tagged images and
# nothing ever removed the old ones except a prune that only ran
# AFTER a successful `compose pull` -- so a run that never got that
# far (this one) left the leak unaddressed forever. That silently
# filled the disk to 100% over ~week of deploys (2026-09-12: 29G/29G
# used, 96 of 100 local images unreferenced, 23.76GB reclaimable) and
# took `git pull` itself down with "No space left on device" before
# the deploy could even determine a rollback target. Prune BEFORE
# doing anything else, not after: `docker image prune -af` only
# removes images with no container referencing them, so it can never
# touch what's currently running -- there is nothing here for a
# mid-flight deploy to lose. Warn-not-fail: a prune failure must not
# block a deploy that doesn't actually need the space this time.
docker image prune -af || echo "WARNING: pre-deploy image prune failed (server-26#129) -- disk pressure may persist"
# Update compose files + mosquitto config
git pull origin main
# Pull pre-built images and restart (no build on the VM)
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --remove-orphans
docker image prune -f
# server-26#51: Firestore rules/indexes deploy used to be attempted
# HERE, over SSH, gated on the VM having firebase-tools installed.
# It never did (no node on the VM), so this silently warned and
# skipped on every deploy for weeks -- PR #124 even auto-closed
# #13/#51 as if it were fixed. Moved to a standalone
# deploy-firestore-rules job below that runs on the Gitea runner
# itself (which always has node), so it no longer depends on
# anything being pre-installed on this VM.
# server-26#65: capture what is actually live BEFORE switching, so
# a bad deploy has something concrete to fall back to. This reads
# from a state file rather than re-deriving it from git log,
# because a PRIOR deploy could itself have failed and already
# rolled back to something older than HEAD~1 -- the file is only
# ever written by the Health check step below, after that step
# has confirmed the tag it names actually answered /health. A
# fresh VM with no file yet falls back to :latest, same escape
# hatch as a manual `up -d` with no TAG set.
#
# server-26#156: `cat missing-file || echo latest` only falls back
# when cat itself fails (nonzero exit) -- a file that EXISTS but is
# EMPTY (the state this file was found in, 2026-09-20) makes cat
# succeed with empty output, so PREV_TAG became "" instead of
# "latest". That "" then failed the emptiness check below and
# exited 1 -- AFTER git pull + up -d had already succeeded -- which
# skips the Health check step entirely (later steps don't run after
# a failure), and Health check is the ONLY thing that ever writes a
# real value here. Self-perpetuating: every deploy failed the same
# way forever, with the app itself deploying fine underneath it.
# ${VAR:-default} covers empty AND unset in one expansion.
PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null)
PREV_TAG="${PREV_TAG:-latest}"
echo "PREV_TAG=$PREV_TAG"
# Deploy THIS commit's images, not :latest. Overlapping runs are
# normal here, and with :latest whichever finishes last wins for
# both -- run 544 asserted its own SHA and found run 545's build
# already serving. compose already supports ${TAG:-latest}, so
# pinning makes each deploy deterministic and a rollback just a
# different tag. A later manual `up -d` on the VM without TAG set
# still falls back to :latest, which is the intended escape hatch.
export TAG=${{ gitea.sha }}
# Pull pre-built images and restart (no build on the VM).
#
# The retry is not defensive padding: this exact step failed fifteen
# deploys in a row (2026-08-18 to 08-20) with containerd unable to
# extract a layer -- "failed to Lchown ... no such file or directory"
# -- a corrupted entry in the snapshot store. Pruning clears the bad
# layer and the second pull succeeds. If it fails again after a
# prune that is a real problem (check the VM's disk) and should stop
# the deploy rather than be retried forever.
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
if ! $COMPOSE pull; then
echo "image pull failed - pruning and retrying once"
docker image prune -af
$COMPOSE pull
fi
$COMPOSE up -d --remove-orphans
# server-26#129: -f alone only removes dangling (untagged) images --
# the SHA-tagged image from every PAST deploy is not dangling, just
# unreferenced once `up -d` swaps the running container to the new
# tag, so it survived this indefinitely. -a catches those too; see
# the pre-pull prune above for why this can't touch anything live.
docker image prune -af
ENDSSH
)
echo "$OUTPUT"
PREV_TAG=$(printf '%s\n' "$OUTPUT" | grep '^PREV_TAG=' | tail -n1 | cut -d'=' -f2)
if [ -z "$PREV_TAG" ]; then
echo "Could not determine the previous tag from deploy output - rollback target unknown."
exit 1
fi
echo "prev_sha=$PREV_TAG" >> "$GITHUB_OUTPUT"
- name: Health check
id: health
run: |
sleep 20
curl -f https://api.${{ secrets.DRB_DOMAIN }}/health || \
(echo "Health check failed" && exit 1)
# Poll rather than sleep-once: the container has to finish starting,
# and a fixed sleep is either too short (flaky red) or wastes time on
# every deploy. A health check that cries wolf gets ignored, which is
# the failure mode this whole job exists to prevent.
BODY=""
for _ in $(seq 1 20); do
sleep 5
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || continue
case "$BODY" in *"${{ gitea.sha }}"*) break ;; esac
done
if [ -z "$BODY" ]; then
echo "Health check failed: /health never responded"; exit 1
fi
echo "$BODY"
# Liveness alone is not enough. A deploy can report success while the
# PREVIOUS container keeps serving -- that is how production ran
# 08-18 code for two days without a single red run. Assert that the
# build which answered is the commit we just pushed.
RUNNING=$(printf '%s' "$BODY" | tr ',' '\n' | grep git_sha | cut -d'"' -f4)
if [ "$RUNNING" != "${{ gitea.sha }}" ]; then
echo "Deployed build is '$RUNNING', expected '${{ gitea.sha }}'."
echo "The container was not actually replaced."
exit 1
fi
# server-26#65: only now -- confirmed by /health, not by "up -d
# returned 0" -- record this as the rollback target for the NEXT
# deploy. A failure to write this is a bookkeeping problem, not a
# deploy problem, so it warns instead of failing the job (a hard
# failure here would trigger the Rollback step below against a
# perfectly good deploy).
ssh -o StrictHostKeyChecking=no \
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
-o ConnectTimeout=15 \
-i /tmp/deploy_key \
drb@${{ secrets.SERVER_IP }} \
"echo '${{ gitea.sha }}' > /opt/drb/.last_good_tag" \
|| echo "warning: failed to persist .last_good_tag - next deploy's rollback target may be stale"
- name: Rollback on failed health check
id: rollback
if: failure()
run: |
# server-26#65 decision 3 / board minutes #62: up -d used to be the
# last word -- a build that passes tests, returns 200, and still
# corrupts incidents on live traffic would stay live for 12+ hours
# before a human noticed. This step is what makes that impossible:
# any failure above (pull, restart, or the health/SHA check) lands
# here and puts the previously-verified tag back.
PREV_TAG="${{ steps.deploy.outputs.prev_sha }}"
if [ -z "$PREV_TAG" ]; then
echo "No previous tag was captured (Deploy step itself failed before recording one) - cannot roll back automatically."
echo "status=skipped" >> "$GITHUB_OUTPUT"
exit 0
fi
echo "Rolling back to $PREV_TAG"
ssh -o StrictHostKeyChecking=no \
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
-o ConnectTimeout=15 \
-i /tmp/deploy_key \
drb@${{ secrets.SERVER_IP }} << ENDSSH
set -e
cd /opt/drb
export TAG=$PREV_TAG
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
if ! \$COMPOSE pull; then
echo "rollback image pull failed - pruning and retrying once"
docker image prune -af
\$COMPOSE pull
fi
\$COMPOSE up -d --remove-orphans
ENDSSH
# Re-verify exactly like the forward health check does: liveness
# alone doesn't prove the rollback took, the SHA has to match the
# tag we just switched back to.
BODY=""
for _ in $(seq 1 12); do
sleep 5
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || continue
case "$BODY" in *"$PREV_TAG"*) break ;; esac
done
RUNNING=$(printf '%s' "$BODY" | tr ',' '\n' | grep git_sha | cut -d'"' -f4)
if [ "$RUNNING" != "$PREV_TAG" ]; then
echo "ROLLBACK FAILED: expected git_sha '$PREV_TAG', got '$RUNNING'."
echo "Production state is UNKNOWN - check the VM by hand immediately."
echo "status=failed" >> "$GITHUB_OUTPUT"
echo "rolled_back_to=$PREV_TAG" >> "$GITHUB_OUTPUT"
exit 1
fi
echo "Rolled back successfully to $PREV_TAG"
echo "status=success" >> "$GITHUB_OUTPUT"
echo "rolled_back_to=$PREV_TAG" >> "$GITHUB_OUTPUT"
deploy-firestore-rules:
name: Deploy Firestore rules & indexes
needs: build
runs-on: ubuntu-latest
# Deliberately independent of the `deploy` job (app containers) and its
# health-check/rollback chain above: a rules/indexes deploy failure has
# nothing to roll back (there is no previous "build" of a ruleset to
# revert to via this pipeline) and must never be conflated with an app
# deploy failure by triggering that job's rollback logic. This job
# failing is its own, separate red run -- picked up by notify-failure
# below -- not a signal to touch the running containers.
steps:
- uses: actions/checkout@v4
- name: Deploy firestore rules and indexes
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
run: |
set -e
# server-26#51: this used to run over SSH on the deploy VM, gated
# on the VM having firebase-tools installed. It never did, so it
# silently warned-and-skipped on every single deploy for weeks.
# Running it here instead means the only prerequisite is a secret
# -- FIREBASE_TOKEN, from `firebase login:ci` -- rather than
# something installed by hand on a machine this pipeline doesn't
# otherwise touch. A missing token now fails this job LOUDLY
# (picked up by notify-failure) instead of a buried warning line
# nobody reads in the app deploy's logs.
if [ -z "$FIREBASE_TOKEN" ]; then
echo "FIREBASE_TOKEN secret is not set -- cannot deploy Firestore rules/indexes." >&2
echo "Generate one with 'firebase login:ci' and add it as a Gitea Actions secret." >&2
exit 1
fi
npm install -g firebase-tools
cd infra/firestore
firebase deploy --only firestore:rules,firestore:indexes \
--project ${{ secrets.FIREBASE_PROJECT_ID }} \
--token "$FIREBASE_TOKEN" --non-interactive
notify-failure:
name: Report a failed deploy
needs: [build, deploy, deploy-firestore-rules]
if: failure()
runs-on: ubuntu-latest
steps:
- name: Post to Discord
# A red run in Gitea is only visible to someone who opens Gitea, and
# nobody did for two days. Same shape as an AI tier dying quietly,
# which is why both now push a message out of the box instead of
# waiting to be discovered. No webhook configured => skip quietly
# rather than fail, since not every deployment will set one.
env:
WEBHOOK: ${{ secrets.DEPLOY_ALERT_WEBHOOK }}
RUN_URL: ${{ gitea.server_url }}/${{ gitea.repository }}/actions/runs/${{ gitea.run_number }}
SHA: ${{ gitea.sha }}
ROLLBACK_STATUS: ${{ needs.deploy.outputs.rollback_status }}
ROLLBACK_SHA: ${{ needs.deploy.outputs.rollback_sha }}
DEPLOY_RESULT: ${{ needs.deploy.result }}
RULES_RESULT: ${{ needs.deploy-firestore-rules.result }}
run: |
if [ -z "$WEBHOOK" ]; then
echo "DEPLOY_ALERT_WEBHOOK is not set - skipping notification."
exit 0
fi
python3 - <<'PY' > /tmp/payload.json
import json, os
sha = os.environ["SHA"][:8]
run_url = os.environ["RUN_URL"]
status = os.environ.get("ROLLBACK_STATUS", "")
rollback_sha = os.environ.get("ROLLBACK_SHA", "")
deploy_result = os.environ.get("DEPLOY_RESULT", "")
rules_result = os.environ.get("RULES_RESULT", "")
# deploy-firestore-rules runs independent of the app deploy/rollback
# chain (see its own job comment), so its failure needs its own
# branch here -- otherwise this fell through to the generic "Build
# failed before any deploy was attempted" text even when the app
# deployed fine and only the Firestore rules/indexes push failed.
if deploy_result != "failure" and rules_result == "failure":
detail = "App deploy succeeded; Firestore rules/indexes deploy FAILED (server-26#51). Rules may be stale — check FIREBASE_TOKEN and the job log."
# server-26#65: the old text here unconditionally claimed
# "production is still running the previous build" -- true only
# when the pull/restart itself failed. It's false the moment a
# build passes the SHA check but has a live logic bug (exactly the
# class of bug the correlator instrumentation exists to catch), or
# once the deploy job's own rollback path has run. Say what
# actually happened instead.
elif status == "success":
detail = "Automatic rollback to `%s` succeeded. Production is back on the previous good build." % rollback_sha[:8]
elif status == "failed":
detail = ("Automatic rollback to `%s` FAILED. Production state is UNKNOWN -- "
"check the VM by hand immediately.") % rollback_sha[:8]
elif status == "skipped":
detail = "No rollback was attempted (no previous tag captured, or build/push failed before any deploy). Check the VM by hand."
else:
detail = "Build failed before any deploy was attempted. Production is unchanged."
print(json.dumps({"content":
"**DRB deploy failed** on `%s`\n%s\n%s" % (sha, run_url, detail)}))
PY
curl -sS -X POST -H "Content-Type: application/json" \
--data @/tmp/payload.json "$WEBHOOK" || echo "notification POST failed"
+4
View File
@@ -44,3 +44,7 @@ recordings/
# OS
.DS_Store
Thumbs.db
# Out of scope - not a deployed service (server-26#56)
drb-telegram-bot/
.claude/worktrees/
+42
View File
@@ -0,0 +1,42 @@
# Gate B3 (server-26#43) -- Engineering Scope
Owner: CTO. Scope only -- no implementation. Ship date unchanged: 2026-09-30.
## 1. The two defaults, testable
- EMS exclusion: for any call whose talkgroup is classified medical, the AI pipeline (Whisper STT + GPT-4o-mini intelligence.py extraction + correlation) must not run. Opt-in only via a per-customer contract flag. Test: upload a call on a talkgroup marked medical, calls/{id}.transcript stays null, no incident_ids.
- Name suppression: no surface serving a calls or incidents document (API, frontend render, alert webhook, future export/Discord/API-key tiers) may return an unredacted transcript, summary, or title to any account -- public, comped, or paid -- until an E&O policy is bound (#43 comment 1). Test: same document, two reads -- direct Firestore read and /incidents/{id} API read -- both redacted.
## 2. Talkgroup-granularity gap
feature_flags.py:80-107 (resolve_flags) only layers a per-system ai_flags dict (routers/systems.py:107-129, flat {flag_name: bool}, no talkgroup key) on top of the global default. DEFERREDs own entry for this file says the fix shape is talkgroup_ai_flags: {tgid: {...}} on the system doc, consulted where flag() is built. That field does not exist. Without it, "EMS excluded, rest of the system processed" is not buildable -- the flag is all-on/all-off per system, and most systems mix EMS with police/fire dispatch under one system_id (the exact case BUSINESS_MODEL section 5.5 is trying to protect against). This data-model change is a hard prerequisite, not an enhancement: add talkgroup_ai_flags: {tgid: {stt_enabled, correlation_enabled}} to the system doc, consult it in resolve_flags() before the system-level flag, default every unclassified talkgroup on a system that has at least one confirmed-medical talkgroup to excluded until explicitly classified.
## 3. Redaction design -- write time, not read time
Pick: compute and store a redacted copy alongside the raw one, at extraction/summarization time. Two sentences: the frontend reads Firestore directly for calls/incidents (CLAUDE.md gotcha -- middleware.ts is UX-only, Firestore rules are the real boundary), and Firestore rules can allow/deny a whole document but cannot mask one field inside it -- so a redaction step that only runs inside c2-cores API responses leaves the exact same unredacted transcript/summary/title readable by any authenticated browser via onSnapshot/getDocs against the collection directly. The only enforcement point that actually covers both paths is: the client-readable document never contains the unredacted field. Raw content moves to a field/subcollection excluded from client-facing Firestore rules and readable only server-side by c2-core (satisfies "never deletion, reversible the day a policy binds" -- #43 comment 1).
incident.title is template-composed from tag/location/talkgroup (incident_correlator.py:380-389), not LLM freeform -- already name-free by construction, no redaction needed there. The actual carriers are calls.transcript (models.py:165) and the GPT summary (summarizer.py:146-161, built directly from raw transcripts, no name-avoidance instruction today).
## 4. A premise in #43 does not hold
#43s body says "entities are already extracted, so the redaction has a data source to work from." Not true as of this read. intelligence.pys extraction prompt (_PROMPT_TEMPLATE, lines 24-72) has no person-name field -- it extracts tags, incident_type, location, vehicles, units, cleared_units, severity. units is explicitly restricted to "unit IDs or officer numbers... never infer or guess" (line 57) -- radio callsigns, not private-citizen names. There is no structured entity to redact against. Redaction must run against unstructured free text (transcript + GPT summary), via a new regex/NER-style pass with its own unmeasured false-negative rate -- the same class of problem #48 raised about the extractor, one level down, on code that does not exist yet.
## 5. Surface inventory (complete)
- drb-frontend: app/incidents/page.tsx, app/incidents/[id]/page.tsx, app/calls/page.tsx, components/CallRow.tsx, components/CallSpineEntry.tsx -- render title/summary/transcript. Every one is backed by a direct Firestore listener per the section 3 gotcha, not just the page component -- any future onSnapshot/getDocs against calls/incidents inherits the same exposure and must be audited, not assumed covered.
- drb-c2-core API: routers/calls.py, routers/incidents.py (JSON responses).
- drb-c2-core/app/internal/alerter.py:56,68 -- transcript_snippet (200 chars, raw, unredacted today) written into alert_events and POSTed to the customers own Discord webhook. This is the live, sellable Pro-tier "Alerting" feature (BUSINESS_MODEL section 3.4 item 1) -- highest-priority surface, it is the actual product hook for the beachhead segment.
- drb-server-discord-bot: checked app/commands/radio.py, app/commands/trips.py -- embeds today are node status/help/trip content only, no incident transcript/summary rendering exists yet. Nothing to redact today; must inherit this design the day incident-to-Discord posting ships.
- drb-telegram-bot: app/handlers/__init__.py is a stub, no incident-surfacing code exists. Same note as above.
- Not yet built, but must inherit the design when built: CSV export, Network-tier API access (lib/apiKeys.ts is an in-memory stub per DEFERRED.md).
## 6. Out of scope for #43
- Raw-audio/live-relay exclusion of EMS talkgroups -- the ruling excludes them from the AI pipeline only, not from live audio/Discord voice relay.
- Building an accurate NER model -- a heuristic/regex redactor is scope; measuring or improving its accuracy is a follow-on issue (mirrors #48, on the redactor instead of the extractor).
- Retroactive redaction of historical calls/incidents already in Firestore (no backfill infra exists -- same unscoped-backfill pattern already logged in DEFERRED.md for _verified_pin). Tracked as a new follow-on issue at ship time, not built now.
- A UI for classifying talkgroups as EMS/medical beyond a minimal toggle reusing the existing per-system ai-flags PUT route pattern (routers/systems.py:107).
## 7. Needs a CEO/owner ruling
- Urgent -- is the comped (friends/family) tier suspended today? #43 comment 1 states suppression must hold "on every surface -- public, comped and paid," and #79 comment says no login proceeds until this ships -- but the comped tier is described in BUSINESS_MODEL section 3.2 as already live with "todays live full product," unredacted. Either comped access is in active breach of the ruling right now, or it is meant to be paused pending this ship date. My recommendation: pause comped access to incident detail/transcript views (or accept and log the breach explicitly) until #43 ships -- silently continuing is worse than either choice on record.
- How is a talkgroup classified EMS/medical? Recommend: name-pattern heuristic (reusing the existing _TG_SUFFIX_RE EMS/rescue matching in intelligence.py:101-108) as the default classification, manual override in the system editor, and default-exclude on no match rather than default-include -- a false negative here is the exact liability #43 exists to prevent.
- Does exclusion/redaction apply retroactively to already-processed calls? Recommend: prospective only for 2026-09-30; backfill is a separate follow-on issue (see section 6).
## 8. Effort estimate vs 2026-09-30
Roughly 6-10 engineering-days, agent-buildable (no human/contractor per GOALS.md), contingent on the section 7 rulings landing quickly -- they gate the design, not just the code:
- Talkgroup-flag data model + resolve_flags() wiring: ~1 day.
- Minimal EMS-classification toggle (reuse ai-flags PUT pattern): ~1-2 days.
- Redacted-copy storage split + Firestore rules change + regex/heuristic redactor + alerter.py snippet redaction + audit of all direct Firestore listeners in frontend: ~4-6 days -- this is the long pole, because section 4 means it is new code, not a wire-up of an existing field.
#48 does not block this. #43 comment 1 is explicit: the 200-call accuracy measurement "can no longer decide whether names are published, because they are suppressed regardless. It remains a Gate B condition for other reasons." Sequence independently.
+8
View File
@@ -19,6 +19,14 @@ services:
- mosquitto_data:/mosquitto/data
- mosquitto_certs:/mosquitto/certs
# c2-core takes ALL of its configuration from ./drb-c2-core/.env — there is
# deliberately no `environment:` block here. An entry in that block wins over
# env_file, so listing a key here (e.g. AGENT_SERVICE_KEY=${AGENT_SERVICE_KEY})
# would let an unset top-level .env silently blank out a value the owner had
# correctly pasted into drb-c2-core/.env. New settings go in
# drb-c2-core/.env.example and, for the VM, in
# infra/ansible/roles/deploy/templates/c2-core.env.j2 + vault.yml.
# AGENT_SERVICE_KEY (server-26#64) is configured that way.
c2-core:
image: ${REGISTRY}/c2-core:${TAG:-latest}
build: ./drb-c2-core
+19
View File
@@ -33,7 +33,26 @@ SUMMARY_INTERVAL_MINUTES=15
CORRELATION_WINDOW_HOURS=4
EMBEDDING_SIMILARITY_THRESHOLD=0.82
# Browser origins allowed to call this API cross-origin (JSON list). The only
# browser caller is the frontend's Archive page (GET /calls/search). Set this
# to the exact origin the frontend is served from — scheme + host, no path.
# Defaults to https://drb.cusano.net. A "*" entry works for local dev but is
# logged as a probable misconfiguration and never gets a credentialed response.
CORS_ORIGINS=["https://drb.cusano.net"]
# Fleet-wide token edge nodes present as X-Enrollment-Token on first boot
# (POST /nodes/enroll). Shared across every node — NOT a per-node secret.
# Generate with: openssl rand -hex 32
ENROLLMENT_TOKEN=
# Shared key the Discord bot presents to reach C2 without Firebase.
# Generate with: openssl rand -hex 32
SERVICE_KEY=
# Agent/automation key for the unattended work session's headless routes
# (GET/PUT /admin/features). DELIBERATELY a different value from SERVICE_KEY —
# reusing the bot's key would make both principals indistinguishable in
# audit_log, which is the whole point of server-26#64. Leave blank to keep the
# agent path closed; the routes still take a Firebase admin token either way.
# Generate with: openssl rand -hex 32
AGENT_SERVICE_KEY=
+6
View File
@@ -8,4 +8,10 @@ RUN pip install uv && uv pip install --system --no-cache-dir -r requirements.txt
COPY app/ ./app/
COPY tests/ ./tests/
# Stamped by CI so /health can prove WHICH build is running. A deploy that
# reports success while the old container keeps running is otherwise silent
# -- exactly how production served two-day-old code for two days.
ARG GIT_SHA=unknown
ENV GIT_SHA=$GIT_SHA
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
+118 -12
View File
@@ -45,7 +45,43 @@ class Settings(BaseSettings):
# while correlation behaviour was being tuned against rules-only output.
# Verify against https://ai.google.dev/gemini-api/docs/models before changing.
corr_cheap_model: str = "gemini-3.6-flash" # was gemini-2.0-flash (shut down)
corr_smart_model: str = "gemini-2.5-pro" # was gemini-1.5-pro (shut down)
# gemini-2.5-pro was closed to new projects by 2026-09 (every tiebreak 404'd
# in the first replay run, server-26#170); Google lists no stable Pro model,
# so the smart tier is the newest stable Flash instead.
corr_smart_model: str = "gemini-3.8-flash" # was gemini-2.5-pro, gemini-1.5-pro
# Transcript correction (server-26#36). Runs inside transcription, once per
# transcribed call above MIN_WORDS_FOR_CORRECTION, so it is priced like STT
# rather than like the correlation tier — cheap model on purpose.
transcript_correction_enabled: bool = True
transcript_correction_model: str = "gemini-3.6-flash"
# Retry Whisper once when its output is degenerate. The same clip produced a
# 56-word ten-code counting run on one attempt and real speech on the next
# (2026-08-23, call e49ea32c), so a hallucination is a coin-flip rather than
# a property of the audio, and discarding on the first bad roll threw away a
# recoverable transcript.
stt_retry_on_degenerate: bool = True
# Place verification (server-26#37). Checks the corrector's location nouns
# against the talkgroup's own anchor instead of stuffing every road in town
# into the prompt, so cost scales with location nouns rather than call volume.
place_verification_enabled: bool = True
# Raw transcript text in alert payloads (server-26#85). Default CLOSED.
# Board minutes #42 suppress person names on every surface until E&O is
# bound, and an alert webhook is the least recoverable surface there is:
# once the text is in a Discord channel we do not own it, cannot unsend
# it, and cannot audit who read it. This switch is the operator-level
# gate and is deliberately NOT reachable from the app -- the per-org
# opt-in alone would let an org owner self-serve their way to somebody
# else's PII. Both gates must be open before any snippet leaves.
alert_transcript_snippet_enabled: bool = False
place_verify_max_per_call: int = 3
# How close a candidate has to sound before it may rewrite a transcript.
# Below this, Places Text Search will confidently hand back the nearest
# business for any garbage string.
place_soundalike_min_ratio: float = 0.6
# An anchor wider than this is not stored at all. A statewide radius would
# confirm any location inside it, so the check would rubber-stamp everything
# while appearing to work — absent anchor means SKIP, never "accept anything".
area_anchor_max_radius_km: float = 60.0
summary_interval_minutes: int = 2 # how often the summary loop runs
correlation_window_hours: int = 2 # slow/location path: max hours since last call
embedding_similarity_threshold: float = 0.93 # slow-path: requires location corroboration
@@ -53,19 +89,59 @@ class Settings(BaseSettings):
embedding_cross_tg_threshold: float = 0.85 # cross-TG path: same dept + 2+ shared units
location_proximity_km: float = 0.5 # radius for location-proximity matching
geocode_max_km: float = 40.0 # reject geocode results farther than this from the node
incident_auto_resolve_minutes: int = 90 # auto-resolve after N minutes with no new calls
incident_auto_resolve_minutes: int = 90 # auto-resolve after N minutes with no new calls (major / unknown severity)
# Most jobs never say 10-8 on the air (replay of 09-22, server-26#170: ~5 of
# ~25 real incidents had an audible clear), so the quiet timer IS the close
# for most of them, and one 90-minute timer kept a lockout or a plate check
# "active" on the portal an hour after it ended. Scaled by the incident's
# severity instead, and made provisional: a timer-closed incident stays
# reopenable for incident_reopen_window_minutes, so a long quiet job that
# comes back on the air rejoins its own incident rather than splitting.
incident_auto_resolve_minutes_routine: int = 30 # routine / minor
incident_auto_resolve_minutes_moderate: int = 60
incident_reopen_window_minutes: int = 90 # since last substantive call
unit_continuity_max_idle_minutes: int = 20 # unit-continuity path: skip if incident idle > this
recorrelation_scan_minutes: int = 60 # re-examine orphaned calls ended within this window
tg_fast_path_idle_minutes: int = 90 # fast path: max minutes since incident last updated
# Dispatch channels only: tier-2 thin calls attach to a lone candidate idle < this.
# Was 10, which is long enough for the channel to have moved on to something else:
# on 2026-08-16 a "72 at Holland Station" incident absorbed a Grand Central train
# meet 9.6 min later, and a status check absorbed a records lookup at 9.7 min.
# Across that dump every correct thin attach was <= 3.4 min idle and every wrong
# one was >= 8.2, so 5 separates them with room on both sides. Genuine
# back-and-forth is handled by the 30-second tier-1 path above this.
# Tier-2 thin calls attach to a lone candidate idle < this, on every
# channel (server-26#133/#134 removed the dispatch/tactical split — a
# channel's name doesn't change how much scrutiny it gets). Was 10, which
# is long enough for the channel to have moved on to something else: on
# 2026-08-16 a "72 at Holland Station" incident absorbed a Grand Central
# train meet 9.6 min later, and a status check absorbed a records lookup
# at 9.7 min. Every correct thin attach in that dump was <= 3.4 min idle
# and every wrong one was >= 8.2, so 5 separates them with room on both
# sides. Genuine back-and-forth is handled by the 30-second tier-1 path
# above this. Also the escape hatch in routers/upload.py's LLM-orphan gate
# (_recent_incident_on_same_talkgroup, server-26#115) — check both call
# sites before retuning this.
tg_dispatch_thin_idle_minutes: int = 5
# ── Hard caps: an incident past either of these stops accepting calls ──────
# Enforced on every correlation path (see _incident_at_capacity). Pairwise fit
# tests judge one call against one incident and cannot see the shape of the
# chain they are building, so these are the only guard against a "work shift"
# incident regardless of how individually plausible each link looked.
#
# 120 minutes: the one incident in the 2026-08-20 dump that was genuinely a
# single event ran 63 minutes (06:15 wrong-way driver → 07:18 closeout), so
# the cap has to clear an hour with real headroom. The four junk chains ran
# 3h41m, 3h43m, 4h05m and 4h09m, so it has to sit well under three hours.
# 120 also equals correlation_window_hours: the location and slow paths
# already refuse to consider a candidate older than that, and the fast path
# was the only one exempt. Making it agree removes that inconsistency rather
# than inventing a new number.
incident_max_duration_minutes: int = 120
# 40 calls: a backstop for a burst that fills up inside the duration cap
# rather than the primary bound. The worst observed chain averaged ~16
# calls/hour while absorbing an ENTIRE dispatch backbone, so 40 calls in
# under two hours means the incident is eating most of the channel — that is
# a chain, not an event. Set deliberately above any plausible single-incident
# call volume (a multi-alarm fire on its own tactical channel) so this cap
# errs toward keeping real incidents whole and lets the duration cap do the
# cutting.
incident_max_calls: int = 40
# Vocabulary learning
vocabulary_induction_interval_hours: int = 24 # how often the induction loop runs
vocabulary_induction_sample_tokens: int = 4000 # ~tokens of transcript text sampled per system
@@ -73,6 +149,21 @@ class Settings(BaseSettings):
# Internal service key — allows server-side services (discord bot) to call C2 without Firebase
service_key: Optional[str] = None
# Automation/agent service key — the unattended work-session agent's own
# credential for the headless routes it needs (currently GET/PUT
# /admin/features).
#
# DELIBERATELY SEPARATE from service_key above, not a second consumer of
# it. service_key is the Discord bot's, and it is handed to a process that
# relays radio traffic to a chat server; sharing it here would make "the
# bot" and "the agent" the same principal in every log line and audit
# entry, so a global AI-cost flag flip could never be attributed to whoever
# actually made it. Two keys, two identities (server-26#64 item 1).
#
# Unset means the agent path is simply closed — the routes still accept a
# Firebase admin token. Generate with: openssl rand -hex 32
agent_service_key: Optional[str] = None
# Fleet-wide token edge nodes present to POST /nodes/enroll on first boot.
# Not a per-node secret — see routers/enrollment.py for why a leaked copy
# of this alone can't steal an already-approved node's key.
@@ -97,9 +188,24 @@ class Settings(BaseSettings):
# between genuinely separate transmissions on a busy dispatch channel.
duplicate_window_seconds: int = 10
# CORS — set to your frontend origin(s) in production, e.g. ["https://app.example.com"]
# Defaults to "*" for local development only.
cors_origins: list[str] = ["*"]
# Browser origins allowed to call this API cross-origin. The only browser
# caller is the frontend's Archive page (GET /calls/search) — every other
# page reads Firestore directly. The frontend is served on the BARE domain
# (see infra Caddyfile.j2 — only drb. and api. have DNS records), so the
# default is that origin, not app.<domain>. Override via CORS_ORIGINS (JSON
# list) if the frontend ever moves; keep infra/.../c2-core.env.j2 in sync.
#
# A "*" entry here still works for local dev but is refused a credentialed
# response: main.py never enables allow_credentials (auth is a Bearer
# header, not a cookie), and it logs a loud ERROR when it sees a wildcard
# in a deployment so a forgotten override is visible.
cors_origins: list[str] = ["https://drb.cusano.net"]
# Discord webhook URL that app/internal/ai_health.py posts to when an AI
# tier (transcription/correlation) transitions into or out of degraded
# state. Empty disables the POST entirely — not every self-hosted
# deployment will set this up, and skipping it must be silent.
ai_alert_webhook_url: str = ""
class Config:
env_file = ".env"
+214
View File
@@ -0,0 +1,214 @@
"""
Shared AI-provider degradation registry.
On the night of 2026-08-18 three independent AI dependency failures (a
retired Gemini model ID, a depleted Gemini balance, an unpayable OpenAI
account) each surfaced only as a single ERROR log line -- and nobody reads
container logs continuously. This module is the fix: every AI call site
reports its outcome here instead of (or in addition to) just logging, so the
current state of every AI tier can be read back over HTTP (see
app/main.py's /health/ai) and pushed out to Discord on state changes.
Tiers are tracked independently and in memory only (module-level singleton,
no Firestore/DI -- consistent with the rest of this codebase). State is lost
on restart, which is fine: a fresh process should re-derive degradation from
the next few calls rather than resurrect a possibly-stale alert.
The load-bearing distinction, from the incident this module exists to
prevent: a PERMANENT condition (retired model, dead billing account, bad API
key) will never clear on its own and must alert on the very first
occurrence. A TRANSIENT condition (rate limit, network blip) clears by
itself constantly and must NOT page anyone for the first failure -- only if
it persists. classify() is the one place that tells the two apart from a
provider error message, because both this module's callers (llm_correlator.py,
transcription.py) need the exact same judgment call and must not each grow
their own slightly-different copy that drifts.
"""
import asyncio
from contextvars import ContextVar
from datetime import datetime, timezone
from typing import Optional
from app.internal.logger import logger
from app.config import settings
TIERS = ("transcription", "correlation_cheap", "correlation_smart", "extraction")
# Consecutive failures a TRANSIENT condition must reach before it alerts.
# Permanent conditions skip this entirely and alert on failure #1.
TRANSIENT_ALERT_THRESHOLD = 5
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _default_state() -> dict:
return {
"degraded": False,
"permanent": False,
"provider": None,
"model": None,
"problem": None,
"fix": None,
"first_seen": None,
"last_seen": None,
"consecutive_failures": 0,
"alerted": False,
}
_state: dict[str, dict] = {t: _default_state() for t in TIERS}
# Set by a replay run to a list it owns; report_degraded appends there instead
# of touching _state while inside a sandbox (see app/internal/replay.py).
_sandbox_failures: ContextVar[Optional[list]] = ContextVar("drb_ai_sandbox_failures", default=None)
def collect_sandbox_failures(sink: Optional[list]):
return _sandbox_failures.set(sink)
def classify(text: str) -> str:
"""
Classify a provider failure message body.
Returns "dead_model", "billing", or "transient".
A depleted balance and an ordinary rate limit both arrive as HTTP 429 --
the status code can't tell them apart, only the message body can. This
logic previously lived independently in llm_correlator.py and (in a
slightly different shape) transcription.py; it now lives here once, and
both call in rather than re-matching the text themselves.
"""
low = text.lower()
if "404" in text or "not found" in low or "no longer available" in low:
return "dead_model"
if (
"credits are depleted" in low
or "prepayment" in low
or "billing" in low
or "insufficient_quota" in low
or "credit" in low
or "exceeded your current quota" in low
):
return "billing"
return "transient"
async def report_degraded(
tier: str,
provider: str,
model: str,
problem: str,
fix: str,
permanent: bool = False,
) -> None:
"""
Record a failure for `tier`. Call this from a failure path, once per
failure (it does its own once-per-episode alert suppression -- do not
gate the call site on that yourself).
permanent=True (dead model, unpayable account, bad key) alerts on this
very call. permanent=False (rate limit, network blip) only alerts once
TRANSIENT_ALERT_THRESHOLD consecutive failures have been reported for
this tier, so an ordinary blip never pages anyone.
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
# A replay's rate limits are not a live outage, and must never page
# the AI-alert webhook or flip /health/ai (app/internal/replay.py).
# They are the run's own problem, so they go to the run instead.
sink = _sandbox_failures.get()
if sink is not None:
sink.append({"tier": tier, "provider": provider, "model": model,
"problem": problem, "permanent": permanent})
return
if tier not in _state:
_state[tier] = _default_state()
entry = _state[tier]
now = _now()
if entry["consecutive_failures"] == 0:
entry["first_seen"] = now
entry["last_seen"] = now
entry["consecutive_failures"] += 1
entry["provider"] = provider
entry["model"] = model
entry["problem"] = problem
entry["fix"] = fix
entry["permanent"] = permanent
should_alert_now = permanent or entry["consecutive_failures"] >= TRANSIENT_ALERT_THRESHOLD
if should_alert_now and not entry["degraded"]:
entry["degraded"] = True
if should_alert_now and not entry["alerted"]:
entry["alerted"] = True
await _post_webhook(
f"**AI tier degraded: {tier}**\n"
f"Provider: {provider} ({model})\n"
f"Problem: {problem}\n"
f"Fix: {fix}\n"
f"Kind: {'permanent' if permanent else 'transient, persisted ' + str(entry['consecutive_failures']) + ' calls'}"
)
async def report_healthy(tier: str) -> None:
"""
Record a successful call for `tier`. Call this on every success, not
just after a failure -- it is what lets a degraded tier recover on its
own instead of staying red forever after one transient blip.
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
return # nor may a replay's success "recover" a real live outage
if tier not in _state:
_state[tier] = _default_state()
entry = _state[tier]
was_alerted = entry["alerted"]
was_degraded = entry["degraded"]
provider, model = entry["provider"], entry["model"]
_state[tier] = _default_state()
# Keep the last-known provider/model around for the recovery message
# and for a quick glance at snapshot() even when healthy.
_state[tier]["provider"] = provider
_state[tier]["model"] = model
if was_alerted:
await _post_webhook(f"**AI tier recovered: {tier}**\nProvider: {provider} ({model})")
elif was_degraded:
# Reached "degraded" internally but never crossed the alert
# threshold before recovering -- nothing was ever posted, so
# nothing needs un-posting. Nothing to do.
pass
def snapshot() -> dict:
"""Current state of every tier, for /health/ai."""
return {tier: dict(entry) for tier, entry in _state.items()}
async def _post_webhook(content: str) -> None:
"""
POST a message to the AI-alert Discord webhook, if one is configured.
Same httpx pattern as app/internal/alerter.py's _post_webhook: short
timeout, never raises. Self-hosted deployments that don't set
ai_alert_webhook_url just skip this silently.
"""
url = settings.ai_alert_webhook_url
if not url:
return
try:
import httpx
async with httpx.AsyncClient(timeout=5.0) as client:
await client.post(url, json={"content": content})
except Exception as e:
logger.warning(f"ai_health: Discord webhook POST failed: {e}")
+46 -1
View File
@@ -6,11 +6,15 @@ talkgroup ID, tags, and transcript. On a match:
1. Creates an AlertEvent document in Firestore.
2. Optionally POSTs a Discord webhook message if the rule has one configured.
Raw transcript text is withheld from both by default -- see _snippet_allowed
and server-26#85.
Never raises — failures are logged as warnings so the pipeline always completes.
"""
import uuid
from datetime import datetime, timezone
from typing import Optional
from app.config import settings
from app.internal.logger import logger
from app.internal import firestore as fstore
@@ -47,13 +51,17 @@ async def check_and_dispatch(
logger.warning(f"Alerter: could not load rules: {e}")
return
# Loop-invariant: every rule here belongs to the same org, so the opt-in is
# resolved once rather than per match.
snippet_allowed = await _snippet_allowed(org_id)
for rule in rules:
matched_keywords = _match_rule(rule, talkgroup_id, tags, transcript)
if not matched_keywords:
continue
alert_id = str(uuid.uuid4())
snippet = _snippet(transcript)
snippet = _snippet(transcript) if snippet_allowed else None
now = datetime.now(timezone.utc).isoformat()
event = {
"alert_id": alert_id,
@@ -85,6 +93,43 @@ async def check_and_dispatch(
await _post_webhook(webhook_url, rule.get("name", ""), talkgroup_name, matched_keywords, snippet)
async def _snippet_allowed(org_id: Optional[str]) -> bool:
"""
Whether raw transcript text may be attached to an alert (server-26#85).
Two gates, both of which must be open:
1. ``settings.alert_transcript_snippet_enabled`` -- the operator switch,
default False, set from the environment and unreachable from the app.
2. ``alert_snippet_opt_in`` on the org document -- the customer's own
explicit, contractual opt-in.
Gate 1 exists because gate 2 alone is not a real control: the frontend
reads and (per the Firestore rules, not ``auth.py``) can write org state
directly from the browser, so an org owner could otherwise opt themselves
into receiving person names lifted from live public-safety traffic. Board
minutes #42 suppress names on every surface until E&O is bound.
Fails CLOSED on any error, and on a call with no org (a pre-tenancy node
that has not been backfilled), because the cost of wrongly withholding a
snippet is a less informative alert and the cost of wrongly emitting one
is unrecallable disclosure to a third party.
"""
if not settings.alert_transcript_snippet_enabled:
return False
if not org_id:
return False
try:
org = await fstore.doc_get("organizations", org_id)
except Exception as e:
logger.warning(
f"Alerter: could not read snippet opt-in for org={org_id}, "
f"withholding transcript: {e}"
)
return False
return bool((org or {}).get("alert_snippet_opt_in"))
def _match_rule(
rule: dict,
talkgroup_id: Optional[int],
+546
View File
@@ -0,0 +1,546 @@
"""
Area context — the ground truth an operator sets about where a channel operates.
One shape, used at two scopes (server-26#36):
area_context: {
municipality?, county?, state?,
center?, radius_km?, resolved_from?, resolved_at?, # backend-written
local_knowledge?: [ { term, meaning } ]
}
WHY EVERY FIELD IS NULLABLE. The system level is only meaningful when it is true
of *every* talkgroup on that system. White Plains PD — it is, so an operator
fills it once and every talkgroup inherits. A statewide Colorado system — it is
not, so they leave it null and fill each talkgroup. Which level someone fills IS
their declaration of how homogeneous the system is, which is what lets one
schema serve both without a `system_type` flag to get out of sync.
WHY `local_knowledge` REPLACED `roads[]`/`landmarks[]`. Radio traffic references
intersections, schools, housing developments, rail stations and nicknames ("the
flats"), none of which fit two lists. And a bare term is half the information:
`11-X-ray` tells a corrector nothing, `11-X-ray — MTA PD patrol unit` is what
lets it recognise the sound.
WHY THE ANCHOR CAN BE ABSENT ON PURPOSE. `center`/`radius_km` exist so a
geocoded place name can be sanity-checked against the area the channel actually
covers (server-26#37). If municipality/county/state only resolve to something as
wide as a state, that check would confirm anything inside it while appearing to
work — worse than useless. So an anchor wider than
`settings.area_anchor_max_radius_km` is not written at all, and an absent anchor
means SKIP THE CHECK, never "accept anything".
THE CLIENT DOES NOT WRITE THE DERIVED FIELDS. `center`, `radius_km`,
`resolved_from` and `resolved_at` are computed here and merged in by the server.
Taking them from the request body is the same defect as the `ten_codes` wipe
fixed in 58efdbd: the frontend does not decide what is in a system document.
"""
import asyncio
import math
from datetime import datetime, timezone
from typing import Any, Optional
from app.config import settings
from app.internal.logger import logger
# Fields an operator sets. Anything else in an incoming body is dropped.
CLIENT_FIELDS = ("municipality", "county", "state", "local_knowledge")
# Fields this module owns. Carried forward from the stored document on every
# write, never read from the request.
SERVER_FIELDS = ("center", "radius_km", "resolved_from", "resolved_at")
# The three that identify a place, in the order they are geocoded.
PLACE_FIELDS = ("municipality", "county", "state")
_anchor_cache: dict[str, Optional[dict]] = {}
def geo_dist_km(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
"""Haversine distance in km between two lat/lon points."""
R = 6371.0
dlat = math.radians(lat2 - lat1)
dlon = math.radians(lon2 - lon1)
a = (
math.sin(dlat / 2) ** 2
+ math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * math.sin(dlon / 2) ** 2
)
return R * 2 * math.asin(math.sqrt(a))
# -- Normalisation -------------------------------------------------------------
def normalize_local_knowledge(raw: Any) -> list[dict]:
"""
Coerce whatever arrived into [{term, meaning}], dropping junk.
Accepts a bare string list too — that is what `roads[]`/`landmarks[]` and the
old flat `vocabulary` look like, and a term with no meaning is still worth
having in the reference list.
"""
if not isinstance(raw, list):
return []
out: list[dict] = []
seen: set[str] = set()
for item in raw:
if isinstance(item, str):
term, meaning = item.strip(), None
elif isinstance(item, dict):
term = str(item.get("term") or "").strip()
meaning = str(item.get("meaning") or "").strip() or None
else:
continue
key = term.lower()
if not term or key in seen:
continue
seen.add(key)
out.append({"term": term, "meaning": meaning} if meaning else {"term": term})
return out
def knowledge_of(area: Optional[dict]) -> list[dict]:
"""
This scope's local knowledge, folding the pre-#36 shape forward.
`roads[]` and `landmarks[]` were the original fields and real systems still
have them stored. Reading them as bare terms means the corrector keeps the
ground truth an operator already entered instead of silently losing it the
day this shipped; they disappear from the document the next time that scope
is saved.
"""
area = area or {}
legacy = list(area.get("roads") or []) + list(area.get("landmarks") or [])
return normalize_local_knowledge(list(area.get("local_knowledge") or []) + legacy)
def normalize(raw: Any) -> dict:
"""Client-supplied area_context -> the stored shape, server fields excluded."""
if not isinstance(raw, dict):
return {}
out: dict[str, Any] = {}
for field in PLACE_FIELDS:
value = raw.get(field)
if isinstance(value, str) and value.strip():
out[field] = value.strip()
knowledge = knowledge_of(raw)
if knowledge:
out["local_knowledge"] = knowledge
return out
def merge_server_fields(incoming: dict, existing: Optional[dict]) -> dict:
"""Carry the backend-owned anchor forward across a client write."""
out = dict(incoming)
for field in SERVER_FIELDS:
if existing and existing.get(field) is not None:
out[field] = existing[field]
return out
def merge_config(incoming: Any, existing: Optional[dict]) -> Any:
"""
Reconcile a client-sent config blob with what the server already owns.
The systems form sends `config.talkgroups[]` in full, so writing it verbatim
destroys everything the backend put there — the resolved anchor and the
pending term queue. That is the same defect as the `ten_codes` wipe fixed in
58efdbd, and the same fix applies: the backend merges its own fields back in
rather than taking dictation from the frontend.
"""
if not isinstance(incoming, dict):
return incoming
incoming_tgs = incoming.get("talkgroups")
if not isinstance(incoming_tgs, list):
return incoming
by_id: dict[int, dict] = {}
for tg in ((existing or {}).get("talkgroups") or []):
if isinstance(tg, dict):
try:
by_id[int(tg.get("id", -1))] = tg
except (TypeError, ValueError):
continue
merged: list[Any] = []
for tg in incoming_tgs:
if not isinstance(tg, dict):
merged.append(tg)
continue
try:
prior = by_id.get(int(tg.get("id", -1))) or {}
except (TypeError, ValueError):
prior = {}
out = dict(tg)
area = normalize(tg.get("area_context"))
prior_area = prior.get("area_context") or {}
if area:
out["area_context"] = merge_server_fields(area, prior_area)
else:
out.pop("area_context", None)
if prior.get(PENDING_KEY):
out[PENDING_KEY] = prior[PENDING_KEY]
merged.append(out)
return {**incoming, "talkgroups": merged}
# -- Scope resolution ----------------------------------------------------------
def effective(system_area: Optional[dict], tg_area: Optional[dict]) -> dict:
"""
Merge the two scopes: talkgroup wins where set, system fills the gaps.
`local_knowledge` concatenates rather than replaces, talkgroup entries first
so they survive any downstream truncation and outrank a system entry for the
same term. A multi-county system whose talkgroup covers one municipality must
not have that municipality's terms buried under a county-wide list.
"""
system_area = system_area or {}
tg_area = tg_area or {}
out: dict[str, Any] = {}
for field in PLACE_FIELDS:
value = tg_area.get(field) or system_area.get(field)
if value:
out[field] = value
knowledge = normalize_local_knowledge(knowledge_of(tg_area) + knowledge_of(system_area))
if knowledge:
out["local_knowledge"] = knowledge
return out
def talkgroup_entry(system_doc: Optional[dict], talkgroup_id: Any) -> dict:
"""The `config.talkgroups[]` entry for this talkgroup, or `{}`."""
if not system_doc or talkgroup_id is None:
return {}
talkgroups = (system_doc.get("config") or {}).get("talkgroups") or []
idx = _tg_index(talkgroups, talkgroup_id)
return talkgroups[idx] if idx >= 0 else {}
def anchor_key(area: Optional[dict]) -> str:
"""
Stable identity of the place an anchor was resolved from.
Stored as `resolved_from`, which is what makes "did this actually change?"
decidable — so the geocode happens when someone edits a town name, not on
every read or every five minutes.
"""
area = area or {}
return "|".join((area.get(f) or "").strip().lower() for f in PLACE_FIELDS)
def has_place(area: Optional[dict]) -> bool:
return bool(anchor_key(area).strip("|"))
def anchor_for(system_area: Optional[dict], tg_area: Optional[dict]) -> Optional[dict]:
"""
The anchor to sanity-check geocoded locations against, or None.
None has one meaning and it is load-bearing: SKIP THE CHECK. It covers an
unconfigured system, an area too wide to discriminate, and a stored anchor
whose `resolved_from` no longer matches the place it was computed for (an
edit landed and the refresh has not run). Accepting a stale or oversized
anchor would rubber-stamp locations while looking like verification.
"""
key = anchor_key(effective(system_area, tg_area))
for area in (tg_area, system_area):
if not area:
continue
center, radius = area.get("center"), area.get("radius_km")
if area.get("resolved_from") == key and center and radius:
try:
return {
"lat": float(center["lat"]),
"lng": float(center["lng"]),
"radius_km": float(radius),
}
except (KeyError, TypeError, ValueError):
continue
return None
# -- Anchor geocoding ----------------------------------------------------------
def _query(area: dict) -> str:
return ", ".join(area[f] for f in PLACE_FIELDS if area.get(f))
async def resolve_anchor(area: dict) -> Optional[dict]:
"""
Geocode municipality/county/state into {center, radius_km}, or None.
The radius comes from the result's own viewport — half its diagonal — so a
village anchors tightly and a county loosely, which is the real difference
we care about. Anything wider than `area_anchor_max_radius_km` is discarded
rather than stored: see the module docstring.
"""
if not has_place(area):
return None
query = _query(area)
if query in _anchor_cache:
return _anchor_cache[query]
if not settings.google_maps_api_key:
logger.warning("GOOGLE_MAPS_API_KEY not set — area anchors cannot be resolved")
return None
import httpx
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(
"https://maps.googleapis.com/maps/api/geocode/json",
params={"address": query, "region": "us", "key": settings.google_maps_api_key},
)
r.raise_for_status()
data = r.json()
if data.get("status") != "OK" or not data.get("results"):
logger.warning(f"Area anchor: {query!r} did not geocode ({data.get('status')})")
_anchor_cache[query] = None
return None
geometry = data["results"][0].get("geometry") or {}
loc = geometry.get("location") or {}
lat, lng = float(loc["lat"]), float(loc["lng"])
viewport = geometry.get("viewport") or {}
ne, sw = viewport.get("northeast"), viewport.get("southwest")
if ne and sw:
radius_km = geo_dist_km(sw["lat"], sw["lng"], ne["lat"], ne["lng"]) / 2
else:
radius_km = settings.geocode_max_km
except Exception as e:
logger.warning(f"Area anchor geocoding failed for {query!r}: {e}")
return None # not cached — a transient failure should be retried
if radius_km > settings.area_anchor_max_radius_km:
logger.info(
f"Area anchor: {query!r} spans ~{radius_km:.0f}km, wider than "
f"area_anchor_max_radius_km={settings.area_anchor_max_radius_km} — storing no "
f"anchor, so verification skips rather than rubber-stamps"
)
_anchor_cache[query] = None
return None
anchor = {
"center": {"lat": lat, "lng": lng},
"radius_km": round(radius_km, 2),
"resolved_from": anchor_key(area),
"resolved_at": datetime.now(timezone.utc).isoformat(),
}
_anchor_cache[query] = anchor
logger.info(f"Area anchor: {query!r} -> ({lat:.4f}, {lng:.4f}) r={radius_km:.1f}km")
return anchor
def _apply(area: dict, anchor: Optional[dict], key: str) -> dict:
"""Write (or clear) the derived fields on one scope's area_context."""
out = {k: v for k, v in area.items() if k not in SERVER_FIELDS}
if anchor:
out.update(anchor)
elif key.strip("|"):
# A place is set but produced no usable anchor. Record that we tried, so
# the next write does not geocode it again for the same answer.
out["resolved_from"] = key
out["resolved_at"] = datetime.now(timezone.utc).isoformat()
return out
async def refresh_anchors(system_doc: dict) -> dict:
"""
Recompute anchors for a system and every talkgroup that sets a place.
Returns a Firestore patch — `{}` when nothing needed resolving. Talkgroups
are refreshed alongside the system because a talkgroup's anchor is derived
from its EFFECTIVE place (its own fields over the system's), so editing the
system's county silently changes what every talkgroup should be anchored to.
Only scopes whose `resolved_from` no longer matches are geocoded, and the
per-query cache means N talkgroups in one town cost one request.
"""
system_area = dict(system_doc.get("area_context") or {})
patch: dict[str, Any] = {}
system_key = anchor_key(system_area)
if system_area.get("resolved_from") != system_key:
anchor = await resolve_anchor(system_area) if has_place(system_area) else None
patch["area_context"] = _apply(system_area, anchor, system_key)
system_area = patch["area_context"]
config = system_doc.get("config") or {}
talkgroups = config.get("talkgroups")
if not isinstance(talkgroups, list):
return patch
updated: list[dict] = []
changed = False
for tg in talkgroups:
if not isinstance(tg, dict):
updated.append(tg)
continue
tg_area = tg.get("area_context") or {}
# No place of its own means it inherits the system's anchor wholesale —
# nothing to store here, and anchor_for() falls back to the system.
if not has_place(tg_area):
if any(tg_area.get(f) is not None for f in SERVER_FIELDS):
tg = {**tg, "area_context": {k: v for k, v in tg_area.items() if k not in SERVER_FIELDS}}
changed = True
updated.append(tg)
continue
key = anchor_key(effective(system_area, tg_area))
if tg_area.get("resolved_from") == key:
updated.append(tg)
continue
anchor = await resolve_anchor(effective(system_area, tg_area))
updated.append({**tg, "area_context": _apply(tg_area, anchor, key)})
changed = True
if changed:
patch["config"] = {**config, "talkgroups": updated}
return patch
# -- Talkgroup-level pending terms ---------------------------------------------
#
# Proposals land on the TALKGROUP and are never promoted to the system
# automatically (server-26#37). The argument is blast radius: a wrong term on a
# talkgroup misleads one channel, the same term at system level misleads every
# channel on that system — including one 400km away on a statewide system, which
# is exactly the context poisoning the scope rule exists to prevent. If a term
# genuinely applies system-wide, carrying it on several talkgroups costs almost
# nothing; auto-promoting a wrong one is expensive to notice.
PENDING_KEY = "local_knowledge_pending"
def _tg_index(talkgroups: list, talkgroup_id: Any) -> int:
try:
wanted = int(talkgroup_id)
except (TypeError, ValueError):
return -1
for i, tg in enumerate(talkgroups):
if not isinstance(tg, dict):
continue
try:
if int(tg.get("id", -1)) == wanted:
return i
except (TypeError, ValueError):
continue
return -1
def _known_terms(tg: dict, system_doc: dict) -> set[str]:
"""Everything this talkgroup already knows, at either scope, plus pending."""
known = {
e["term"].lower()
for e in effective(system_doc.get("area_context"), tg.get("area_context"))
.get("local_knowledge", [])
}
known |= {str(t).lower() for t in (tg.get("vocabulary") or [])}
known |= {str(t).lower() for t in (system_doc.get("vocabulary") or [])}
known |= {str(p.get("term", "")).lower() for p in (tg.get(PENDING_KEY) or [])}
return known
async def add_pending(system_id: str, talkgroup_id: Any, entries: list[dict]) -> int:
"""
Queue proposed {term, meaning} entries on one talkgroup for human review.
Returns how many were actually queued. Nothing here writes to
`local_knowledge` — approval is a person's decision, always.
"""
from app.internal import firestore as fstore
if fstore.in_sandbox():
return 0 # a replay proposes nothing to the live review queue
if not system_id or talkgroup_id is None or not entries:
return 0
system_doc = await fstore.doc_get("systems", system_id)
if not system_doc:
return 0
config = dict(system_doc.get("config") or {})
talkgroups = list(config.get("talkgroups") or [])
idx = _tg_index(talkgroups, talkgroup_id)
if idx < 0:
return 0
tg = dict(talkgroups[idx])
known = _known_terms(tg, system_doc)
now = datetime.now(timezone.utc).isoformat()
queued: list[dict] = []
for entry in entries:
term = str(entry.get("term") or "").strip()
if not term or term.lower() in known:
continue
known.add(term.lower())
queued.append({
"term": term,
"meaning": entry.get("meaning") or None,
"source": entry.get("source") or "verifier",
"added_at": now,
"source_call_ids": entry.get("source_call_ids") or [],
})
if not queued:
return 0
tg[PENDING_KEY] = list(tg.get(PENDING_KEY) or []) + queued
talkgroups[idx] = tg
config["talkgroups"] = talkgroups
await fstore.doc_update("systems", system_id, {"config": config})
logger.info(
f"Local knowledge: {len(queued)} term(s) proposed for talkgroup "
f"{talkgroup_id} on system {system_id}: {[q['term'] for q in queued]}"
)
return len(queued)
async def resolve_pending(system_id: str, talkgroup_id: Any, term: str, approve: bool) -> bool:
"""Approve a pending term onto the talkgroup, or dismiss it. Never promotes."""
from app.internal import firestore as fstore
system_doc = await fstore.doc_get("systems", system_id)
if not system_doc:
return False
config = dict(system_doc.get("config") or {})
talkgroups = list(config.get("talkgroups") or [])
idx = _tg_index(talkgroups, talkgroup_id)
if idx < 0:
return False
tg = dict(talkgroups[idx])
pending = list(tg.get(PENDING_KEY) or [])
match = next((p for p in pending if str(p.get("term", "")).lower() == term.lower()), None)
if match is None:
return False
tg[PENDING_KEY] = [p for p in pending if p is not match]
if approve:
area = dict(tg.get("area_context") or {})
area["local_knowledge"] = normalize_local_knowledge(
list(area.get("local_knowledge") or [])
+ [{"term": match["term"], "meaning": match.get("meaning")}]
)
tg["area_context"] = area
talkgroups[idx] = tg
config["talkgroups"] = talkgroups
await fstore.doc_update("systems", system_id, {"config": config})
return True
async def refresh_anchors_bg(system_id: str) -> None:
"""Fire-and-forget refresh, for callers that must not block on Maps."""
from app.internal import firestore as fstore
try:
doc = await fstore.doc_get("systems", system_id)
if not doc:
return
patch = await refresh_anchors(doc)
if patch:
await fstore.doc_update("systems", system_id, patch)
except Exception as e:
logger.warning(f"Area anchor refresh failed for system {system_id}: {e}")
def schedule_refresh(system_id: str) -> None:
"""Kick a refresh without making the caller wait for the geocoder."""
try:
asyncio.get_running_loop().create_task(refresh_anchors_bg(system_id))
except RuntimeError: # no loop (tests, scripts) — nothing to schedule
pass
+68
View File
@@ -220,6 +220,74 @@ async def require_service_key_or_admin(
return decoded
# ---------------------------------------------------------------------------
# Automation / agent principal
# ---------------------------------------------------------------------------
# Identity written into audit_log when the agent key is what authenticated a
# request. A Firebase admin gets their own uid/email instead, so the two are
# always distinguishable after the fact — which is the point.
AGENT_PRINCIPAL_UID = "agent-service"
AGENT_PRINCIPAL_EMAIL = "agent-service@drb.internal"
async def require_agent_key_or_admin(
credentials: Optional[HTTPAuthorizationCredentials] = Security(_bearer),
) -> dict:
"""Accept either the agent service key or a Firebase admin token.
Deliberately does NOT accept ``settings.service_key``. That key belongs to
the Discord bot, and honouring it here would collapse two principals into
one unattributable identity in every log line and audit entry — the exact
thing server-26#64 exists to end. The bot has no business flipping
platform-wide AI flags either way.
Exists so the unattended runbook can flip AI flags over HTTP instead of
SSHing into the container and writing ``config/ai_features`` with the admin
SDK, which needs a full container shell to move a cost switch.
The ``settings.agent_service_key and ...`` guard is load-bearing, not
stylistic: ``secrets.compare_digest("", "")`` is a MATCH, so any form of
``compare_digest(token, settings.agent_service_key or "")`` would turn a
deployment that never configured the key into one that accepts an empty
credential. Check the key is configured first and never substitute a
placeholder. (``require_service_key`` states the same intent by raising
503 when unset; both are correct, this one just stays open to admins.)
"""
if not credentials:
raise HTTPException(status_code=401, detail="Missing authorization token")
token = credentials.credentials
if settings.agent_service_key and secrets.compare_digest(token, settings.agent_service_key):
return {
"service": True,
"principal": "agent",
"uid": AGENT_PRINCIPAL_UID,
"email": AGENT_PRINCIPAL_EMAIL,
}
try:
decoded = firebase_auth.verify_id_token(token)
except Exception:
raise HTTPException(status_code=401, detail="Invalid or expired token")
if get_role(decoded) != "admin":
raise HTTPException(status_code=403, detail="Admin access required")
return decoded
def describe_actor(principal: dict) -> tuple[str, str]:
"""Return ``(actor_uid, actor_email)`` for an audit entry.
Works for any credential shape the dependencies above produce, so an audit
call site never has to switch on principal type itself.
"""
if principal.get("principal") == "agent":
return AGENT_PRINCIPAL_UID, AGENT_PRINCIPAL_EMAIL
if principal.get("service"):
return "service", "service@drb.internal"
if principal.get("node"):
node_id = principal.get("node_id") or "unknown"
return f"node:{node_id}", ""
return principal.get("uid") or "unknown", principal.get("email") or ""
# ---------------------------------------------------------------------------
# Simple in-memory sliding-window rate limiter
# ---------------------------------------------------------------------------
@@ -0,0 +1,135 @@
"""
Upstream dispatch-vs-chatter classifier — SHADOW MODE (server-26#115 follow-up).
Three live measurement windows (CORRELATION_REVIEW_0907.md, _0907b.md, _0912.md)
and two consensus-layer fixes (#125, #126) all converged on the same conclusion:
the actual non-event-promotion problem lives upstream of correlation entirely.
Radio housekeeping — unit check-ins, roll call, bare 10-4/10-8/98 acknowledgements
— has no incident content for `intelligence.extract_scenes` to find, but nothing
stops it from being sent to the scene-extraction LLM and coming out the other end
as a thin "scene" for the correlator to then judge. See CORRELATION_REVIEW_0912.md
("Reminder: the real fix is still unscoped") and issue #115.
This module is that classifier. It is a PURE function of the transcript text —
no Firestore, no LLM call, no side effects — so it is cheap to run on every
transcript and cheap to test against real dumps offline.
SHADOW MODE ONLY. As of this module's introduction, nothing skips scene
extraction based on this verdict. `intelligence.extract_scenes` calls
`classify_chatter` purely to record the verdict on the call doc
(`chatter_classifier_verdict` / `chatter_classifier_reason`) so it becomes
observable in the next `/admin` correlation-debug dump, exactly like
`corr_gate_veto` (server-26#115 / PR #126). See the TODO at that call site for
what has to be true before this flips live.
Precision over recall, deliberately. A false positive here — flagging a REAL
event as chatter — would, once live, silently mean that event never gets a
scene, never gets tags/location/severity, and never has a chance to become an
incident. That is a much bigger, harder-to-notice failure than a false
negative (a housekeeping call that still goes through the existing expensive
pipeline and gets judged "not an incident" the same way it is today). When a
transcript doesn't clearly match one of the shapes below, this returns
(False, None) and the existing pipeline runs exactly as it does today.
Patterns are drawn from hand-labeled examples in CORRELATION_REVIEW_0907b.md
and CORRELATION_REVIEW_0912.md, cross-referenced against the real transcripts
in corr_dump_9-7_0437am.json / corr_dump_9-7_pm.json / corr_dump_9-12.json —
not invented regexes. See the backtest script referenced in the PR for the
per-dump catch rate and false-positive count.
"""
import re
from typing import Optional
# Police/law-enforcement phonetic alphabet words (APCO + NATO). Deliberately
# duplicated from intelligence.py's `_PHONETIC_ALPHA_WORDS` rather than
# imported — intelligence.py imports this module (to write the shadow-mode
# verdict onto the call doc), so importing back would be circular. Keep the
# two sets in sync if either changes; they're small and rarely touched.
_PHONETIC_ALPHA_WORDS = frozenset({
# APCO (law enforcement)
"adam", "baker", "charles", "david", "edward", "frank", "george", "henry",
"ida", "john", "king", "lincoln", "mary", "nora", "ocean", "paul", "queen",
"robert", "sam", "tom", "union", "victor", "william", "x-ray", "young", "zebra",
# NATO
"alpha", "bravo", "charlie", "delta", "echo", "foxtrot", "golf", "hotel",
"india", "juliet", "kilo", "lima", "mike", "november", "oscar", "papa",
"quebec", "romeo", "sierra", "tango", "uniform", "whiskey", "yankee", "zulu",
})
_TOKEN_RE = re.compile(r"[a-z0-9][a-z0-9\-]*")
# Bare radio-procedure words that carry zero incident content by themselves.
# Deliberately small and literal — this is not a general stopword list, it's
# the exact vocabulary observed in hand-labeled chatter transcripts. Words
# that are ambiguous outside a pure-procedure context (e.g. "location",
# "call", "phone", "number", "go") are left OUT on purpose: including them
# risks reducing a real, substantive transcript down to nothing.
_FILLER_WORDS = frozenset({
"to", "this", "is", "the", "a", "and", "for", "you", "can", "i", "in",
"on", "of", "that", "just", "from", "out", "ok", "okay", "at", "be",
"show", "me", "mark", "marked", "charge", "standby", "stand", "by",
"clear", "available", "affirm", "affirmative", "negative", "copy",
"copies", "received", "roger",
})
# Agency/procedural designators — who's being addressed, not what happened.
_RADIO_DESIGNATORS = frozenset({
"central", "dispatch", "headquarters", "hq", "post", "unit", "sergeant",
"sgt", "metro", "mta", "division", "county",
})
_ROLL_CALL_RE = re.compile(r"\broll\s*call\b")
def _tokenize(transcript: str) -> list[str]:
return _TOKEN_RE.findall(transcript.lower())
def _is_filler_token(token: str) -> bool:
# Any token starting with a digit is a unit ID, 10-code, badge/post
# number, or call-number fragment ("10-4", "6-8", "72-holland",
# "11-victor", "98", "114") — procedural, not incident content. This is
# deliberately broad: a real event transcript that happens to include a
# digit-led token (an address number, a case number) still has other,
# non-digit descriptive words left over, so this alone never reduces a
# real transcript to nothing. See the backtest for confirmation.
if token[0].isdigit():
return True
return (
token in _FILLER_WORDS
or token in _RADIO_DESIGNATORS
or token in _PHONETIC_ALPHA_WORDS
)
def classify_chatter(transcript: Optional[str]) -> tuple[bool, Optional[str]]:
"""
Pure classification of a transcript as non-event radio housekeeping.
Returns (is_chatter, reason):
(True, "roll_call") — contains a roll-call announcement
(True, "bare_acknowledgement") — every token is a callsign/10-code/
procedural filler word; nothing else
(False, None) — not confidently chatter; let the
existing pipeline run as today
Takes only the transcript. Other call metadata (talkgroup, severity, tags)
doesn't exist yet at the point this needs to run — this classifier is
upstream of the scene-extraction call that produces those fields — so it
deliberately doesn't take them as input.
"""
if not transcript or not transcript.strip():
return False, None
lowered = transcript.lower()
if _ROLL_CALL_RE.search(lowered):
return True, "roll_call"
tokens = _tokenize(transcript)
if not tokens:
return False, None
if any(not _is_filler_token(t) for t in tokens):
return False, None
return True, "bare_acknowledgement"
+32
View File
@@ -0,0 +1,32 @@
"""
Pipeline clock.
`now()` is `datetime.now(timezone.utc)` everywhere except inside a replay run
(app/internal/replay.py), which pins it to the replayed call's own time so the
correlator's recency windows, the idle-resolve sweep and every started_at /
updated_at / resolved_at it writes behave the way they did live.
A ContextVar rather than a module global: a replay runs as a background task
alongside real uploads, and each asyncio task (and every asyncio.to_thread it
spawns) carries its own copy of the context, so a pinned clock can never leak
into a live call's pipeline.
"""
from contextvars import ContextVar
from datetime import datetime, timezone
from typing import Optional
_pinned: ContextVar[Optional[datetime]] = ContextVar("drb_clock_pinned", default=None)
def now() -> datetime:
pinned = _pinned.get()
return pinned if pinned is not None else datetime.now(timezone.utc)
def pin(when: Optional[datetime]):
"""Pin the clock for the current context. Returns a token for `unpin`."""
return _pinned.set(when)
def unpin(token) -> None:
_pinned.reset(token)
+190 -5
View File
@@ -6,7 +6,8 @@ in-memory TTL cache so flag reads don't add a Firestore round-trip to every
call upload.
"""
import time
from typing import Any
from contextvars import ContextVar
from typing import Any, Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
@@ -19,8 +20,38 @@ _DEFAULTS: dict[str, bool] = {
"correlation_enabled": True,
"summaries_enabled": True,
"vocabulary_learning_enabled": True,
# Transcript correction runs inside transcribe_call and spends Gemini
# tokens plus Places quota on every transcribed call. Until server-26#76
# it was reachable only through an env var and an ansible run, which meant
# an "STT-only" evaluation window was never STT-only and its cost could
# not be attributed (server-26#45).
#
# NOT a pure cost lever. The corrector is also the noise gate: it is what
# sets not_speech, and transcription.py returns nothing for a call it
# flags. _is_degenerate does not catch what the corrector catches, so with
# this off, recogniser noise reaches extraction as a real transcript, comes
# back with no units/tags/location, is judged thin, and auto-attaches to the
# most recent incident on the talkgroup with no fit check. Turning this off
# while correlation_enabled is on therefore pushes over-merging -- do not do
# it during an evaluation window.
"transcript_correction_enabled": True,
}
# A replay run (app/internal/replay.py) states exactly which AI steps it runs,
# independent of the live switches — the whole point is re-running the pipeline
# while live AI is OFF. ContextVar so the override never reaches a live upload.
_forced: ContextVar[Optional[dict[str, bool]]] = ContextVar("drb_forced_flags", default=None)
def force_flags(flags: Optional[dict[str, bool]]):
"""Override resolve_flags() for the current context. Returns a reset token."""
return _forced.set(flags)
def unforce_flags(token) -> None:
_forced.reset(token)
_cache: dict[str, Any] = {}
_cache_ts: float = 0.0
@@ -48,15 +79,169 @@ async def get_flags() -> dict[str, bool]:
return dict(_cache)
async def set_flags(updates: dict[str, bool]) -> dict[str, bool]:
"""Write flag updates to Firestore and invalidate the cache."""
global _cache, _cache_ts
async def _cascade_to_systems(clean: dict[str, bool]) -> tuple[list[dict], list[dict]]:
"""Clear per-system ``ai_flags`` overrides for the keys just set globally.
Returns ``(changes, errors)``.
Why clearing rather than overwriting with the new value: an override that
stays present, merely agreeing with the global switch for now, defeats the
NEXT flip exactly the same way. Removing it makes the system inherit, which
is the same semantics the human-facing route already offers
(``PUT /systems/{id}/ai-flags`` with null → "clear override, inherit
global").
Systems are discovered by scanning for documents that actually carry an
``ai_flags`` map — never a hardcoded id list. Two systems carry overrides
today; a third added tomorrow would silently defeat a global shutoff if
this were pinned to the current pair.
"""
changes: list[dict] = []
errors: list[dict] = []
systems = await fstore.collection_list("systems")
for system in systems:
sid = system.get("system_id")
ai_flags = system.get("ai_flags")
# Only documents that actually carry the map. A system with no
# overrides already inherits, so there is nothing to cascade to.
if not sid or not isinstance(ai_flags, dict) or not ai_flags:
continue
removed = {k: ai_flags[k] for k in clean if k in ai_flags}
if not removed:
continue
remaining = {k: v for k, v in ai_flags.items() if k not in clean}
try:
await fstore.doc_update("systems", sid, {"ai_flags": remaining})
except Exception as e:
# Report rather than swallow: a half-applied cascade is the exact
# failure mode this helper exists to prevent, so it must be visible
# in the log and the audit entry.
logger.error(f"Feature flags: cascade to system '{sid}' failed ({e})")
errors.append({"system_id": sid, "error": str(e)})
continue
changes.append({
"system_id": sid,
"cleared_overrides": removed,
"now_inherits": {k: clean[k] for k in removed},
})
return changes, errors
async def set_flags(
updates: dict[str, bool],
actor: tuple[str, str] | None = None,
cascade: bool = False,
) -> dict[str, bool]:
"""Write flag updates to Firestore, invalidate the cache, and audit it.
``actor`` is ``(actor_uid, actor_email)`` — see auth.describe_actor. It is
optional so existing callers keep working; an unattributed flip is logged
as "unknown" rather than not logged at all.
``cascade`` also clears the matching per-system ``ai_flags`` overrides, so
one call is a total flip. Defaults to False deliberately — see the route's
comment in routers/admin.py.
Returns the resulting global flags dict, unchanged in shape: the admin UI
(drb-frontend/lib/c2api.ts setFeatureFlags) types the response as
Record<string, boolean>, so cascade/audit detail goes to the log and the
audit entry rather than into this payload.
"""
global _cache_ts
clean = {k: bool(v) for k, v in updates.items() if k in _DEFAULTS}
if not clean:
raise ValueError(f"No recognised flag keys in update: {list(updates)}")
# Force a fresh read for the "before" side of the audit entry: the TTL
# cache can be up to _TTL seconds stale, and a wrong previous value in an
# audit log is worse than none.
_cache_ts = 0.0
before = await get_flags()
await fstore.doc_set(_COLLECTION, _DOC_ID, clean)
_cache_ts = 0.0 # force re-read on next get_flags()
logger.info(f"Feature flags updated: {clean}")
return await get_flags()
cascaded: list[dict] = []
cascade_errors: list[dict] = []
if cascade:
cascaded, cascade_errors = await _cascade_to_systems(clean)
logger.info(
f"Feature flags: cascaded {list(clean)} to {len(cascaded)} system(s), "
f"{len(cascade_errors)} error(s)"
)
after = await get_flags()
# The audit entry is a record OF the write, never a precondition for it.
# audit_log lives in the same Firestore that just accepted the flag write,
# so a failure here is nearly always transient — losing the flip (or 500ing
# a route that already succeeded, which invites a retry that flips it back)
# would be a far worse outcome than an unrecorded flip that is still in the
# service log above.
try:
# Deferred import: app.internal.audit pulls in firestore, and this
# module is imported from router module scope.
from app.internal import audit
actor_uid, actor_email = actor or ("unknown", "")
changed = {
k: {"from": before.get(k), "to": after.get(k)}
for k in clean
if before.get(k) != after.get(k)
}
await audit.write_audit(
actor_uid=actor_uid,
actor_email=actor_email,
action="feature_flags.update",
details={
"requested": clean,
"changed": changed,
"before": before,
"after": after,
"cascade": cascade,
"cascaded_systems": cascaded,
"cascade_errors": cascade_errors,
},
)
except Exception as e:
logger.error(f"Feature flags: audit write failed ({e}) — flag change stands")
return after
async def resolve_flags(system_id: str | None):
"""
Resolve the AI feature flags for one radio system.
Returns ``(flags, flag)``: ``flags`` is the raw global config/ai_features
document, and ``flag(name)`` layers the system's own ``ai_flags`` on top of
it. A system flag of False beats a global True, but a global False beats
everything -- config/ai_features is the master switch, which is the whole
point of having one (server-26#75, server-26#76).
Every AI spend path resolves through here. A path that reads ``flags``
directly re-introduces #75; a path that reads neither re-introduces #76.
"""
from app.internal import firestore as _fstore
forced = _forced.get()
if forced is not None:
full = {k: bool(forced.get(k, False)) for k in _DEFAULTS}
return full, lambda name: full.get(name, False)
flags = await get_flags()
system_ai_flags: dict = {}
if system_id:
sys_doc = await _fstore.doc_get_cached("systems", system_id)
system_ai_flags = (sys_doc or {}).get("ai_flags") or {}
def flag(name: str) -> bool:
if not flags[name]: # global master off
return False
return system_ai_flags.get(name, True) # system override, else inherit
return flags, flag
+55 -7
View File
@@ -1,5 +1,6 @@
import asyncio
import time as _time
from contextvars import ContextVar
from typing import Optional, Any
import firebase_admin
from firebase_admin import credentials, firestore as fs
@@ -7,6 +8,15 @@ from google.cloud.firestore_v1.base_query import FieldFilter
from app.config import settings
from app.internal.logger import logger
# Re-exported so callers never need their own `firebase_admin.firestore` import
# just to delete a field. server-26#96/#114 review: `doc_set(..., merge=True)`
# merges nested maps by key but can never REMOVE one — writing `{"scenes": {}}`
# to clear a map is a no-op, not a delete. Use `doc_update(coll, id, {"field":
# fstore.DELETE_FIELD})` (or doc_set + merge, DELETE_FIELD works under both)
# whenever a re-extraction/reprocess path needs a stale nested field gone
# rather than merged over.
DELETE_FIELD = fs.DELETE_FIELD
# ---------------------------------------------------------------------------
# In-memory TTL cache for rarely-changing documents (systems, nodes config)
# ---------------------------------------------------------------------------
@@ -31,23 +41,61 @@ _init_firebase()
db = fs.client(database_id=settings.firestore_database)
# ---------------------------------------------------------------------------
# Replay sandbox (app/internal/replay.py)
# ---------------------------------------------------------------------------
# While a replay run is executing, every read and write the pipeline makes to
# `calls` or `incidents` is redirected to that run's own subcollections under
# replay_runs/{run_id}/, so re-running the pipeline over past traffic can never
# touch a live call or incident. A subcollection keeps the same collection ID
# ("calls"/"incidents"), so the composite indexes prod queries depend on apply
# to it unchanged. Everything else (systems, nodes, config) is read from prod
# as-is. ContextVar for the same reason as app/internal/clock.py: the redirect
# follows the replay task and never a concurrent live upload.
SANDBOXED_COLLECTIONS = frozenset({"calls", "incidents"})
_sandbox_root: ContextVar[Optional[str]] = ContextVar("drb_fstore_sandbox", default=None)
def enter_sandbox(root: Optional[str]):
"""Redirect calls/incidents under `root` (e.g. "replay_runs/<id>") for this context."""
return _sandbox_root.set(root)
def exit_sandbox(token) -> None:
_sandbox_root.reset(token)
def in_sandbox() -> bool:
"""True inside a replay run. Anything that writes live state OTHER than
calls/incidents (AI health alerts, pending-term queues) checks this and
stands down — the redirect below only covers the two sandboxed collections."""
return _sandbox_root.get() is not None
def _path(collection: str) -> str:
root = _sandbox_root.get()
if root and collection in SANDBOXED_COLLECTIONS:
return f"{root}/{collection}"
return collection
# ---------------------------------------------------------------------------
# Thin async wrappers — firebase-admin is synchronous, run in thread executor
# ---------------------------------------------------------------------------
async def doc_set(collection: str, doc_id: str, data: dict, merge: bool = True) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.set, data, merge=merge)
async def doc_get(collection: str, doc_id: str) -> Optional[dict]:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
snap = await asyncio.to_thread(ref.get)
return snap.to_dict() if snap.exists else None
async def doc_update(collection: str, doc_id: str, data: dict) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.update, data)
@@ -57,7 +105,7 @@ async def collection_list(collection: str, **filters) -> list[dict]:
Optional keyword filters: field=value pairs passed as equality where-clauses.
"""
def _query():
ref = db.collection(collection)
ref = db.collection(_path(collection))
for field, value in filters.items():
ref = ref.where(filter=FieldFilter(field, "==", value))
return [doc.to_dict() for doc in ref.stream()]
@@ -94,7 +142,7 @@ async def collection_where(
unscoped equality-only lookups can keep using collection_list().
"""
def _query():
ref = db.collection(collection)
ref = db.collection(_path(collection))
for field, op, value in conditions:
ref = ref.where(filter=FieldFilter(field, op, value))
for field, direction in (order_by or []):
@@ -109,7 +157,7 @@ async def collection_where(
async def doc_delete(collection: str, doc_id: str) -> None:
ref = db.collection(collection).document(doc_id)
ref = db.collection(_path(collection)).document(doc_id)
await asyncio.to_thread(ref.delete)
@@ -119,7 +167,7 @@ async def doc_get_cached(collection: str, doc_id: str, ttl: float = 300.0) -> Op
Use for documents that change rarely (systems config, node assignments).
Default TTL is 5 minutes — a write will be visible within that window.
"""
key = f"{collection}/{doc_id}"
key = f"{_path(collection)}/{doc_id}"
now = _time.monotonic()
entry = _doc_cache.get(key)
if entry and now < entry[0]:
File diff suppressed because it is too large Load Diff
+452 -54
View File
@@ -15,11 +15,27 @@ import re
from typing import Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import ai_health
from app.internal import area_context
from app.internal.chatter_classifier import classify_chatter
# Location validity is defined once, by the module that owns the incident's
# location/pin invariant. incident_correlator does not import this module, so
# this is not a cycle.
from app.internal.incident_correlator import clean_location, location_is_unit
_PROMPT_TEMPLATE = """You are analyzing a P25 public safety radio recording. The audio was transcribed by Whisper through a digital radio vocoder, which introduces errors. Each numbered transmission is a separate PTT press from a different radio.
SCENE DETECTION:
A busy dispatch channel sometimes captures back-to-back conversations about multiple concurrent incidents in a single recording. Detect whether this recording contains ONE scene (all transmissions relate to a single event) or MULTIPLE scenes (clearly distinct dispatch conversations with different units being assigned, different locations, different event types). Assign short status transmissions (10-4, en route, acknowledgements) with no clear scene context to the most recent scene before them in the list.
A busy dispatch channel sometimes captures back-to-back conversations about multiple concurrent incidents in a single recording. Your default is ONE scene. Return MULTIPLE scenes ONLY when the recording clearly contains two or more SEPARATE EVENTS — different incidents at different places, with no shared units, no shared subject, and no conversational thread connecting them.
These do NOT make a new scene — keep them in the same scene:
- a different unit or speaker joining the same event
- a follow-up transmission about the same job (records check, case number, tow/mileage, a unit clearing, an ETA, a location correction)
- the same subject or location being discussed again minutes later
- an administrative or status exchange that follows an event on the same channel
If you are unsure whether two exchanges are one event or two, treat them as ONE.
Assign short status transmissions (10-4, en route, acknowledgements) with no clear scene context to the most recent scene before them in the list.
Always respond with the scenes array, even for a single scene.
@@ -45,14 +61,13 @@ Response format — a JSON object with a "scenes" array. Each scene:
severity: one of "routine" | "minor" | "moderate" | "major"
resolved: true if this scene explicitly signals incident closure, false otherwise
reassignment: true if a unit is breaking from their current scene to respond to a completely different call — whether dispatch-initiated ("Baker, can you clear and respond to...", "Adam, break from that and go to...") OR unit-initiated ("Show me headed to the vehicle complaint", "Can you show me to that call", a unit going 10-8 and self-requesting a new assignment). False if the unit is reporting in on their current scene, giving a status update, or requesting information about their existing call.
transcript_corrected: corrected text for this scene's transmissions only, or null
Rules:
- location: prefer intersections > addresses > mile markers > route+town > route alone > town alone. Dispatch-provided addresses take priority over unit-reported positions. Empty string if none.
- tags: describe WHAT happened, not WHERE. Specific, lowercase, hyphenated. Do not use location names, road names, talkgroup names, or place names as tags (wrong: "lower-macy's", "canvas-route-6", "route-202"; right: "suspect-search", "shoplifting", "vehicle-pursuit"). Do not repeat incident_type as a tag.
- units: ONLY identifiers that appear verbatim in the transcript. Use speaker role inference to distinguish units being dispatched from units acknowledging — both should be included. Never infer or guess unit IDs not present in the text.
- units: ONLY identifiers that appear verbatim in the transcript. Use speaker role inference to distinguish units being dispatched from units acknowledging — both should be included. Never infer or guess unit IDs not present in the text. If a unit ID format is given below, use it to recognise a unit spoken in a shortened or partial form (e.g. just the phonetic name alone) as the same unit — but still only extract what is actually said, never fabricate the full form.
- Do not invent details not present in the transcript.
- incident_type: let the talkgroup channel be your primary signal. Use "fire" ONLY if the talkgroup is clearly a fire/rescue channel OR the transcript explicitly describes active fire, smoke, flames, or structure fire activation. Police or EMS referencing a fire scene → use "police" or "ems". When the channel is a police channel and nothing in the transcript contradicts it, return "police" — do NOT fall back to "other" merely because the transmission is administrative. Reserve "other" for traffic that genuinely belongs to no emergency service (rail operations, public works, utility coordination). Reserve "unknown" for transcripts too garbled to place at all.
- incident_type: FIRST decide whether this transmission has any incident behind it at all, using the same bar as the "routine" severity rule below — pure administrative/status traffic with nothing describable happening: post/unit check-ins, roll call, bare acknowledgements ("10-4", "copy", "received"), records/report exchanges, "show me admin"/"show me available", a status ten-code with no event attached. If it is administrative/status-only, return "unknown" — this applies on EVERY channel, including a police channel; do not let the channel default override it (server-26#138: forcing a channel default onto content-free chatter is what let radio housekeeping open incidents). Only once real event content is present, let the talkgroup channel be your primary signal for WHICH type. Use "fire" ONLY if the talkgroup is clearly a fire/rescue channel OR the transcript explicitly describes active fire, smoke, flames, or structure fire activation. Police or EMS referencing a fire scene → use "police" or "ems". When the channel is a police channel, a real event is present, and nothing in the transcript contradicts it, return "police". Reserve "other" for a real event that genuinely belongs to no emergency service (rail operations, public works, utility coordination) — not for administrative chatter, which is "unknown" per above regardless of channel. Also reserve "unknown" for transcripts too garbled to place at all. A unit reporting its OWN activity is a real event, not status traffic: "on a stop" / traffic stop / car stop, "out with a vehicle", "put me out with a pedestrian/subject" — return "police", tag it (e.g. "traffic-stop", "pedestrian-assist"), severity at least "minor". The plate/license lookups for that stop belong to it.
- severity: ALWAYS return one of the four values. Judge the underlying event, not how dramatic the words sound.
"routine" — administrative/status traffic with no incident behind it: mileage and transport logging, radio checks, acknowledgements, shift changes, track block/power requests, records lookups.
"minor" — a real but low-stakes call: lift assist, parking complaint, past-tense larceny report, noise complaint, welfare check.
@@ -60,19 +75,15 @@ Rules:
"major" — life safety or major property loss: structure fire, vehicle pursuit, shots fired, entrapment, cardiac arrest, officer needing assistance.
- ten_codes: interpret radio codes using the department reference provided below. Do not guess codes not listed.
- resolved: true only when the scene explicitly signals "Code 4", "all clear", "10-42", "in custody", "patient transported", "fire out", "GOA", "negative contact", "scene clear".
- cleared_units: only include units that explicitly stated their own back-in-service status in this recording (e.g. "Unit 7, 10-8", "Baker-1 available", "E-14 back in service", or the department ten-code for available/back-in-service listed above). Silence or absence of a unit is NOT clearance. A scene-wide Code 4 belongs in resolved=true, not here — cleared_units is for individual unit availability signals only.
- cleared_units: include a unit whose back-in-service/available status is stated in this recording — either the unit self-reporting (e.g. "Unit 7, 10-8", "Baker-1 available", "E-14 back in service", or the department ten-code for available/back-in-service listed above) OR dispatch confirming that SPECIFIC unit's status back to them (e.g. the unit asks "how do you show me" and dispatch replies "showing you available" / "in service"). The unit ID must be identifiable either way — a bare "clear" or "10-8" with no unit attached to it is NOT clearance; do not guess which unit said it. Silence or absence of a unit is NOT clearance. A scene-wide Code 4 belongs in resolved=true, not here — cleared_units is for individual unit availability signals only.
- reassignment: only true when a unit is explicitly being pulled to a completely new call or location. A unit going en route to their first dispatch is NOT a reassignment. Routine status updates, acknowledgements, and scene updates are NOT reassignments.
- transcript_corrected: fix only clear STT/vocoder errors (e.g. "Several" → "10-4", misheard street names, garbled unit IDs). Keep all radio language as-is — do NOT decode codes into plain English. Return null if accurate.
System: {system_id}
Talkgroup: {talkgroup_name}
{ten_codes_block}{vocabulary_block}{transcript_block}"""
{ten_codes_block}{vocabulary_block}{unit_format_block}{transcript_block}"""
# The incident_type enum offered to the model in EXTRACTION_PROMPT. Kept here
# rather than only in the prompt so a model that invents a value cannot write it
# into incident.type. "unknown" is deliberately absent — it is a real answer
# from the model but not a usable type, and is normalised to None alongside
# anything unrecognised.
# "unknown" is deliberately absent — normalises to None, which is what lets
# the creation gate veto a content-free call (server-26#138).
_VALID_INCIDENT_TYPES = frozenset({"fire", "ems", "police", "accident", "other"})
# Geographic bias radius for geocoding — half-width in degrees (~55 km)
@@ -143,6 +154,23 @@ def _build_ten_codes_block(ten_codes: dict[str, str]) -> str:
return f"Department ten-codes:\n{lines}\n\n"
def _build_unit_format_block(unit_format_hint: Optional[str]) -> str:
"""
server-26#<pending> — unit ID formats vary per department (e.g. Yorktown:
"<district>-<phonetic>", "5-David", sometimes spoken as bare "David";
County: "<location>-<number>", "SAM-1", "airport-3", "parks-4") with no
shared pattern across systems. Without a per-system hint, the model has
no way to recognise a unit ID it hasn't seen phrased that way before, and
that failure compounds into cleared_units and reassignment detection,
both of which depend on first recognising which token IS the unit.
Owner-authored free text per system (systems/{id}.unit_format_hint via
PUT /systems/{id}/unit-format) — no auto-induction yet.
"""
if not unit_format_hint:
return ""
return f"This system's unit ID format: {unit_format_hint}\n\n"
async def extract_scenes(
call_id: str,
transcript: str,
@@ -160,7 +188,7 @@ async def extract_scenes(
Each scene dict contains:
tags, incident_type, location, location_coords, resolved,
severity, vehicles, units, transcript_corrected,
severity, vehicles, units, transcript, transcript_corrected,
segment_indices, embedding
Side-effect: updates calls/{call_id} in Firestore with merged tags,
@@ -169,12 +197,15 @@ async def extract_scenes(
"""
vocabulary: list[str] = []
ten_codes: dict[str, str] = {}
unit_format_hint: str = ""
if system_id:
# Single cached read — vocabulary and ten_codes live on the same document.
# Single cached read — vocabulary, ten_codes and unit_format_hint all
# live on the same document.
system_doc = await fstore.doc_get_cached("systems", system_id)
if system_doc:
vocabulary = system_doc.get("vocabulary") or []
ten_codes = system_doc.get("ten_codes") or {}
vocabulary = system_doc.get("vocabulary") or []
ten_codes = system_doc.get("ten_codes") or {}
unit_format_hint = system_doc.get("unit_format_hint") or ""
if _is_garbage_transcript(transcript):
logger.warning(
@@ -187,6 +218,28 @@ async def extract_scenes(
pass
return []
# server-26#127 — SHADOW MODE ONLY. Computes whether this transcript looks
# like non-event radio housekeeping (roll call, bare 10-4/10-8/98
# acknowledgements, unit check-ins) and records the verdict on the call
# doc, but does NOT skip extraction anywhere below — every path runs
# exactly as it did before this landed. Deliberately ahead of the ≤5-word
# skip: most bare acknowledgements ARE ≤5 words, and the first pass of
# this feature put the classifier after that return, so it never saw the
# bulk of its own target population — a review backtest against three
# live dumps found 82% of what it would have flagged already exits above
# as transcript_too_short, meaning a shadow-mode window would have shown
# roughly a fifth of the real catch rate. Computing it once, here, and
# folding the result into whichever skip/continue path runs below fixes
# that without adding a second Firestore write.
# TODO(server-26#127): flip this from shadow to live (skip extraction and
# write skip_reason="non_event_chatter" instead of just recording the
# verdict) once a live shadow-mode window confirms 0 false positives on
# real production traffic — pay particular attention to whole-transcript
# vs contains-anywhere matching for "roll call" and to digit-hyphen street
# addresses (e.g. "72-Holland"), both flagged as classifier risks that the
# dump backtest could not surface on its own.
chatter_is_chatter, chatter_reason = classify_chatter(transcript)
# Transcripts with ≤5 words carry no extractable intelligence — GPT hallucinates
# units and tags from thin context (e.g. "Main Lot", "10-4", "David").
if len(transcript.split()) <= 5:
@@ -194,22 +247,64 @@ async def extract_scenes(
f"Intelligence: call {call_id} — transcript too short for extraction "
f"({len(transcript.split())} words), skipping"
)
cleared_unit = _short_clearance_unit(transcript)
try:
# Severity is still recorded: a five-word acknowledgement is genuinely
# routine traffic, and downstream code treats a missing severity as
# "not yet processed" rather than "nothing happened".
await fstore.doc_set("calls", call_id, {
updates = {
"skip_reason": "transcript_too_short",
"severity": "routine",
})
"chatter_classifier_verdict": chatter_is_chatter,
"chatter_classifier_reason": chatter_reason,
}
if cleared_unit:
updates["units"] = [cleared_unit]
updates["cleared_units"] = [cleared_unit]
await fstore.doc_set("calls", call_id, updates)
except Exception:
pass
if cleared_unit:
logger.info(f"Intelligence: call {call_id} — short clearance from {cleared_unit!r}")
return [_clearance_scene(transcript, cleared_unit)]
# "Adam 3 on a stop" is the whole report of a stop, and it is <=5
# words: the self-initiated backstop has to run here too or the most
# common phrasing never opens an incident.
tags, typ, sev = _self_initiated_backstop(transcript, [], None, "routine", talkgroup_name)
if tags:
logger.info(f"Intelligence: call {call_id} — short self-initiated report {tags}")
return [{**_clearance_scene(transcript, ""), "units": [], "cleared_units": [],
"tags": tags, "incident_type": typ, "severity": sev}]
return []
raw_scenes: list[dict] = await asyncio.to_thread(
_sync_extract,
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
)
try:
await fstore.doc_set("calls", call_id, {
"chatter_classifier_verdict": chatter_is_chatter,
"chatter_classifier_reason": chatter_reason,
})
except Exception:
pass
try:
raw_scenes: list[dict] = await asyncio.to_thread(
_sync_extract,
transcript, talkgroup_name, talkgroup_id, system_id, segments, vocabulary, ten_codes,
unit_format_hint,
)
except Exception as e:
text = str(e)
kind = ai_health.classify(text)
logger.warning(f"GPT-4o-mini extraction failed for call {call_id}: {text}")
await ai_health.report_degraded(
"extraction", "openai", "gpt-4o-mini",
{"billing": "the OpenAI account is out of credit",
"dead_model": "model is unavailable"}.get(kind, f"extraction failed: {text[:200]}"),
{"billing": "Top up OpenAI billing",
"dead_model": "Update the extraction model in intelligence.py"}.get(kind, "Usually transient"),
permanent=kind != "transient",
)
return []
await ai_health.report_healthy("extraction")
if not raw_scenes:
return []
@@ -223,13 +318,44 @@ async def extract_scenes(
node_lat = node_doc.get("lat")
node_lon = node_doc.get("lon")
# The talkgroup's own anchor and place, when an operator has described it
# (server-26#36). This is what "where is this channel" should mean; the node
# position below is only the fallback for a system nobody has described.
tg_anchor: Optional[dict] = None
tg_area: dict = {}
if system_id:
system_doc = await fstore.doc_get_cached("systems", system_id)
if system_doc:
system_area = system_doc.get("area_context") or {}
tg_entry = area_context.talkgroup_entry(system_doc, talkgroup_id)
own_area = tg_entry.get("area_context") or {}
tg_area = area_context.effective(system_area, own_area)
tg_anchor = area_context.anchor_for(system_area, own_area)
processed: list[dict] = []
for scene in raw_scenes:
tags: list[str] = scene.get("tags") or []
incident_type: Optional[str] = scene.get("incident_type") or None
location: Optional[str] = scene.get("location") or None
# A location that is not a place ("49", from "Flames from 49") is
# rejected here, at the source: it never reaches the geocoder, the call
# document, the correlator or the summarizer prompt — which used to
# repeat it back as "A fire incident was reported at location 49".
# See incident_correlator.clean_location (server-26#23).
location: Optional[str] = clean_location(scene.get("location"))
vehicles: list[str] = scene.get("vehicles") or []
units: list[str] = scene.get("units") or []
# A "location" that is also one of this scene's own units is a unit
# call-sign, not a place. Both lists come from the same extraction pass,
# so the disagreement is free to detect and the string must be dropped
# before it reaches the geocoder — anchored place verification will
# otherwise resolve "Post 1-2" to a confident, plausible, wrong pin in
# the right town. See server-26#52.
if location and location_is_unit(location, units):
logger.info(
f"Intelligence: dropping location {location!r} — it is one of "
f"this scene's units, not a place"
)
location = None
cleared_units: list[str] = scene.get("cleared_units") or []
# Every call carries a severity — it is the signal the correlator uses to
# decide whether a call is incident-worthy at all, so it must never be
@@ -266,20 +392,29 @@ async def extract_scenes(
# Build the most specific query possible: location + municipality + state.
# e.g. "High Street" → "High Street, Yorktown, New York"
# This prevents generic street names from resolving to wrong-country results.
#
# Prefer the place an operator actually set over the one guessed from
# the talkgroup's name and the node's reverse-geocoded position. A name
# like "Ossining PD" gives a municipality with no state behind it, which
# is how a generic street name ends up resolving in the wrong half of
# the country.
location_coords: Optional[dict] = None
if location and node_lat is not None and node_lon is not None:
muni = _municipality_from_tg(talkgroup_name)
state = await _get_node_state(node_id or "", node_lat, node_lon) if node_id else ""
county = _node_county_cache.get(node_id or "") if node_id else ""
parts = [location]
if muni:
parts.append(muni)
if county:
parts.append(county)
if state:
parts.append(state)
if location:
node_state, node_county = "", ""
if not area_context.has_place(tg_area) and node_id and node_lat is not None and node_lon is not None:
# Only worth the (cached-after-first-call) reverse-geocode
# when nothing better already describes this talkgroup.
node_state = await _get_node_state(node_id, node_lat, node_lon)
node_county = _node_county_cache.get(node_id) or ""
parts, tg_named_region = _location_query_parts(
location, tg_area, talkgroup_name, node_state, node_county,
)
query = ", ".join(parts)
location_coords = await _geocode_location(query, node_lat, node_lon)
if tg_anchor or (node_lat is not None and node_lon is not None):
location_coords = await _geocode_location(
query, node_lat, node_lon, anchor=tg_anchor,
trust_named_region=tg_named_region,
)
# Embed this scene's content
scene_text = _build_scene_embed_text(
@@ -287,6 +422,14 @@ async def extract_scenes(
)
embedding = await asyncio.to_thread(_sync_embed, scene_text)
scene_transcript = _scene_transcript_text(
transcript, segments, segment_indices, transcript_corrected
)
tags, incident_type, severity = _self_initiated_backstop(
scene_transcript or transcript, tags, incident_type, severity, talkgroup_name,
)
processed.append({
"tags": tags,
"incident_type": incident_type,
@@ -298,6 +441,7 @@ async def extract_scenes(
"severity": severity,
"resolved": resolved,
"reassignment": reassignment,
"transcript": scene_transcript,
"transcript_corrected": transcript_corrected,
"segment_indices": segment_indices,
"embedding": embedding,
@@ -314,8 +458,9 @@ async def extract_scenes(
updates: dict = {"tags": all_tags, "severity": primary["severity"]}
if primary["location"]:
updates["location"] = primary["location"]
if primary["location_coords"]:
# Both, together, always — a re-extraction that produces a new address
# must not leave the previous address's pin on the call (server-26#23).
updates["location"] = primary["location"]
updates["location_coords"] = primary["location_coords"]
if all_units:
updates["units"] = all_units
@@ -344,6 +489,118 @@ async def extract_scenes(
return processed
# Self-initiated activity: a unit putting itself "on a stop" or "out with" a
# vehicle/pedestrian. Replay of 09-22 (server-26#170): every traffic stop on
# the Ch 1 channel ("45 Adam on a stop, Eastbound Central Express", "CM2 on
# the stop, southbound") came back untyped/untagged/routine from extraction —
# read as status traffic — so the creation gate never opened an incident and
# the stop was visible only in the archive. The prompt now says so too; this
# is the deterministic backstop, because a tag is what the creation gate
# counts as substance (incident_correlator.has_event_substance).
# Only a unit's own "on a stop" self-report — a bare "traffic stop"/"car stop"
# mention (a plate lookup on a records channel, a dispatcher's question) is
# left to the prompt, and "pull over" is too common in non-stop traffic
# ("Medic 2 pull over to the side") to trust (review of 433b35d).
_SELF_INITIATED = (
(re.compile(r"\bon (a|the) (traffic |car |vehicle |motor vehicle )?stop\b", re.IGNORECASE),
"traffic-stop"),
(re.compile(r"\b((put|show) me out with|out with (a|one) (pedestrian|vehicle|disabled|male|female|"
r"subject|party|juvenile))\b", re.IGNORECASE),
"self-initiated"),
)
_NEGATED = re.compile(r"\b(not|don't|dont|no|never)\s+(\S+\s+){0,5}$", re.IGNORECASE)
# "Train 4 holding on the stop", "out with a disabled on the bridge": on rail,
# bridge/tunnel, fire and EMS channels these phrases are operations, not a
# police stop. Everywhere else — including "Ch 1 (Patched ...)", which is where
# the stops actually are — the backstop applies.
_NO_BACKSTOP_TG = re.compile(r"\b(mta|rail|railroad|train|transit|bridges? and tunnels|fire|ems|"
r"rescue|ambulance|dpw|public works)\b", re.IGNORECASE)
def _self_initiated_backstop(
text: str, tags: list, incident_type: Optional[str], severity: str,
talkgroup_name: Optional[str] = None,
) -> tuple[list, Optional[str], str]:
if talkgroup_name and _NO_BACKSTOP_TG.search(talkgroup_name):
return tags, incident_type, severity
for pattern, tag in _SELF_INITIATED:
m = pattern.search(text or "")
if not m or _NEGATED.search(text[: m.start()]):
continue
if tag not in tags:
tags = [*tags, tag]
incident_type = incident_type or "police"
if severity == "routine":
severity = "minor"
return tags, incident_type, severity
# "45-9, I'm clear." / "Vehicle 1, clear." / "Car 12 10-8" — a unit reporting
# itself back in service is the one signal that ends an incident, and it is
# almost always five words or fewer, which is exactly the population the
# too-short skip above keeps away from GPT. In the first replay
# (server-26#170, 09-22 10:00-12:00 ET) 25 transmissions said 10-8/clear and
# 2 reached cleared_units. Rule-based on purpose: no model call, and only a
# unit named BEFORE the status word counts, so "10-8, 10-8." or "CMT clear."
# (no number) clears nobody rather than guessing.
_CLEAR_WORD_RE = re.compile(
r"\b(clear|10-?8|10-?98|back in service|in service|available)\b", re.IGNORECASE
)
_TEN_CODE_TOKEN_RE = re.compile(r"^10-?\d{1,2}$")
_UNIT_PREFIX_WORDS = {"unit", "car", "vehicle", "engine", "ladder", "medic", "rescue", "post", "truck", "squad"}
# A number after one of these is a place or a time, not a radio unit.
_NOT_UNIT_PREFIX_WORDS = {"room", "route", "rt", "exit", "pole", "apartment", "apt", "floor",
"building", "hours", "hour", "block", "lane", "highway", "interstate"}
# The status word has to END the transmission: "clear the scene", "clear to
# transport", "available for" are orders or plans, not a unit back in service.
_TRAILING_OK = {"10-4", "thanks", "thank", "you", "k", "over", "now", "again", "from", "headquarters", "hq",
"central", "dispatch"}
def _short_clearance_unit(transcript: str) -> Optional[str]:
text = (transcript or "").strip()
if not text or "?" in text:
return None # "45-9, are you clear?" asks; it doesn't report
m = _CLEAR_WORD_RE.search(text)
if not m:
return None
before = [t.strip(".,;:!") for t in text[: m.start()].split()]
before = [t for t in before if t]
after = [t.strip(".,;:!").lower() for t in text[m.end():].split()]
if any(t and t not in _TRAILING_OK for t in after):
return None
if any(t.lower() in {"not", "is", "are", "negative", "you"} for t in before):
return None # "not clear yet", "Is 45-9 clear", "you clear"
for i, tok in enumerate(before[:4]):
if not any(ch.isdigit() for ch in tok) or _TEN_CODE_TOKEN_RE.match(tok):
continue
prev = before[i - 1].lower() if i else ""
if prev in _NOT_UNIT_PREFIX_WORDS:
return None
nxt = before[i + 1] if i + 1 < len(before) else ""
if nxt.lower() in _NOT_UNIT_PREFIX_WORDS:
return None # "1400 hours, clear"
if nxt.isdigit():
tok = f"{tok}-{nxt}" # "45 9 clear" is unit 45-9, not unit 45
elif nxt.isalpha() and nxt[0].isupper() and nxt.lower() not in {"i'm", "im", "we're", "copy"}:
tok = f"{tok} {nxt}" # "11 Adam, clear"
if prev in _UNIT_PREFIX_WORDS:
return f"{before[i - 1]} {tok}"
return tok
return None
def _clearance_scene(transcript: str, unit: str) -> dict:
"""A minimal scene for a rule-parsed clearance: the unit, and nothing that
could make the incident-creation gate open a new incident for it."""
return {
"tags": [], "incident_type": None, "location": None, "location_coords": None,
"resolved": False, "severity": "routine", "vehicles": [], "units": [unit],
"cleared_units": [unit], "reassignment": False, "transcript": transcript,
"transcript_corrected": None, "segment_indices": [], "embedding": None,
}
def _geo_dist_km(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
"""Haversine distance in km between two lat/lon points."""
R = 6371.0
@@ -403,12 +660,46 @@ async def _get_node_state(node_id: str, lat: float, lon: float) -> str:
async def _geocode_location(
location_str: str, node_lat: float, node_lon: float
location_str: str,
node_lat: Optional[float] = None,
node_lon: Optional[float] = None,
anchor: Optional[dict] = None,
trust_named_region: bool = False,
) -> Optional[dict]:
"""
Geocode using Google Maps Geocoding API, biased toward the node's area.
Returns {"lat": float, "lng": float} or None if geocoding fails or the
result is farther than geocode_max_km from the node (wrong-jurisdiction guard).
Geocode using Google Maps Geocoding API, biased toward the channel's area.
Returns {"lat": float, "lng": float}, or None if geocoding fails or the
result lands outside the area this channel covers.
THE REFERENCE POINT IS THE TALKGROUP, NOT THE NODE (server-26#6 / #37). This
used to reject anything more than geocode_max_km (40km) from the receiving
node, which conflates an antenna with a jurisdiction: a system can span a
county or several, so a node legitimately sits far from the area a talkgroup
covers, and real dispatch locations were being thrown away for it. When the
talkgroup has a resolved anchor, that is the reference and its own radius is
the bound. Distance-from-node stays only as the fallback for a system nobody
has described yet — it was always a stand-in for this.
server-26#159: "a system nobody has described yet" turned out to include
systems that describe themselves — "New York City - NYPD Citywide 2 Patch"
names its own coverage area right in the talkgroup name, parsed into the
query by `_municipality_from_tg`, but a large aggregated/patched feed like
this is routinely received 40-70km from an antenna that happens to sit
wherever the node owner lives. Real, correctly-geocoded addresses on that
feed were being rejected by the node-distance check every single time —
location_coords stayed permanently null for the whole system, which killed
the location_proximity correlation signal and let duplicate incidents form
for the same event reported at two nearby addresses two minutes apart.
`trust_named_region` is True exactly when the query already carries a place
name that isn't the node's own position — operator-set area_context, or a
municipality parsed from the talkgroup's own name. In that case a distant
node is not evidence of a bad geocode, so the node-distance check is
skipped and precision is judged by `location_type` alone (still required
to be ROOFTOP/RANGE_INTERPOLATED/GEOMETRIC_CENTER, below). This does not
touch the anchor path at all — an anchor's own radius is always authoritative
when one has been resolved.
"""
import httpx
from app.config import settings
@@ -417,9 +708,24 @@ async def _geocode_location(
logger.warning("GOOGLE_MAPS_API_KEY not set — geocoding disabled")
return None
if anchor:
ref_lat, ref_lon = anchor["lat"], anchor["lng"]
max_km = anchor["radius_km"]
# Bias box scaled to the anchor rather than a fixed half-degree, so a
# village biases tightly and a county loosely.
delta = max(max_km / 111.0, 0.05)
ref_label = "anchor"
elif node_lat is not None and node_lon is not None:
ref_lat, ref_lon = node_lat, node_lon
max_km = settings.geocode_max_km
delta = _GEO_DELTA
ref_label = "node"
else:
return None
bounds = (
f"{node_lat - _GEO_DELTA},{node_lon - _GEO_DELTA}"
f"|{node_lat + _GEO_DELTA},{node_lon + _GEO_DELTA}"
f"{ref_lat - delta},{ref_lon - delta}"
f"|{ref_lat + delta},{ref_lon + delta}"
)
params = {
"address": location_str,
@@ -457,15 +763,28 @@ async def _geocode_location(
return None
loc = result["geometry"]["location"]
lat, lng = float(loc["lat"]), float(loc["lng"])
dist_km = _geo_dist_km(node_lat, node_lon, lat, lng)
if dist_km > settings.geocode_max_km:
logger.warning(
f"Geocoding rejected '{location_str}' → ({lat:.4f}, {lng:.4f}) "
f"— {dist_km:.1f}km from node exceeds geocode_max_km={settings.geocode_max_km}"
dist_km = _geo_dist_km(ref_lat, ref_lon, lat, lng)
if dist_km > max_km:
# server-26#159: the node-distance bound is a proxy for "is
# this plausible" that only makes sense when the node's own
# position is our best guess at the area — never when the
# query already names a different region on its own terms.
if not (ref_label == "node" and trust_named_region):
logger.warning(
f"Geocoding rejected '{location_str}' → ({lat:.4f}, {lng:.4f}) "
f"— {dist_km:.1f}km from {ref_label} exceeds {max_km:.1f}km"
)
return None
logger.info(
f"Geocoding '{location_str}' → ({lat:.4f}, {lng:.4f}) is "
f"{dist_km:.1f}km from the receiving node, past {max_km:.1f}km — "
f"accepted anyway: the query names its own region, not the node's"
)
return None
coords = {"lat": lat, "lng": lng}
logger.info(f"Geocoded '{location_str}' → {coords} ({dist_km:.1f}km from node) [{location_type}]")
logger.info(
f"Geocoded '{location_str}' → {coords} "
f"({dist_km:.1f}km from {ref_label}) [{location_type}]"
)
return coords
except Exception as e:
logger.warning(f"Geocoding failed for '{location_str}': {e}")
@@ -486,14 +805,89 @@ def _municipality_from_tg(tg_name: Optional[str]) -> Optional[str]:
return cleaned
def _location_query_parts(
location: str,
tg_area: dict,
talkgroup_name: Optional[str],
node_state: str,
node_county: str,
) -> tuple[list[str], bool]:
"""
Build the geocode query parts for `location`, plus whether the query names
a region the *talkgroup itself* covers (operator-set area_context, or a
municipality parsed from the talkgroup's own name) rather than one guessed
from wherever the receiving node happens to sit (server-26#159).
That distinction matters downstream: `_geocode_location`'s node-distance
sanity check is only a valid proxy for "is this plausible" when the node's
own position is the best guess we have at the area. A citywide/patched
feed ("New York City - NYPD Citywide 2 Patch") names its own coverage area
right in the talkgroup name — grafting the node's own county onto that
(Ossining-style: valid when the feed genuinely is local to the node,
actively wrong when it names a distant region of its own) would make the
query self-contradictory, so the node's COUNTY is used only when nothing
better names the place. The node's STATE is coarse enough to still be
correct either way and is kept in both branches.
"""
parts = [location]
if area_context.has_place(tg_area):
parts += [tg_area[f] for f in area_context.PLACE_FIELDS if tg_area.get(f)]
return parts, True
muni = _municipality_from_tg(talkgroup_name)
if muni:
parts += [p for p in (muni, node_state) if p]
else:
parts += [p for p in (node_county, node_state) if p]
return parts, muni is not None
def _build_transcript_block(transcript: str, segments: Optional[list[dict]]) -> str:
"""Format transcript as numbered transmissions if segments are available."""
if segments and len(segments) > 1:
lines = [f"{i+1}. [{s['start']}s] {s['text']}" for i, s in enumerate(segments)]
# 0-based labels, matching the prompt's "0-based indices into the
# numbered transmissions" — the model echoes these back as
# `segment_indices`, which _build_scene_embed_text and the per-scene
# `transcript` (server-26#102) then slice with directly.
lines = [f"{i}. [{s['start']}s] {s['text']}" for i, s in enumerate(segments)]
return f"Transmissions ({len(segments)}):\n" + "\n".join(lines)
return f"Transcript:\n{transcript}"
def _scene_transcript_text(
transcript: str,
segments: Optional[list[dict]],
segment_indices: Optional[list[int]],
transcript_corrected: Optional[str],
) -> str:
"""
This scene's own words, unprefixed — the segments it owns, joined.
server-26#102: the correlator's LLM tier reads this per scene instead of
the call doc's whole-call transcript, so on a multi-scene call scene N is
no longer judged against scenes 1..N-1's text.
Never returns "". Anything that would leave the slice empty — no
`segment_indices` (a single-segment call is never numbered by
`_build_transcript_block`), or indices that are out of range / not ints —
falls back to the whole-call transcript, which for a single-scene call is
the same text and for a mis-sliced multi-scene call is at least this
call's own words. `_sync_extract`'s prompt documents 0-based indices and
`_build_transcript_block` numbers to match, so no base normalisation here.
"""
if transcript_corrected:
return transcript_corrected
if segments and segment_indices:
joined = " ".join(
segments[i]["text"]
for i in segment_indices
if isinstance(i, int) and 0 <= i < len(segments)
)
if joined:
return joined
return transcript
def _build_scene_embed_text(
transcript: str,
segments: Optional[list[dict]],
@@ -519,6 +913,7 @@ def _sync_extract(
segments: Optional[list[dict]],
vocabulary: Optional[list[str]] = None,
ten_codes: Optional[dict[str, str]] = None,
unit_format_hint: Optional[str] = None,
) -> list[dict]:
"""Call GPT-4o-mini and return a list of scene dicts."""
from app.config import settings
@@ -536,6 +931,7 @@ def _sync_extract(
system_id=system_id or "unknown",
ten_codes_block=_build_ten_codes_block(ten_codes or {}),
vocabulary_block=build_gpt_vocab_block(vocabulary or []),
unit_format_block=_build_unit_format_block(unit_format_hint),
)
try:
@@ -558,9 +954,11 @@ def _sync_extract(
except json.JSONDecodeError as e:
logger.warning(f"GPT-4o-mini returned non-JSON: {e}")
return []
except Exception as e:
logger.warning(f"GPT-4o-mini extraction failed: {e}")
return []
# Any other exception is the API call itself failing (no credit, rate
# limit, outage) and propagates to extract_scenes, which reports it to
# ai_health. Swallowing it here made "OpenAI is down" indistinguishable
# from "nothing happened on the radio" — the extraction tier existed in
# /health/ai but nothing ever reported to it.
def _sync_embed(text: str) -> Optional[list[float]]:
+96 -24
View File
@@ -24,6 +24,7 @@ import json
from datetime import datetime, timezone
from typing import Optional
from app.internal.logger import logger
from app.internal import ai_health
from app.config import settings
@@ -44,7 +45,18 @@ def _fmt_idle(inc: dict, now: datetime) -> str:
def _inc_summary(inc: dict, now: datetime) -> str:
# server-26#115: the model was given no title and no talkgroup, so it
# could not tell that "car alarms, Mohegan Park Ave" and "car alarms,
# Mohegan Park Avenue" on the same channel were one incident — it defaulted
# to "new". Title is the single strongest human-readable signal for "is
# this the same event"; talkgroup is what makes same-channel continuation
# obvious.
parts = [f"id:{inc['incident_id']}", f"type:{inc.get('type') or '?'}"]
tgs = inc.get("talkgroup_ids") or []
if tgs:
parts.append(f"tg:[{', '.join(str(t) for t in tgs[:3])}]")
if inc.get("title"):
parts.append(f"title:{inc['title']!r}")
if inc.get("location"):
parts.append(f"loc:{inc['location']}")
units = inc.get("units") or []
@@ -60,7 +72,13 @@ def _inc_summary(inc: dict, now: datetime) -> str:
def _call_block(ctx: dict) -> str:
lines = []
call_doc = ctx["call_doc"]
transcript = call_doc.get("transcript_corrected") or call_doc.get("transcript")
# The SCENE's own transcript, resolved in _build_context (server-26#102).
# Falls back to the call doc for a ctx built without a scene (tests, sweep).
transcript = (
ctx.get("scene_transcript")
or call_doc.get("transcript_corrected")
or call_doc.get("transcript")
)
if transcript:
lines.append(f"Transcript: {transcript[:700]}")
if ctx["tags"]:
@@ -73,19 +91,50 @@ def _call_block(ctx: dict) -> str:
lines.append(f"Units: {ctx['call_units']}")
if ctx["call_vehicles"]:
lines.append(f"Vehicles: {ctx['call_vehicles']}")
if ctx["talkgroup_name"]:
lines.append(f"Talkgroup: {ctx['talkgroup_name']}")
if ctx["talkgroup_name"] or ctx.get("talkgroup_id") is not None:
# Both the name and the id — _inc_summary emits numeric tg ids, so the
# id is what makes the "same talkgroup" rule in _RULES evaluable
# (server-26#115 review).
tgid = ctx.get("talkgroup_id")
name = ctx["talkgroup_name"] or "?"
lines.append(f"Talkgroup: {name}" + (f" (id {tgid})" if tgid is not None else ""))
return "\n".join(lines) if lines else "(no details)"
def _prompt_incidents(recent: list[dict]) -> list[dict]:
"""The ≤20 candidates shown to the model, most-recently-active first.
`ctx["recent"]` is an unordered slice of a Firestore result with no
order_by, so a busy 2h window (~40 active incidents) meant the model saw
an arbitrary half of the candidates (server-26#115 review). Sorting by
updated_at desc also makes each row's `idle:` field monotonic.
"""
def _key(inc: dict):
return str(inc.get("updated_at") or inc.get("started_at") or "")
return sorted(recent, key=_key, reverse=True)[:20]
_SCHEMA = '{"action": "link" | "new" | "orphan", "incident_id": "<id_string or null>", "reasoning": "<one sentence>"}'
_RULES = """
Rules:
- "link" only with clear positive evidence: same units, same geocoded location, or semantically identical scene on the same talkgroup within the last few minutes.
- A call on a DIFFERENT talkgroup than an incident requires unit overlap or geocoded location match — topic similarity alone is not enough.
- "new" only if the call has a clear incident_type AND describes a distinct, identifiable scene.
- "orphan" when in doubt — conservative is always correct.
Rules (this system OVER-SPLITS — a real incident routinely gets shattered into
5-10 duplicates. A wrong link is cheap; a duplicate incident is the failure
mode. Bias accordingly.):
- Prefer "link" when the call plausibly continues a recent incident ON THE SAME
TALKGROUP: same or overlapping units, the same or an adjacent location (treat
"Ave"/"Avenue", "St"/"Street", "Rd"/"Road" as identical; a house number plus
the same street is the same place), the same subject/vehicle/case number, or a
follow-up beat ("units clearing", "negative contact", "tow en route", "event
number 214-201", a status update) to an incident that is only a few minutes
idle. The bar for "link" on the same talkgroup is LOW.
- Reserve "new" for a call that clearly describes a DIFFERENT event from every
recent incident — a different place, different units, and a different subject,
not merely a different transmission about the same job.
- "orphan" a call that is not an incident at all: radio checks, roll call,
a unit marking on/off duty or 10-8/10-98, mileage/log entries, a bare
acknowledgement. Do not open a "new" incident for these.
- A call on a DIFFERENT talkgroup than an incident still requires unit overlap
or a geocoded/location match — topic similarity alone is not enough there.
- Do NOT link just because both calls involve police or both mention a road.
"""
@@ -94,7 +143,7 @@ def _build_decide_prompt(ctx: dict) -> str:
now = ctx["now"]
recent = ctx["recent"]
inc_block = (
"\n".join(_inc_summary(inc, now) for inc in recent[:20])
"\n".join(_inc_summary(inc, now) for inc in _prompt_incidents(recent))
if recent else "(none)"
)
return (
@@ -112,7 +161,7 @@ def _build_tiebreak_prompt(rules_decision: dict, llm_decision: dict, ctx: dict)
now = ctx["now"]
recent = ctx["recent"]
inc_block = (
"\n".join(_inc_summary(inc, now) for inc in recent[:20])
"\n".join(_inc_summary(inc, now) for inc in _prompt_incidents(recent))
if recent else "(none)"
)
@@ -226,6 +275,12 @@ async def decide(call_id: str, ctx: dict) -> Optional[dict]:
if ctx["is_thin_call"]:
return None # thin calls have no transcript/units/coords to reason about
if _is_clearance_only(ctx):
# "45-9, I'm clear." carries one fact: which unit is done. Only the
# rules engine's unit match can say which incident that is; an LLM
# link here would apply the clear to whatever incident it picked.
return None
if not ctx["recent"]:
return None # no incidents to correlate against — rules handles new-only
@@ -239,18 +294,29 @@ async def decide(call_id: str, ctx: dict) -> Optional[dict]:
f"action={decision['action']} incident={_id} "
f"reasoning={decision['reasoning']!r}"
)
await ai_health.report_healthy("correlation_cheap")
return decision
except Exception as e:
_log_llm_failure("LLM correlator", call_id, settings.corr_cheap_model, e)
await _log_llm_failure("LLM correlator", "correlation_cheap", call_id, settings.corr_cheap_model, e)
return None
def _is_clearance_only(ctx: dict) -> bool:
units = ctx.get("call_units") or []
cleared = ctx.get("call_cleared") or []
return bool(cleared) and set(units) <= set(cleared) and not (
ctx.get("tags") or ctx.get("location") or ctx.get("call_vehicles") or ctx.get("incident_type")
)
_dead_models: set[str] = set()
def _log_llm_failure(where: str, call_id: str, model: str, exc: Exception) -> None:
async def _log_llm_failure(where: str, tier: str, call_id: str, model: str, exc: Exception) -> None:
"""
Log an LLM failure, escalating a dead model ID to ERROR once per model.
Log an LLM failure, escalating a dead model ID to ERROR once per model,
and report it to the shared app.internal.ai_health registry either way
(which is what drives /health/ai and the Discord degradation alert).
A per-call WARNING was the only signal that gemini-2.0-flash had been shut
down, and since every failure falls back to the rules decision the pipeline
@@ -260,27 +326,32 @@ def _log_llm_failure(where: str, call_id: str, model: str, exc: Exception) -> No
that will never fix itself, so it gets ERROR and says what to do.
"""
text = str(exc)
low = text.lower()
kind = ai_health.classify(text)
if "404" in text or "not found" in low or "no longer available" in low:
_log_tier_down(where, model, "model is unavailable",
"Update CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py", text)
if kind == "dead_model":
await _log_tier_down(where, tier, model, "model is unavailable",
"Update CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py", text)
return
# A depleted balance reads as 429, the same status as an ordinary rate limit,
# but it is the opposite kind of problem: a rate limit clears on its own and a
# dead account never does. Matching on the billing wording keeps a burst of
# rate limits at WARNING while an empty account escalates like a bad model ID.
if "credits are depleted" in low or "prepayment" in low or "billing" in low:
_log_tier_down(where, model, "the Gemini account is out of credit",
"Top up billing at https://ai.studio/projects", text)
# dead account never does. ai_health.classify() keeps a burst of rate limits
# at WARNING while an empty account escalates like a bad model ID.
if kind == "billing":
await _log_tier_down(where, tier, model, "the Gemini account is out of credit",
"Top up billing at https://ai.studio/projects", text)
return
logger.warning(f"{where} failed for call {call_id}: {text}")
await ai_health.report_degraded(
tier, "gemini", model, "transient API error",
"no action needed unless this persists", permanent=False,
)
def _log_tier_down(where: str, model: str, problem: str, fix: str, text: str) -> None:
async def _log_tier_down(where: str, tier: str, model: str, problem: str, fix: str, text: str) -> None:
"""ERROR once per model, not once per call — this runs at radio-traffic volume."""
await ai_health.report_degraded(tier, "gemini", model, problem, fix, permanent=True)
if model in _dead_models:
return
_dead_models.add(model)
@@ -306,9 +377,10 @@ async def tiebreak(rules_decision: dict, llm_decision: dict, ctx: dict) -> dict:
f"action={decision['action']} incident={_id} "
f"reasoning={decision['reasoning']!r}"
)
await ai_health.report_healthy("correlation_smart")
return decision
except Exception as e:
_log_llm_failure("LLM tiebreak", call_id, settings.corr_smart_model, e)
await _log_llm_failure("LLM tiebreak", "correlation_smart", call_id, settings.corr_smart_model, e)
return rules_decision
+14 -10
View File
@@ -6,6 +6,7 @@ import paho.mqtt.client as mqtt
from app.config import settings
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import talkgroups
from app.internal.tenancy import FOUNDING_ORG_ID
@@ -110,6 +111,8 @@ class MQTTHandler:
"assigned_system_id": None,
"approval_status": "pending",
"node_type": payload.get("node_type", "fixed"),
"secondary_sdr_mode": payload.get("secondary_sdr_mode", "none"),
"sdr_count": payload.get("sdr_count", 1),
"enforce_override_timeout": payload.get("enforce_override_timeout", True),
"is_overridden": False,
"override_system_id": None,
@@ -140,6 +143,11 @@ class MQTTHandler:
updates["node_type"] = node_type
updates["enforce_override_timeout"] = enforce_timeout
if "secondary_sdr_mode" in payload:
updates["secondary_sdr_mode"] = payload["secondary_sdr_mode"]
if "sdr_count" in payload:
updates["sdr_count"] = payload["sdr_count"]
if node_type == "portable":
updates["is_overridden"] = False
updates["override_system_id"] = None
@@ -219,16 +227,12 @@ class MQTTHandler:
else datetime.now(timezone.utc)
)
# Prefer the name from OP25 metadata; fall back to the system config
tgid_name = payload.get("tgid_name") or ""
if not tgid_name and system_id and payload.get("tgid"):
system_doc = await fstore.doc_get_cached("systems", system_id)
if system_doc:
tgid_int = int(payload["tgid"])
for tg in system_doc.get("config", {}).get("talkgroups", []):
if int(tg.get("id", -1)) == tgid_int:
tgid_name = tg.get("name", "")
break
# Prefer the name from OP25 metadata; fall back to the system config.
# The lookup lives in internal/talkgroups.py because /upload needs the
# identical resolution and used to go without it — see server-26#34.
tgid_name = await talkgroups.resolve(
system_id, payload.get("tgid"), hint=payload.get("tgid_name") or None
) or ""
doc = {
"call_id": call_id,
+255
View File
@@ -0,0 +1,255 @@
"""
Place verification — is the name the corrector produced a real place *here*?
The transcript corrector (`transcript_correction.py`) substitutes sound-alikes
against a reference list. It has no way to tell whether its own output is a real
place, so "Cool Parts, Illinois" and "Shout out to Optum" are exactly as
acceptable to it as a genuine street name. This module is the check
(server-26#37).
MAPS AS A VERIFIER, NOT AS PROMPT STUFFING. Injecting every road and POI in a
town would be hundreds of names on a pass that runs on every transcribed call.
Instead we take the handful of location-shaped nouns a transcript actually
contains and ask one question per noun:
1. Geocode it, bounded by the talkgroup's anchor.
2. Inside the radius -> accept, done.
3. Outside, or no result -> look for a sound-alike that DOES resolve inside.
4. Found one -> correct to it, and propose {term, meaning} to that
talkgroup's local_knowledge as pending.
Cost scales with location nouns, not call volume, and every verified miss
permanently improves the reference data for that channel.
NO ANCHOR MEANS SKIP, NOT ACCEPT. An anchor too wide to discriminate is not
stored at all (see `area_context`), and without one this module returns
immediately. A statewide radius would confirm anything inside it, which is worse
than not checking — it looks like verification and is not.
THE FREE TIER RUNS FIRST. A sound-alike among the terms the operator already
entered costs nothing and is more trustworthy than anything Maps guesses, so
`local_knowledge` and `vocabulary` are searched before any request goes out.
"""
import re
from difflib import SequenceMatcher
from typing import Any, Optional
from app.config import settings
from app.internal import area_context
from app.internal.logger import logger
# Soundex-style consonant classes. Letters that a vocoder + Whisper routinely
# swap land in the same bucket, so "Optum"/"Ossining" stay far apart while
# "Snowden"/"Snowdon" collapse together.
_CLASSES = {
"b": "1", "f": "1", "p": "1", "v": "1",
"c": "2", "g": "2", "j": "2", "k": "2", "q": "2", "s": "2", "x": "2", "z": "2",
"d": "3", "t": "3",
"l": "4",
"m": "5", "n": "5",
"r": "6",
}
_DIGRAPHS = (("ph", "f"), ("gh", "g"), ("ck", "k"), ("wr", "r"), ("kn", "n"), ("wh", "w"))
def _norm(text: str) -> str:
return re.sub(r"[^a-z0-9]+", " ", (text or "").lower()).strip()
def phonetic_key(text: str) -> str:
"""
Consonant-class skeleton of a name. Vowels drop out; a run of the same class
collapses unless a vowel separates it.
"""
letters = re.sub(r"[^a-z]", "", (text or "").lower())
for a, b in _DIGRAPHS:
letters = letters.replace(a, b)
out: list[str] = []
prev = ""
for ch in letters:
code = _CLASSES.get(ch, "")
if code and code != prev:
out.append(code)
prev = code if ch not in "aeiouyhw" else ""
return "".join(out)
def sounds_like(heard: str, candidate: str) -> float:
"""
0..1 similarity, the better of the phonetic and the literal comparison.
Both are needed: Whisper errors are sometimes phonetic ("5 acre" for
"5-baker") and sometimes near-spellings ("Croton Ave" for "Croton Avenue"),
and a key comparison alone scores the second one poorly.
"""
literal = SequenceMatcher(None, _norm(heard), _norm(candidate)).ratio()
ka, kb = phonetic_key(heard), phonetic_key(candidate)
phonetic = SequenceMatcher(None, ka, kb).ratio() if ka and kb else 0.0
return max(literal, phonetic)
# -- Maps ----------------------------------------------------------------------
def _place_suffix(area: dict) -> str:
parts = [area[f] for f in area_context.PLACE_FIELDS if area.get(f)]
return ", ".join(parts)
async def _geocode_in_anchor(query: str, anchor: dict) -> Optional[dict]:
"""Geocode `query` and return its coords only if they land inside the anchor."""
from app.internal.intelligence import _geocode_location
coords = await _geocode_location(query, anchor=anchor)
return coords
async def _places_soundalike(heard: str, anchor: dict) -> Optional[dict]:
"""
Ask Maps for places near the anchor matching the misheard text.
Places Text Search does its own fuzzy matching against a biased region, which
is usually enough — but "usually" is not a standard, so the result still has
to pass `sounds_like` before it is allowed to rewrite a transcript. Without
that guard the API happily returns the nearest gas station for any garbage
string.
"""
if not settings.google_maps_api_key:
return None
import httpx
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(
"https://maps.googleapis.com/maps/api/place/textsearch/json",
params={
"query": heard,
"location": f"{anchor['lat']},{anchor['lng']}",
"radius": int(anchor["radius_km"] * 1000),
"key": settings.google_maps_api_key,
},
)
r.raise_for_status()
data = r.json()
except Exception as e:
logger.warning(f"Place search failed for {heard!r}: {e}")
return None
if data.get("status") not in ("OK", "ZERO_RESULTS"):
logger.warning(f"Place search for {heard!r} returned {data.get('status')}")
return None
for result in (data.get("results") or [])[:5]:
name = (result.get("name") or "").strip()
loc = (result.get("geometry") or {}).get("location") or {}
if not name or "lat" not in loc:
continue
distance = area_context.geo_dist_km(
anchor["lat"], anchor["lng"], float(loc["lat"]), float(loc["lng"])
)
if distance > anchor["radius_km"]:
continue
score = sounds_like(heard, name)
if score >= settings.place_soundalike_min_ratio:
return {"term": name, "meaning": result.get("formatted_address") or None, "score": score}
return None
def _known_soundalike(heard: str, area: dict) -> Optional[dict]:
"""Best sound-alike among terms the operator already entered. Free."""
best: Optional[dict] = None
for entry in area.get("local_knowledge") or []:
term = entry.get("term") or ""
if not term or _norm(term) == _norm(heard):
continue
score = sounds_like(heard, term)
if score >= settings.place_soundalike_min_ratio and (best is None or score > best["score"]):
best = {"term": term, "meaning": entry.get("meaning"), "score": score, "known": True}
return best
# -- Public --------------------------------------------------------------------
def _substitute(text: str, swaps: list[tuple[str, str]]) -> str:
for heard, replacement in swaps:
text = re.sub(rf"\b{re.escape(heard)}\b", replacement, text, flags=re.IGNORECASE)
return text
async def verify(
call_id: str,
text: str,
segments: Optional[list[dict]],
locations: list[str],
system_area: Optional[dict],
tg_area: Optional[dict],
system_id: Optional[str] = None,
talkgroup_id: Optional[Any] = None,
) -> tuple[Optional[str], Optional[list[dict]]]:
"""
Check the corrector's location nouns against the talkgroup's anchor.
Returns (text, segments) with verified substitutions applied, or (None, None)
when nothing changed. Like correction itself, this is an improvement and
never a dependency: any failure leaves the transcript exactly as it was.
"""
if not settings.place_verification_enabled or not locations:
return None, None
anchor = area_context.anchor_for(system_area, tg_area)
if not anchor:
return None, None # load-bearing: no anchor means skip, never accept
area = area_context.effective(system_area, tg_area)
suffix = _place_suffix(area)
swaps: list[tuple[str, str]] = []
proposals: list[dict] = []
for heard in locations[: settings.place_verify_max_per_call]:
heard = (heard or "").strip()
if not heard:
continue
query = f"{heard}, {suffix}" if suffix else heard
try:
if await _geocode_in_anchor(query, anchor):
continue # real place, in the right area — nothing to do
candidate = _known_soundalike(heard, area) or await _places_soundalike(heard, anchor)
except Exception as e:
logger.warning(f"Place verification failed for {heard!r} on call {call_id}: {e}")
continue
if not candidate:
logger.info(
f"Place verification: {heard!r} (call {call_id}) does not resolve near the "
f"anchor and has no sound-alike that does — leaving it alone"
)
continue
swaps.append((heard, candidate["term"]))
if not candidate.get("known"):
proposals.append({
"term": candidate["term"],
"meaning": candidate.get("meaning"),
"source": "place_verifier",
"source_call_ids": [call_id],
})
logger.info(
f"Place verification: {heard!r} -> {candidate['term']!r} "
f"(score {candidate['score']:.2f}, call {call_id})"
)
if not swaps:
return None, None
if proposals and system_id and talkgroup_id is not None:
try:
await area_context.add_pending(system_id, talkgroup_id, proposals)
except Exception as e:
logger.warning(f"Could not queue verified terms for call {call_id}: {e}")
new_text = _substitute(text or "", swaps)
new_segments = None
if segments:
new_segments = [{**s, "text": _substitute(s.get("text", ""), swaps)} for s in segments]
if all(a["text"] == b.get("text") for a, b in zip(new_segments, segments)):
new_segments = None
return (new_text if new_text != (text or "") else None), new_segments
@@ -20,6 +20,46 @@ from app.internal.logger import logger
from app.internal import firestore as fstore
from app.config import settings
# server-26#131: minimum time since the real-time pipeline (routers/upload.py
# _run_intelligence_pipeline) marked intelligence_started_at before the sweep
# will touch a call, even though it already looks orphaned. STT + scene
# extraction + correlation is a multi-second-to-low-minutes chain (Whisper,
# then a Gemini call per scene); without this buffer the sweep could pick up
# a call mid-pipeline — no incident_id/corr_path written yet — and correlate
# it a second time, independently, sometimes onto a different incident than
# the real-time path lands on. That race is what #131 found (same call in
# two incidents' call_ids, ~2% of linked calls). A call with no
# intelligence_started_at at all (pre-#131 call doc, or the marker write
# itself failed) is NOT held back by this — absence isn't evidence of an
# in-flight pipeline, and #131's own bug predates this field existing.
#
# 15, not 5: neither OpenAI's Whisper client nor Gemini's call in
# llm_correlator.py sets a request timeout (server-26#153), so a hung call can
# run well past a few minutes on SDK-default retries, and this constant is a
# guess against that unbounded tail, not a measured bound. Raising it costs
# nothing on the recovery side: a call that finished processing (linked OR
# genuinely orphaned) always has corr_path set (_apply_and_log writes it even
# on the orphan action), so it's already excluded by the
# `not c.get("corr_path")` filter below and never reaches this check at all —
# this constant only ever delays calls that are still actually running.
MIN_MINUTES_SINCE_PIPELINE_START = 15
# Standard link-only retry budget before a call is tombstoned corr_path="unlinked".
MAX_SWEEP_ATTEMPTS = 3
# server-26#115 — a call the consensus LLM-orphan gate parked (llm=orphan vs
# rules=new, no substance) gets a longer budget. The gate fires before any
# incident for the job may exist, so the substantive call that would justify
# linking can land well after the standard ~6 min. Still link-only: a genuinely
# thin call must not mint an incident, and the rules creation gate would re-orphan
# it anyway.
GATED_ORPHAN_SWEEP_ATTEMPTS = 10
def _max_sweep_attempts(call: dict) -> int:
if call.get("corr_consensus") == "llm_orphan_gate":
return GATED_ORPHAN_SWEEP_ATTEMPTS
return MAX_SWEEP_ATTEMPTS
async def recorrelation_loop() -> None:
interval = settings.summary_interval_minutes * 60
@@ -36,8 +76,23 @@ async def recorrelation_loop() -> None:
logger.error(f"Re-correlation sweep failed: {e}")
def _pipeline_likely_still_running(call: dict, now: datetime) -> bool:
"""server-26#131 — True when the real-time pipeline marked
intelligence_started_at recently enough that it's probably still mid-flight
(STT / scene extraction / correlation), so the sweep should not race it.
No marker at all (older call doc, or the marker write itself failed)
returns False — absence isn't evidence of an in-flight pipeline."""
started = _parse_dt(call.get("intelligence_started_at"))
if not started:
return False
age_minutes = (now - started).total_seconds() / 60
return age_minutes < MIN_MINUTES_SINCE_PIPELINE_START
async def _run_sweep_pass() -> None:
cutoff = datetime.now(timezone.utc) - timedelta(minutes=settings.recorrelation_scan_minutes)
from app.internal import clock
now = clock.now()
cutoff = now - timedelta(minutes=settings.recorrelation_scan_minutes)
# Server-side range query: only calls that ended within the scan window.
# Filter incident_id=null client-side (Firestore can't query for missing fields).
@@ -46,10 +101,9 @@ async def _run_sweep_pass() -> None:
("status", "==", "ended"),
("ended_at", ">=", cutoff),
])
# corr_path="unlinked" is written after MAX_SWEEP_ATTEMPTS failures.
# corr_path="unlinked" is written after the attempt budget is exhausted.
# Allows a few retries so a welfare-check call can link to an escalation
# incident that is created a few minutes later, without sweeping 30× forever.
MAX_SWEEP_ATTEMPTS = 3
orphans = [
c for c in recent_ended
if not c.get("incident_ids") and not c.get("incident_id")
@@ -61,7 +115,8 @@ async def _run_sweep_pass() -> None:
# the thin path minutes later and attached to whatever was most recent —
# a second route into the over-merge the thin fix above addresses.
and not c.get("skip_reason")
and c.get("corr_sweep_count", 0) < MAX_SWEEP_ATTEMPTS
and c.get("corr_sweep_count", 0) < _max_sweep_attempts(c)
and not _pipeline_likely_still_running(c, now)
]
if not orphans:
@@ -90,6 +145,11 @@ async def _recorrelate_orphan(call: dict) -> bool:
return False
# All data needed for correlation was stored by the first-pass extraction.
# embedding/severity are no longer read from the call doc inside
# _build_context (server-26#80/#95) — the sweep re-links a whole call, not a
# scene, so it passes the call doc's stored (primary-scene) values here. It
# is link-only (create_if_new=False), so a borrowed severity cannot open a
# new incident off this path.
incident_id = await incident_correlator.correlate_call(
call_id = call_id,
node_id = call.get("node_id", ""),
@@ -101,6 +161,9 @@ async def _recorrelate_orphan(call: dict) -> bool:
location = call.get("location"),
location_coords= call.get("location_coords"),
cleared_units = call.get("cleared_units") or [],
embedding = call.get("embedding"),
severity = call.get("severity"),
transcript = call.get("transcript_corrected") or call.get("transcript"),
reference_time = started_at, # anchor window to when the call happened
create_if_new = False, # never create — link-only
)
@@ -112,12 +175,12 @@ async def _recorrelate_orphan(call: dict) -> bool:
)
return True
# Increment the attempt counter. Once MAX_SWEEP_ATTEMPTS is reached the
# orphan filter above will stop picking this call up, and we write
# corr_path="unlinked" as a permanent tombstone.
# Increment the attempt counter. Once the budget is reached the orphan filter
# above will stop picking this call up, and we write corr_path="unlinked" as
# a permanent tombstone.
attempts = call.get("corr_sweep_count", 0) + 1
update: dict = {"corr_sweep_count": attempts}
if attempts >= 3:
if attempts >= _max_sweep_attempts(call):
update["corr_path"] = "unlinked"
await fstore.doc_set("calls", call_id, update)
return False
+650
View File
@@ -0,0 +1,650 @@
"""
Replay — re-run the intelligence pipeline over past calls, in a sandbox.
Live AI windows are the only way the correlator has ever been measured, and
each one costs days of real time and whatever the credits allow: a change
ships, AI goes on, traffic trickles in, someone pulls a dump. Recordings are
kept whether AI is on or not, so the traffic to measure against already
exists. A replay run takes a time range of real calls, feeds them through
the SAME pipeline code the live upload path runs (routers/upload.py
`_extract_and_correlate`) in their original order with the clock pinned to
each call's own end time, and writes everything to
replay_runs/{run_id}/calls|incidents instead of the live collections. The
same range can then be replayed after every change and the runs compared.
Three modes, cheapest last:
audio re-transcribe the saved audio (Whisper + correction), then
extract and correlate. For ranges where AI was off.
transcripts reuse the transcript already on each call, re-run extraction
and correlation.
reuse reuse the scenes an earlier run extracted, re-run correlation
only. Extraction is an LLM call and never returns quite the
same thing twice, so this is the mode that isolates a
correlator change from extraction noise.
What never happens in a replay: alerts, summaries, vocabulary learning, and
any write to a live call or incident. The sandbox is enforced by the
ContextVar redirect in app/internal/firestore.py, not by this module
remembering to use different collection names.
One run at a time per process — a run spends real AI credits and its cost is
only estimated, so two concurrent runs would be two unbounded bills.
"""
import asyncio
import os
import statistics
import uuid
from collections import Counter
from datetime import datetime, timedelta, timezone
from typing import Optional
from app.config import settings
from app.internal import ai_health, clock
from app.internal import firestore as fstore
from app.internal.feature_flags import force_flags, unforce_flags
from app.internal.logger import logger
RUNS = "replay_runs"
MODES = ("audio", "transcripts", "reuse")
MAX_CALLS = 5000
MAX_RANGE_DAYS = 7
# Extraction/transcription run ahead of correlation with this much
# concurrency. They depend only on the call itself; correlation depends on
# every call before it and is kept strictly in order.
PREFETCH = 6
# Rough per-unit AI prices for the pre-run estimate and the running tally.
# Estimates, not a bill — nothing in DRB reads a real invoice (server-26#45).
USD_WHISPER_PER_MIN = 0.006
USD_PER_EXTRACTION = 0.0005 # gpt-4o-mini scene extraction + embedding
USD_PER_CORRECTION = 0.0003 # Gemini flash transcript correction
USD_PER_GEOCODE = 0.005 # Google geocode, roughly one per located scene
USD_PER_LLM_CORRELATE = 0.0005 # Gemini flash consensus decision
# Fields the pipeline writes onto a call doc. Stripped when a call is copied
# into the sandbox so the replay recomputes them instead of inheriting the
# live answer. Anything else on the doc (ids, times, talkgroup, srcaddr, audio
# location) is an input and is kept.
_DERIVED = {
"transcript", "transcript_corrected", "transcript_not_speech",
"segments", "segments_corrected", "scenes", "incident_id", "incident_ids",
"tags", "location", "location_coords", "location_mentions", "units",
"vehicles", "cleared_units", "severity", "incident_type", "type",
"embedding", "skip_reason", "intelligence_started_at", "reassignment",
"resolved", "has_updates", "audio_url",
}
_DERIVED_PREFIXES = ("corr_", "chatter_classifier_", "eval_")
_TRANSCRIPT_FIELDS = (
"transcript", "transcript_corrected", "transcript_not_speech",
"segments", "segments_corrected",
)
_active_run_id: Optional[str] = None
_active_task: Optional[asyncio.Task] = None
_cancel: set[str] = set()
class ReplayBusy(RuntimeError):
pass
def sandbox_root(run_id: str) -> str:
return f"{RUNS}/{run_id}"
def _scenes_coll(run_id: str) -> str:
# Extracted scenes (embeddings included) live beside the sandbox, not on
# its call docs, so reading a run's calls for metrics or the incident view
# doesn't haul every scene's embedding along a second time.
return f"{sandbox_root(run_id)}/scenes"
def active_run_id() -> Optional[str]:
if _active_task is not None and not _active_task.done():
return _active_run_id
return None
# ---------------------------------------------------------------------------
# Call selection
# ---------------------------------------------------------------------------
def _as_dt(value) -> Optional[datetime]:
if value is None:
return None
if isinstance(value, datetime):
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
try:
dt = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
except ValueError:
return None
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
async def select_calls(
org_id: str,
date_from: datetime,
date_to: datetime,
system_ids: Optional[list[str]] = None,
cap: int = MAX_CALLS,
) -> tuple[list[dict], bool]:
"""
Live calls in [date_from, date_to] for this org, oldest first.
Pages newest-first because the one composite index on calls that carries
org_id is (org_id ASC, started_at DESC); ordering the other way would need
a new index for no gain. Returns (calls, truncated) — truncated means the
range holds more than `cap` calls and the caller must narrow it rather
than silently replaying only part of it.
"""
out: list[dict] = []
cursor = None
page = 1000
while True:
rows = await fstore.collection_where(
"calls",
[("org_id", "==", org_id),
("started_at", ">=", date_from),
("started_at", "<=", date_to)],
order_by=[("started_at", "DESCENDING")],
limit_to=page,
start_after={"started_at": cursor} if cursor is not None else None,
)
for c in rows:
if c.get("duplicate_of"):
continue # another node's copy — live never processes these either
if system_ids and c.get("system_id") not in system_ids:
continue
out.append(c)
if len(out) > cap:
return sorted(out[:cap], key=_call_time), True
if len(rows) < page:
break
cursor = rows[-1].get("started_at")
return sorted(out, key=_call_time), False
def _call_time(call: dict) -> datetime:
return _as_dt(call.get("started_at")) or datetime.min.replace(tzinfo=timezone.utc)
def _pipeline_time(call: dict) -> datetime:
"""When the live pipeline would have run for this call: at upload, i.e. call end."""
return _as_dt(call.get("ended_at")) or _call_time(call)
def _duration_s(call: dict) -> float:
# Call docs carry no duration field; the node reports start and end.
start, end = _as_dt(call.get("started_at")), _as_dt(call.get("ended_at"))
return max(0.0, (end - start).total_seconds()) if start and end else 0.0
def estimate(calls: list[dict], mode: str) -> dict:
n = len(calls)
with_transcript = sum(1 for c in calls if c.get("transcript_corrected") or c.get("transcript"))
audio_min = sum(_duration_s(c) for c in calls) / 60
with_audio = sum(1 for c in calls if c.get("audio_gcs_uri"))
# Roughly a third of calls carry a geocodable location (09-22 dump: 92/373).
per_call = USD_PER_EXTRACTION + USD_PER_LLM_CORRELATE + USD_PER_GEOCODE / 3
if mode == "audio":
usd = audio_min * USD_WHISPER_PER_MIN + with_audio * (USD_PER_CORRECTION + per_call)
elif mode == "transcripts":
usd = with_transcript * per_call
else:
usd = n * USD_PER_LLM_CORRELATE
return {
"calls": n,
"calls_with_transcript": with_transcript,
"calls_with_audio": with_audio,
"audio_minutes": round(audio_min, 1),
"est_cost_usd": round(usd, 2),
}
# ---------------------------------------------------------------------------
# Run lifecycle
# ---------------------------------------------------------------------------
async def start_run(
*,
org_id: str,
date_from: datetime,
date_to: datetime,
mode: str,
system_ids: Optional[list[str]],
source_run_id: Optional[str],
label: str,
actor: str,
) -> dict:
global _active_run_id, _active_task
if active_run_id():
raise ReplayBusy(f"Replay {active_run_id()} is still running.")
if mode not in MODES:
raise ValueError(f"mode must be one of {MODES}")
if date_to <= date_from:
raise ValueError("date_to must be after date_from")
if date_to - date_from > timedelta(days=MAX_RANGE_DAYS):
raise ValueError(f"Range is capped at {MAX_RANGE_DAYS} days.")
if mode == "reuse":
src = await fstore.doc_get(RUNS, source_run_id or "")
if not src or src.get("org_id") != org_id:
raise ValueError("reuse mode needs a source_run_id from an earlier run in this org")
if src.get("status") != "done":
raise ValueError("The source run did not finish; its scenes are incomplete.")
calls, truncated = await select_calls(org_id, date_from, date_to, system_ids)
if truncated:
raise ValueError(f"Range holds more than {MAX_CALLS} calls — narrow it.")
if not calls:
raise ValueError("No calls in that range.")
run_id = uuid.uuid4().hex[:12]
now = datetime.now(timezone.utc).isoformat()
doc = {
"run_id": run_id,
"org_id": org_id,
"label": label or "",
"mode": mode,
"source_run_id": source_run_id if mode == "reuse" else None,
"date_from": date_from.isoformat(),
"date_to": date_to.isoformat(),
"system_ids": system_ids or [],
"git_sha": os.getenv("GIT_SHA", "unknown"),
"created_by": actor,
"created_at": now,
"status": "running",
"estimate": estimate(calls, mode),
"progress": {"total": len(calls), "done": 0, "errors": 0},
"metrics": None,
"errors": [],
}
await fstore.doc_set(RUNS, run_id, doc, merge=False)
_active_run_id = run_id
_active_task = asyncio.create_task(_run(run_id, org_id, calls, mode, source_run_id))
return doc
def request_cancel(run_id: str) -> bool:
if active_run_id() != run_id:
return False
_cancel.add(run_id)
return True
async def get_run(run_id: str) -> Optional[dict]:
doc = await fstore.doc_get(RUNS, run_id)
return await _reconcile(doc) if doc else None
async def list_runs(org_id: str) -> list[dict]:
docs = await fstore.collection_list(RUNS, org_id=org_id)
docs = [await _reconcile(d) for d in docs]
return sorted(docs, key=lambda d: d.get("created_at") or "", reverse=True)
async def _reconcile(doc: dict) -> dict:
"""A run left "running" by a process that restarted (a deploy) never finishes."""
if doc.get("status") == "running" and doc.get("run_id") != active_run_id():
doc["status"] = "interrupted"
await fstore.doc_set(RUNS, doc["run_id"], {"status": "interrupted"})
return doc
async def delete_run(run_id: str) -> None:
if active_run_id() == run_id:
raise ReplayBusy("Cancel the run before deleting it.")
token = fstore.enter_sandbox(sandbox_root(run_id))
try:
for coll, key in (("calls", "call_id"), ("incidents", "incident_id"),
(_scenes_coll(run_id), "call_id")):
for d in await fstore.collection_list(coll):
if d.get(key):
await fstore.doc_delete(coll, d[key])
finally:
fstore.exit_sandbox(token)
await fstore.doc_delete(RUNS, run_id)
async def sandbox_contents(run_id: str) -> tuple[list[dict], list[dict]]:
token = fstore.enter_sandbox(sandbox_root(run_id))
try:
incidents = await fstore.collection_list("incidents")
calls = await fstore.collection_list("calls")
finally:
fstore.exit_sandbox(token)
return incidents, calls
# ---------------------------------------------------------------------------
# The run itself
# ---------------------------------------------------------------------------
def _flags_for(mode: str) -> dict[str, bool]:
return {
"stt_enabled": mode == "audio",
"transcript_correction_enabled": mode == "audio",
"correlation_enabled": True,
"summaries_enabled": False,
"vocabulary_learning_enabled": False,
}
def _stored_input(call: dict) -> tuple[Optional[str], list]:
"""
The transcript + segments live extraction was handed for this call.
Not simply `transcript_corrected`: live extraction overwrites that field
with its primary scene's rewrite (intelligence.py), so on a call with
several scenes it now holds only scene 0's text. The corrector's own
output survives intact in `segments_corrected`, so rebuild from those
when they exist; otherwise correction never produced anything and live
extraction read the raw Whisper transcript.
"""
if call.get("transcript_not_speech"):
return None, [] # transcribe_call hands nothing downstream for noise
corrected = call.get("segments_corrected") or []
if corrected:
text = " ".join(str(seg.get("text") or "").strip() for seg in corrected).strip()
return (text or call.get("transcript")), corrected
return call.get("transcript"), call.get("segments") or []
def _extraction_fields(sb_call: dict) -> dict:
"""What extraction wrote onto the call doc (tags, units, location,
embedding, skip_reason, ...), minus everything correlation wrote. A reuse
run restores these so the orphan sweep, which reads them straight off the
call doc, sees what it saw in the source run."""
return {
k: v for k, v in sb_call.items()
if (k in _DERIVED or k.startswith("chatter_classifier_"))
and k not in _TRANSCRIPT_FIELDS
and k not in ("scenes", "incident_id", "incident_ids", "intelligence_started_at")
}
def _sandbox_seed(call: dict, mode: str) -> dict:
keep_transcript = mode in ("transcripts", "reuse")
seed = {}
for k, v in call.items():
if k in _TRANSCRIPT_FIELDS:
if keep_transcript:
seed[k] = v
continue
if k in _DERIVED or k.startswith(_DERIVED_PREFIXES):
continue
seed[k] = v
# Calls are seeded ahead of the replay clock (see PREFETCH). The orphan
# re-correlation sweep selects status=="ended" calls by ended_at, so a
# seeded call keeping its real status would be swept up as an "orphan"
# before its own turn. Its real status is restored when it is processed.
seed["status"] = "replay_pending"
return seed
async def _prepare(call: dict, mode: str, source_scenes: dict[str, dict]) -> dict:
"""
Everything per call that doesn't depend on other calls: seed the sandbox
doc, then transcribe and/or extract. Runs ahead of correlation.
Returns {"transcript", "scenes", "skip"} for the in-order stage.
"""
from app.internal import intelligence, talkgroups, transcription
call_id = call["call_id"]
await fstore.doc_set("calls", call_id, _sandbox_seed(call, mode), merge=False)
talkgroup_name = await talkgroups.resolve(
call.get("system_id"), call.get("talkgroup_id"),
hint=call.get("talkgroup_name"), call_doc=call,
)
transcript: Optional[str] = None
segments: list = []
if mode == "audio":
if call.get("audio_gcs_uri"):
transcript, segments = await transcription.transcribe_call(
call_id, call["audio_gcs_uri"], talkgroup_name,
system_id=call.get("system_id"), talkgroup_id=call.get("talkgroup_id"),
)
else:
transcript, segments = _stored_input(call)
if mode == "reuse":
src = source_scenes.get(call_id)
if src is None:
return {"skip": "not_in_source_run", "talkgroup_name": talkgroup_name}
if src.get("call_fields"):
# skip_reason gates upload.py's no-scene fallback and the orphan
# sweep correlates from tags/units/location on the call doc, so
# extraction's call-level output comes across with its scenes.
await fstore.doc_set("calls", call_id, src["call_fields"])
return {"transcript": transcript, "scenes": src.get("scenes") or [],
"talkgroup_name": talkgroup_name}
scenes: list = []
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=call.get("talkgroup_id"), system_id=call.get("system_id"),
segments=segments, node_id=call.get("node_id"),
)
return {"transcript": transcript, "scenes": scenes, "talkgroup_name": talkgroup_name}
async def _sweeps_until(t: datetime, state: dict) -> None:
"""Run the live periodic sweeps (idle auto-resolve, orphan re-correlation) at every tick up to t."""
from app.internal import recorrelation_sweep, summarizer
interval = timedelta(minutes=settings.summary_interval_minutes)
if state["last_sweep"] is None:
state["last_sweep"] = t
return
while state["last_sweep"] + interval <= t:
state["last_sweep"] += interval
tok = clock.pin(state["last_sweep"])
try:
await summarizer._resolve_stale_incidents()
await recorrelation_sweep._run_sweep_pass()
finally:
clock.unpin(tok)
async def _run(run_id: str, org_id: str, calls: list[dict], mode: str,
source_run_id: Optional[str]) -> None:
from app.routers.upload import _extract_and_correlate
global _active_run_id
progress = {"total": len(calls), "done": 0, "errors": 0, "skipped": 0,
"extractions": 0, "audio_minutes": 0.0}
errors: list[str] = []
status = "done"
source_scenes: dict[str, dict] = {}
if mode == "reuse" and source_run_id:
rows = await fstore.collection_list(_scenes_coll(source_run_id))
source_scenes = {r["call_id"]: r for r in rows if r.get("call_id")}
sb_token = fstore.enter_sandbox(sandbox_root(run_id))
fl_token = force_flags(_flags_for(mode))
ai_failures: list = []
ai_token = ai_health.collect_sandbox_failures(ai_failures)
try:
sem = asyncio.Semaphore(PREFETCH)
async def prep(call: dict):
async with sem:
tok = clock.pin(_pipeline_time(call))
try:
return await _prepare(call, mode, source_scenes)
finally:
clock.unpin(tok)
pending: dict[int, asyncio.Task] = {}
sweep_state = {"last_sweep": None}
last_t = None
for i, call in enumerate(calls):
for j in range(i, min(i + PREFETCH * 2, len(calls))):
if j not in pending:
pending[j] = asyncio.create_task(prep(calls[j]))
if run_id in _cancel:
status = "cancelled"
break
fatal = _fatal_ai_failure(ai_failures)
if fatal:
# An unfunded or retired model fails every call the same way;
# finishing the run would only produce a sandbox of orphans
# that looks like a correlation result and isn't one.
status = "failed"
errors.append(f"aborted: {fatal}")
break
t = _pipeline_time(call)
last_t = t
try:
prepared = await pending.pop(i)
await _sweeps_until(t, sweep_state)
if prepared.get("skip"):
progress["skipped"] += 1
else:
tok = clock.pin(t)
try:
await fstore.doc_set("calls", call["call_id"], {
"status": call.get("status") or "ended",
"intelligence_started_at": t.isoformat(),
})
_, _, scenes = await _extract_and_correlate(
call_id=call["call_id"],
node_id=call.get("node_id"),
system_id=call.get("system_id"),
talkgroup_id=call.get("talkgroup_id"),
talkgroup_name=prepared["talkgroup_name"],
transcript=prepared["transcript"],
scenes=prepared["scenes"],
)
finally:
clock.unpin(tok)
# Kept whole (embedding included) so a later "reuse" run
# can correlate from exactly these scenes.
sb_call = await fstore.doc_get("calls", call["call_id"]) or {}
await fstore.doc_set(_scenes_coll(run_id), call["call_id"], {
"call_id": call["call_id"],
"scenes": scenes,
"call_fields": _extraction_fields(sb_call),
}, merge=False)
if prepared["transcript"] and mode != "reuse":
progress["extractions"] += 1
if mode == "audio":
progress["audio_minutes"] += _duration_s(call) / 60
except Exception as e:
progress["errors"] += 1
if len(errors) < 20:
errors.append(f"{call.get('call_id')}: {type(e).__name__}: {e}"[:300])
logger.warning(f"Replay {run_id}: call {call.get('call_id')} failed: {e}")
progress["done"] = i + 1
if (i + 1) % 25 == 0:
await fstore.doc_set(RUNS, run_id, {"progress": dict(progress), "errors": errors})
for task in pending.values():
task.cancel()
if status == "done" and last_t is not None:
# Let every incident age out exactly as it would have live.
await _sweeps_until(
last_t + timedelta(minutes=settings.incident_auto_resolve_minutes
+ 2 * settings.summary_interval_minutes),
sweep_state,
)
incidents = await fstore.collection_list("incidents")
sb_calls = await fstore.collection_list("calls")
metrics = compute_metrics(incidents, sb_calls)
metrics["est_cost_usd"] = _running_cost(progress, metrics, mode)
metrics["ai_failures"] = dict(Counter(f"{f['tier']}: {f['problem']}" for f in ai_failures))
except Exception as e:
status = "failed"
errors.append(f"run: {type(e).__name__}: {e}"[:300])
metrics = None
logger.error(f"Replay {run_id} failed: {e}")
finally:
ai_health._sandbox_failures.reset(ai_token)
unforce_flags(fl_token)
fstore.exit_sandbox(sb_token)
_cancel.discard(run_id)
_active_run_id = None
await fstore.doc_set(RUNS, run_id, {
"status": status,
"progress": progress,
"errors": errors,
"metrics": metrics,
"finished_at": datetime.now(timezone.utc).isoformat(),
})
logger.info(f"Replay {run_id} {status}: {progress}")
FATAL_AFTER = 5
def _fatal_ai_failure(failures: list) -> Optional[str]:
"""A tier that failed permanently (no credit, dead model) FATAL_AFTER times."""
permanent = Counter(
f"{f['tier']} ({f['provider']} {f['model']}): {f['problem']}"
for f in failures if f.get("permanent")
)
for what, n in permanent.items():
if n >= FATAL_AFTER:
return what
return None
def _running_cost(progress: dict, metrics: dict, mode: str) -> float:
usd = progress["audio_minutes"] * USD_WHISPER_PER_MIN
if mode == "audio":
usd += progress["extractions"] * USD_PER_CORRECTION
usd += progress["extractions"] * (USD_PER_EXTRACTION + USD_PER_GEOCODE / 3)
usd += metrics.get("llm_decisions", 0) * USD_PER_LLM_CORRELATE
return round(usd, 2)
# ---------------------------------------------------------------------------
# Scoring
# ---------------------------------------------------------------------------
def compute_metrics(incidents: list[dict], calls: list[dict]) -> dict:
"""
The numbers that say whether incidents are being tracked, from one run's
sandbox. Same questions every correlation review has asked by hand, so two
runs over the same range compare directly.
"""
sizes = [len(i.get("call_ids") or []) for i in incidents]
resolved_via = Counter(
(i.get("resolved_via") or ("unknown" if i.get("status") == "resolved" else "still_active"))
for i in incidents
)
corr_path: Counter = Counter()
consensus: Counter = Counter()
for c in calls:
scenes = c.get("scenes") or {}
records = [s.get("corr_debug") or {} for s in scenes.values()] if scenes else [c]
for r in records:
corr_path[r.get("corr_path") or "none"] += 1
consensus[r.get("corr_consensus") or "none"] += 1
linked = sum(1 for c in calls if c.get("incident_ids"))
llm = sum(n for k, n in consensus.items() if k not in ("none", "rules_only"))
return {
"calls": len(calls),
"calls_linked": linked,
"calls_orphaned": len(calls) - linked,
"incidents": len(incidents),
"single_call_incidents": sum(1 for s in sizes if s == 1),
"single_call_pct": round(100 * sum(1 for s in sizes if s == 1) / len(sizes), 1) if sizes else None,
"median_calls_per_incident": statistics.median(sizes) if sizes else None,
"max_calls_in_incident": max(sizes) if sizes else None,
"incidents_with_units_cleared": sum(1 for i in incidents if i.get("units_cleared")),
"incidents_with_coords": sum(1 for i in incidents if i.get("location_coords")),
"resolved_via": dict(resolved_via),
"corr_path": dict(corr_path),
"corr_consensus": dict(consensus),
"llm_decisions": llm,
}
+24 -2
View File
@@ -44,11 +44,33 @@ def _safe_audio_filename(filename: str, call_id: str) -> str:
The original extension is preserved only if it's a known audio type.
"""
ext = os.path.splitext(filename)[-1].lower() if filename else ""
if ext not in (".mp3", ".wav", ".ogg", ".m4a", ".aac", ".flac"):
if ext not in AUDIO_CONTENT_TYPES:
ext = ".mp3"
return f"{call_id}{ext}"
# Extension → Content-Type. The node used to send nothing but 16 kbps MP3, so
# "audio/mpeg" was hardcoded at every point audio is written or served; it now
# sends FLAC (lossless, for Whisper's benefit — see call_recorder.py's AUDIO_*
# constants) and a stored object mislabelled audio/mpeg will not play in a
# browser. Old .mp3 objects keep working: the map is keyed off the real
# extension, not off what the current node happens to produce.
AUDIO_CONTENT_TYPES = {
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".ogg": "audio/ogg",
".m4a": "audio/mp4",
".aac": "audio/aac",
}
def content_type_for(name: str) -> str:
"""Content-Type for a stored audio object, by extension. Defaults to MP3."""
ext = os.path.splitext(name or "")[-1].lower()
return AUDIO_CONTENT_TYPES.get(ext, "audio/mpeg")
async def upload_audio(data: bytes, filename: str, call_id: str = "") -> Optional[str]:
"""Upload audio bytes to GCS and return the canonical gs:// URI, or None if disabled."""
if not settings.gcs_bucket:
@@ -65,7 +87,7 @@ async def upload_audio(data: bytes, filename: str, call_id: str = "") -> Optiona
else:
client = storage.Client()
blob = client.bucket(settings.gcs_bucket).blob(blob_path)
blob.upload_from_string(data, content_type="audio/mpeg")
blob.upload_from_string(data, content_type=content_type_for(safe_name))
try:
await asyncio.to_thread(_upload)
+105 -10
View File
@@ -15,6 +15,39 @@ from app.internal import firestore as fstore
from app.config import settings
def _scene_sort_key(scene_index: str):
"""Numeric-first sort so a >=10-scene call's entries still read in order."""
return (0, int(scene_index)) if scene_index.isdigit() else (1, scene_index)
def _scene_text_for_incident(doc: dict, incident_id: str) -> Optional[str]:
"""
The text of `doc` (a call doc) that actually belongs to `incident_id`.
server-26#96 records, per scene, which incident_id that scene's
correlation decision resolved to (incident_correlator._apply_and_log's
`scenes.<index>.incident_id`). Use that to pick only the scene(s) of this
call that are genuinely part of this incident, joining more than one if
several scenes happened to link into the same incident.
Falls back to transcript_corrected-or-transcript when the call doc has no
`scenes` field (predates server-26#96) or — defensively — when it has one
but nothing in it names this incident_id (should not happen for a call_id
that's actually in this incident's call_ids, but silently dropping a
call's contribution to its own summary would be a worse failure mode than
falling back to the whole-call text).
"""
scenes = doc.get("scenes") or {}
matched = [
scene.get("transcript")
for _, scene in sorted(scenes.items(), key=lambda kv: _scene_sort_key(kv[0]))
if scene.get("incident_id") == incident_id and scene.get("transcript")
]
if matched:
return "\n".join(matched)
return doc.get("transcript_corrected") or doc.get("transcript")
async def summarizer_loop() -> None:
from app.internal.feature_flags import get_flags
interval = settings.summary_interval_minutes * 60
@@ -25,9 +58,14 @@ async def summarizer_loop() -> None:
flags = await get_flags()
if flags["summaries_enabled"]:
await _run_summary_pass()
await _resolve_stale_incidents()
else:
logger.info("Summaries disabled — skipping summary pass and stale incident sweep")
logger.info("Summaries disabled — skipping summary pass")
# Deliberately outside the flag. Auto-resolving a quiet incident is
# pure Firestore with no model call in it, and gating it behind the
# AI kill switch meant nothing ever auto-resolved in the standing
# flags-off configuration — leaving every incident "active" forever
# and growing the candidate set every correlation reads.
await _resolve_stale_incidents()
except Exception as e:
logger.error(f"Summarizer pass failed: {e}")
@@ -43,20 +81,45 @@ async def _run_summary_pass() -> None:
async def _summarize_incident(inc: dict) -> None:
from app.internal.feature_flags import get_flags
incident_id = inc.get("incident_id")
if not incident_id:
return
flags = await get_flags()
if not flags["summaries_enabled"]:
logger.info(f"Summaries disabled — skipping summary for incident {incident_id}")
return
call_ids: list[str] = inc.get("call_ids", [])
if not call_ids:
return
# Fetch transcripts for all calls in this incident
# Fetch transcripts for all calls in this incident.
#
# server-26#114: a call links into an incident one SCENE at a time (see
# incident_correlator._apply_decision / server-26#96's `scenes` map on the
# call doc), and the same call_id can appear in more than one incident's
# call_ids — once per scene, each scene possibly landing in a different
# incident. Reading doc["transcript"] (the whole call, raw) meant an
# incident's summary was built partly on text from a DIFFERENT scene of
# that call that this incident has nothing to do with, and ignored
# transcript_corrected entirely.
#
# _scene_text_for_incident reads the specific scene(s) whose corr_debug
# recorded a link into THIS incident_id. For a call doc that predates
# this fix (no `scenes` field) it falls back to
# transcript_corrected-or-transcript — the one-liner half of #114, worth
# doing even for old-schema docs since it stops raw-transcript summaries.
transcripts: list[str] = []
for cid in call_ids:
doc = await fstore.doc_get("calls", cid)
if doc and doc.get("transcript"):
transcripts.append(doc["transcript"])
if not doc:
continue
text = _scene_text_for_incident(doc, incident_id)
if text:
transcripts.append(text)
if not transcripts:
# No transcripts yet — clear stale flag and wait for next pass
@@ -79,14 +142,41 @@ async def _summarize_incident(inc: dict) -> None:
await fstore.doc_set("incidents", incident_id, updates)
def _auto_resolve_minutes(inc: dict) -> int:
"""Quiet time before a timer close, by severity (see config: incident_auto_resolve_minutes_*)."""
sev = (inc.get("severity") or "").lower()
if sev in ("routine", "minor"):
return settings.incident_auto_resolve_minutes_routine
if sev == "moderate":
return settings.incident_auto_resolve_minutes_moderate
return settings.incident_auto_resolve_minutes
async def _expire_reopen_windows(now) -> None:
"""A timer-closed incident stops being reopenable once its window passes,
so the correlator's reopenable pool stays bounded."""
window = timedelta(minutes=settings.incident_reopen_window_minutes)
for inc in await fstore.collection_list("incidents", status="resolved", reopenable=True):
try:
updated = datetime.fromisoformat(str(inc.get("updated_at", "")).replace("Z", "+00:00"))
if updated.tzinfo is None:
updated = updated.replace(tzinfo=timezone.utc)
except ValueError:
updated = None
if updated is None or now - updated > window:
await fstore.doc_set("incidents", inc["incident_id"], {"reopenable": False})
async def _resolve_stale_incidents() -> None:
"""Auto-resolve active incidents that have had no new calls for incident_auto_resolve_minutes."""
"""Timer-close active incidents that have been quiet longer than their severity allows."""
from app.internal import clock
await _expire_reopen_windows(clock.now())
all_active = await fstore.collection_list("incidents", status="active")
if not all_active:
return
now = datetime.now(timezone.utc)
cutoff = timedelta(minutes=settings.incident_auto_resolve_minutes)
from app.internal import clock
now = clock.now()
count = 0
for inc in all_active:
@@ -100,8 +190,13 @@ async def _resolve_stale_incidents() -> None:
if updated_dt.tzinfo is None:
updated_dt = updated_dt.replace(tzinfo=timezone.utc)
idle_minutes = (now - updated_dt).total_seconds() / 60
if idle_minutes > settings.incident_auto_resolve_minutes:
await fstore.doc_set("incidents", incident_id, {"status": "resolved"})
if idle_minutes > _auto_resolve_minutes(inc):
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": now.isoformat(),
"resolved_via": "idle_timeout",
"reopenable": True,
})
from app.internal.incident_correlator import maybe_resolve_parent
await maybe_resolve_parent(incident_id)
logger.info(
+77
View File
@@ -0,0 +1,77 @@
"""
Talkgroup name resolution.
C2 owns the `systems` collection, and a system's config carries the full
talkgroup table — id and human name for every channel the node scans. The edge
node only knows the name when OP25 happened to have it in the loaded tags file,
so `tgid_name` on a call_start, and the `talkgroup_name` form field on /upload,
are both frequently empty for a talkgroup C2 can name perfectly well.
This resolver is the single place that closes that gap. `mqtt_handler` had its
own copy of the lookup on the call_start path, so calls got a name written to
their document while the /upload path — the one that drives transcription,
correlation and, critically, the incident *title* — kept whatever empty string
the node sent. The result was 84 of 100 incidents named "Ems — TGID 9048"
instead of "Ems — Ossining Police Dispatch" (server-26#34).
Order of preference: whatever the caller was given, then the call document
(written at call_start), then the system config. Returns None when nothing
knows the name, so callers keep their existing "TGID {id}" fallback.
"""
from typing import Optional
from app.internal import firestore as fstore
from app.internal.logger import logger
async def name_from_system(system_id: Optional[str], talkgroup_id: Optional[int]) -> Optional[str]:
"""Look a talkgroup's name up in its system's config. None if unknown."""
if not system_id or talkgroup_id is None:
return None
try:
tgid_int = int(talkgroup_id)
except (TypeError, ValueError):
return None
system_doc = await fstore.doc_get_cached("systems", system_id)
if not system_doc:
return None
for tg in system_doc.get("config", {}).get("talkgroups", []):
try:
if int(tg.get("id", -1)) == tgid_int:
return tg.get("name") or None
except (TypeError, ValueError):
continue
return None
async def resolve(
system_id: Optional[str],
talkgroup_id: Optional[int],
hint: Optional[str] = None,
call_doc: Optional[dict] = None,
) -> Optional[str]:
"""
Best available human name for a talkgroup.
`hint` is whatever the caller already had (OP25 metadata, a form field).
`call_doc` is an already-fetched call document, if the caller has one —
passing it avoids a second read.
"""
if hint:
return hint
if call_doc:
from_doc = call_doc.get("talkgroup_name")
if from_doc:
return from_doc
resolved = await name_from_system(system_id, talkgroup_id)
if resolved:
logger.info(
f"Resolved talkgroup name from system config: "
f"TGID {talkgroup_id} → {resolved!r}"
)
return resolved
@@ -0,0 +1,378 @@
"""
Transcript correction — the second opinion on what was said.
Whisper hears a P25 vocoder through a narrowband channel and guesses at proper
nouns it has no reason to know: street names, business names, unit call signs.
It guesses confidently, so the output reads like speech and is wrong in exactly
the places that matter downstream — "Cool Parts, Illinois" and "Shout out to
Optum" both became incident locations.
Correction used to be a line in intelligence.py's EXTRACTION_PROMPT, which put
it in the wrong place twice over (server-26#36): the same model call that
extracted units, location and severity emitted the correction *afterwards*, so
extraction reasoned over uncorrected text; and it sat behind
`correlation_enabled`, so during a cost-controlled STT-only window nothing was
ever corrected at all. It belongs here, between transcription and everything
that consumes a transcript.
WHY A SEPARATE PASS AND NOT A WHISPER PROMPT: Whisper treats its prompt as
preceding transcript text and will happily continue a pattern it finds there —
an enumerated ten-code prompt made it emit "10-4. 10-5. 10-6. …" over silence
(see transcription.py). Vocabulary can never be a transcription prior. A
corrector that receives an already-produced transcript plus a reference list has
no series to extend; it can only substitute what it was given.
SCOPE RESOLUTION: reference data is merged from the talkgroup and the system,
**talkgroup first**. The specific beats the general — a system spanning several
counties may have one talkgroup covering a single municipality, and that
municipality's streets must not be buried under a county-wide list. A
single-municipality system is the degenerate case: populate the system level and
every talkgroup inherits it.
THE PROMPT'S OWN RULES ARE NOT ENFORCED (server-26#162). "Do NOT expand
ten-codes" and "NEVER add information" are instructions to the model, not
checks on its output — `correct()` used to accept `raw["corrected"]` verbatim.
Caught live: the same call came back with "10-7" rewritten to "10-13" in one
place and "10-4" in another, and "7" expanded into "ShotSpotter" — a real code
swapped for a different real code reads exactly as confident and trustworthy
as a correct one, which is worse than leaving the raw mishearing in place. The
model isn't graded on this at write time; `_code_tokens()` is a
verify-what-you-can-cheaply-check backstop, not a fix to the model's judgment:
it only catches a code-shaped token changing, not a wrong word substituted for
another equally plausible word.
"""
import asyncio
import json
import re
from typing import Any, Optional
from app.config import settings
from app.internal import area_context
from app.internal import firestore as fstore
from app.internal import place_verifier
from app.internal.logger import logger
# A transcript this short has no proper nouns to get wrong — "10-4.", "6-2,
# stand by." — and 9 of 29 calls in the 2026-08-23 sample sat at or under this.
# Skipping them is most of the cost saving for none of the value.
MIN_WORDS_FOR_CORRECTION = 4
_PROMPT = """You are correcting a police/fire radio transcript produced by an automatic speech recogniser.
The recogniser hears a low-bitrate vocoded radio channel. It reliably mishears proper nouns — street names, business names, town names, unit call signs — and substitutes common words that sound similar. Your job is to put back what was almost certainly said.
{context_block}
Rules:
- Change ONLY what is likely a mishearing. If a phrase is already plausible radio traffic, leave it exactly as it is.
- Prefer a name from the reference lists above when the transcript contains something that sounds like it. That is the entire point of this pass.
- NEVER add information. No new sentences, no invented units, no addresses that are not implied by the audio's own words.
- Keep radio language as radio language. Do NOT expand ten-codes or signals into plain English: "10-4" stays "10-4".
- Keep the speaker's structure and order. This is not a rewrite or a summary.
- If the text is clearly not speech at all — a counting run like "10-11. 10-12. 10-13.", or one phrase repeating many times over static — set not_speech to true.
Return JSON:
corrected: the corrected transcript, or null if nothing needed changing
segments: REQUIRED when numbered transmissions are given below — the corrected
text for each one, as an array of exactly the same length and order.
Never merge, split, reorder or drop a transmission; an unchanged one
is returned verbatim. Omit this field entirely when no transmissions
are numbered.
not_speech: true if this is recogniser noise rather than a transmission
changed: list of ["heard" -> "corrected"] pairs you applied, for audit
locations: every place name in your corrected output, exactly as it appears
there — streets, intersections, businesses, schools, towns,
landmarks. Include ones you are unsure of; that is the point.
A unit call sign or a person's name is NOT a location.
{transcript}"""
def _render_input(text: str, segments: Optional[list[dict]]) -> str:
"""Numbered transmissions when we have them, so corrections stay aligned."""
if segments and len(segments) > 1:
lines = [f"{i + 1}. {s.get('text', '')}" for i, s in enumerate(segments)]
body = "\n".join(lines)
return f"Transmissions ({len(segments)}):\n{body}"
return f"Transcript:\n{text}"
def _dedupe(items: list[str]) -> list[str]:
"""Preserve order, drop case-insensitive duplicates."""
seen: set[str] = set()
out: list[str] = []
for item in items:
key = (item or "").strip().lower()
if key and key not in seen:
seen.add(key)
out.append(item.strip())
return out
# Ten-codes ("10-4"), unit/signal shorthand ("4-2"), and the digit-group
# fragments radio traffic reads out loud ("7-2-1" of a case number) all share
# this shape. The guard below does not need to know which of those a given
# token is — it only needs the SET of them to survive a "correction"
# unchanged, in order. A model rewriting "10-7" as "10-13" is not the kind of
# mishearing this pass exists to fix (server-26#162).
_CODE_TOKEN_RE = re.compile(r"\b\d{1,3}(?:-\d{1,3})+\b")
def _code_tokens(text: str) -> list[str]:
return _CODE_TOKEN_RE.findall(text or "")
def _talkgroup_entry(system_doc: dict, talkgroup_id: Optional[int]) -> dict:
"""The config.talkgroups[] entry for this talkgroup, or {}."""
if talkgroup_id is None:
return {}
try:
wanted = int(talkgroup_id)
except (TypeError, ValueError):
return {}
for tg in (system_doc.get("config") or {}).get("talkgroups", []) or []:
try:
if int(tg.get("id", -1)) == wanted:
return tg
except (TypeError, ValueError):
continue
return {}
def _area_lines(area: dict) -> list[str]:
"""
Render a merged area_context as prompt lines. Empty when nothing is set.
One block, not one per scope: by the time this runs the two scopes have
already been merged with talkgroup ahead of system, and showing the model
two competing lists invites it to pick from the wrong one.
"""
if not area:
return []
lines: list[str] = []
place = ", ".join(
str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f)
)
if place:
lines.append(f"Area covered by this channel: {place}")
knowledge = area.get("local_knowledge") or []
if knowledge:
lines.append("Local names heard on this channel:")
lines.extend(
f" {e['term']} — {e['meaning']}" if e.get("meaning") else f" {e['term']}"
for e in knowledge
)
return lines
async def resolve_context(system_id: Optional[str], talkgroup_id: Optional[int]) -> dict:
"""
Merge the reference data a corrector needs, talkgroup ahead of system.
Returns {"vocabulary", "ten_codes", "area_lines", "area", "system_area",
"tg_area"}. Empty everywhere is legitimate — a system nobody has configured
yet. The two raw scopes come back alongside the merge because the place
verifier needs them to pick an anchor (server-26#37).
"""
empty: dict[str, Any] = {
"vocabulary": [], "ten_codes": {}, "area_lines": [],
"area": {}, "system_area": {}, "tg_area": {},
}
if not system_id:
return empty
system_doc = await fstore.doc_get_cached("systems", system_id)
if not system_doc:
return empty
tg = _talkgroup_entry(system_doc, talkgroup_id)
# Talkgroup terms first so they survive any downstream truncation.
vocabulary = _dedupe(
list(tg.get("vocabulary") or []) + list(system_doc.get("vocabulary") or [])
)
# Ten-codes: system-wide reference, with talkgroup entries overriding a
# code that means something different on this channel.
ten_codes = dict(system_doc.get("ten_codes") or {})
ten_codes.update(tg.get("ten_codes") or {})
system_area = system_doc.get("area_context") or {}
tg_area = tg.get("area_context") or {}
area = area_context.effective(system_area, tg_area)
return {
"vocabulary": vocabulary,
"ten_codes": ten_codes,
"area_lines": _area_lines(area),
"area": area,
"system_area": system_area,
"tg_area": tg_area,
}
def build_context_block(context: dict, talkgroup_name: Optional[str]) -> str:
"""Render resolved context into the prompt's reference section."""
lines: list[str] = []
if talkgroup_name:
lines.append(f"Channel: {talkgroup_name}")
lines.extend(context.get("area_lines") or [])
vocabulary = context.get("vocabulary") or []
if vocabulary:
lines.append("Known local names and terms: " + ", ".join(vocabulary))
ten_codes = context.get("ten_codes") or {}
if ten_codes:
rendered = ", ".join(f"{code}={meaning}" for code, meaning in sorted(ten_codes.items()))
lines.append(f"Ten-codes used on this system: {rendered}")
return ("\n".join(lines) + "\n") if lines else ""
def _sync_gemini(model_name: str, prompt: str) -> dict:
import google.generativeai as genai # lazy import — only when needed
genai.configure(api_key=settings.gemini_api_key)
model = genai.GenerativeModel(
model_name,
generation_config={"response_mime_type": "application/json"},
)
return json.loads(model.generate_content(prompt).text)
async def correct(
call_id: str,
text: str,
segments: Optional[list[dict]] = None,
system_id: Optional[str] = None,
talkgroup_id: Optional[int] = None,
talkgroup_name: Optional[str] = None,
) -> tuple[Optional[str], Optional[list[dict]], bool]:
"""
Second-opinion pass over a transcript.
Returns (corrected_text, corrected_segments, not_speech). ``None`` for
either correction means "no change" — the corrector found nothing to fix,
could not run, or returned segments that did not line up. Callers keep the
original in that case; correction is an improvement, never a dependency.
Segments matter as much as the joined text: intelligence.py builds its
extraction prompt from NUMBERED SEGMENTS whenever there is more than one,
so a correction that only fixed the joined transcript would never reach the
model on exactly the multi-transmission calls that carry the most content.
"""
if not settings.gemini_api_key or not settings.transcript_correction_enabled:
return None, None, False
if len((text or "").split()) < MIN_WORDS_FOR_CORRECTION:
return None, None, False
context = await resolve_context(system_id, talkgroup_id)
prompt = _PROMPT.format(
context_block=build_context_block(context, talkgroup_name),
transcript=_render_input(text, segments),
)
try:
raw = await asyncio.to_thread(
_sync_gemini, settings.transcript_correction_model, prompt
)
except Exception as e:
# Never fail the transcript over a failed correction — the raw text is
# still worth having. ai_health reporting is the caller's business.
logger.warning(f"Transcript correction failed for call {call_id}: {e}")
return None, None, False
not_speech = bool(raw.get("not_speech"))
corrected = raw.get("corrected")
if not isinstance(corrected, str) or not corrected.strip():
corrected = None
elif corrected.strip() == (text or "").strip():
corrected = None
# Segment alignment is non-negotiable: scene extraction maps scenes back to
# transmissions by INDEX (segment_indices), so a returned array of the wrong
# length would silently attribute the wrong audio to a scene. Wrong length,
# wrong type, or any non-string entry and the segments are discarded whole —
# the joined correction still stands.
corrected_segments: Optional[list[dict]] = None
if segments and len(segments) > 1:
returned = raw.get("segments")
if (
isinstance(returned, list)
and len(returned) == len(segments)
and all(isinstance(x, str) for x in returned)
):
corrected_segments = [
{**seg, "text": new.strip() or seg.get("text", "")}
for seg, new in zip(segments, returned)
]
if all(s["text"] == o.get("text") for s, o in zip(corrected_segments, segments)):
corrected_segments = None
elif returned is not None:
logger.warning(
f"Transcript correction for call {call_id} returned "
f"{len(returned) if isinstance(returned, list) else type(returned).__name__} "
f"segment(s) against {len(segments)} — discarding segment corrections"
)
# Maps has the last word on place names (server-26#37). The corrector can
# only match against the list it was handed, so a plausible-sounding invention
# — "Cool Parts, Illinois" — reads exactly like a real street to it. The
# verifier geocodes each location noun against the talkgroup's anchor and,
# on a miss, looks for a sound-alike that does resolve there. It runs on the
# corrected copy so it judges the text everything downstream will actually
# read, and it skips entirely when there is no discriminating anchor.
if not not_speech:
locations = [x for x in (raw.get("locations") or []) if isinstance(x, str)]
try:
verified_text, verified_segments = await place_verifier.verify(
call_id,
corrected or text,
corrected_segments or segments,
locations,
context.get("system_area"),
context.get("tg_area"),
system_id=system_id,
talkgroup_id=talkgroup_id,
)
except Exception as e:
logger.warning(f"Place verification failed for call {call_id}: {e}")
verified_text, verified_segments = None, None
if verified_text:
corrected = verified_text
if verified_segments:
corrected_segments = verified_segments
# server-26#162: a code-shaped token ("10-7", "4-2", a case-number
# fragment like "7-2-1") changing at all — not just going missing, any
# change — means the model touched something this pass has no business
# touching. Reject that half of the correction outright rather than trust
# a rewrite that already broke its own instructions once. Checked against
# the ORIGINAL text/segment, not each other, so a joined-text correction
# and a segment correction are judged independently, same as everywhere
# else in this function.
if corrected is not None and _code_tokens(corrected) != _code_tokens(text):
logger.warning(
f"Transcript correction for call {call_id} changed code-shaped "
f"tokens ({_code_tokens(text)} -> {_code_tokens(corrected)}) — "
f"discarding the joined correction"
)
corrected = None
if corrected_segments is not None:
for seg, orig in zip(corrected_segments, segments or []):
if _code_tokens(seg["text"]) != _code_tokens(orig.get("text", "")):
logger.warning(
f"Transcript correction for call {call_id} changed "
f"code-shaped tokens in a segment — discarding segment corrections"
)
corrected_segments = None
break
if corrected or corrected_segments or not_speech:
changed = raw.get("changed") or []
logger.info(
f"Transcript correction ({settings.transcript_correction_model}): call {call_id} "
f"not_speech={not_speech} segments={'yes' if corrected_segments else 'no'} "
f"changes={changed if isinstance(changed, list) else '?'}"
)
return corrected, corrected_segments, not_speech
+122 -27
View File
@@ -11,6 +11,9 @@ import os
from typing import Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import ai_health
from app.internal import transcript_correction
from app.config import settings
# Whisper treats `prompt` as preceding transcript text, not instructions.
# Writing it as actual radio speech primes the vocabulary toward P25 codes
@@ -93,12 +96,12 @@ def _is_degenerate(text: str, segments: list[dict]) -> bool:
return False
_billing_reported = False
def _log_transcribe_failure(call_id: str, exc: Exception) -> None:
async def _log_transcribe_failure(call_id: str, exc: Exception) -> None:
"""
Log a transcription failure, escalating an unpayable account to ERROR once.
Log a transcription failure, escalating a permanent condition to ERROR
once (via app.internal.ai_health, which also drives the /health/ai
endpoint and the Discord degradation alert) and reporting it to the
shared registry either way.
Transcription failing returns None and the pipeline carries on by design, so
a per-call WARNING is invisible: no transcript means no extraction, which
@@ -110,23 +113,41 @@ def _log_transcribe_failure(call_id: str, exc: Exception) -> None:
The same failure mode already bit the Gemini correlator twice (a retired
model ID, then a depleted balance), which is why this is worth the code.
"""
global _billing_reported
text = str(exc)
low = text.lower()
kind = ai_health.classify(text)
if ("insufficient_quota" in low or "billing" in low
or "credit" in low or "exceeded your current quota" in low):
if not _billing_reported:
_billing_reported = True
logger.error(
"Transcription: the OpenAI account cannot be billed -- EVERY call is "
"now stored with no transcript, so extraction, correlation and "
"incidents are all dead downstream. Top up at "
f"https://platform.openai.com/settings/organization/billing. API said: {text}"
)
if kind == "billing":
problem = "the OpenAI account cannot be billed"
fix = "top up at https://platform.openai.com/settings/organization/billing"
logger.error(
"Transcription: the OpenAI account cannot be billed -- EVERY call is "
"now stored with no transcript, so extraction, correlation and "
"incidents are all dead downstream. Top up at "
f"https://platform.openai.com/settings/organization/billing. API said: {text}"
)
await ai_health.report_degraded(
"transcription", "openai", settings.stt_model, problem, fix, permanent=True
)
return
if kind == "dead_model":
problem = "the STT model is unavailable"
fix = "update STT_MODEL in config.py"
logger.error(
f"Transcription: the configured model ({settings.stt_model!r}) is unavailable "
"-- EVERY call is now stored with no transcript, so extraction, correlation "
f"and incidents are all dead downstream. Update STT_MODEL in config.py. API said: {text}"
)
await ai_health.report_degraded(
"transcription", "openai", settings.stt_model, problem, fix, permanent=True
)
return
logger.warning(f"Transcription failed for call {call_id}: {text}")
await ai_health.report_degraded(
"transcription", "openai", settings.stt_model,
"transient API error", "no action needed unless this persists", permanent=False,
)
async def transcribe_call(
@@ -134,6 +155,7 @@ async def transcribe_call(
gcs_uri: str,
talkgroup_name: Optional[str] = None,
system_id: Optional[str] = None,
talkgroup_id: Optional[int] = None,
) -> tuple[Optional[str], list[dict]]:
"""
Transcribe audio at the given GCS URI and store the result in Firestore.
@@ -146,34 +168,107 @@ async def transcribe_call(
return None, []
try:
transcript, segments = await asyncio.to_thread(
transcript, segments, degenerate = await asyncio.to_thread(
_sync_transcribe, gcs_uri, talkgroup_name
)
# A hallucination is a coin-flip, not a property of the clip: call
# e49ea32c produced a 56-word ten-code counting run on one attempt and
# ordinary speech on the next, same audio and temperature=0. Discarding
# on the first bad roll threw away a recoverable transcript, so spend
# one more request before giving up.
if degenerate and settings.stt_retry_on_degenerate:
logger.info(f"Retrying transcription for call {call_id} after degenerate output")
transcript, segments, degenerate = await asyncio.to_thread(
_sync_transcribe, gcs_uri, talkgroup_name
)
if degenerate:
logger.warning(
f"Transcription for call {call_id} was degenerate twice — giving up"
)
except Exception as e:
_log_transcribe_failure(call_id, e)
await _log_transcribe_failure(call_id, e)
return None, []
# No exception means the provider call itself succeeded (this also
# covers transcripts discarded as degenerate/hallucinated output —
# that's a filtering decision, not a provider failure), so the
# transcription tier is healthy and any prior degradation clears.
await ai_health.report_healthy("transcription")
if transcript:
updates: dict = {"transcript": transcript}
if segments:
updates["segments"] = segments
# Second opinion, before anything downstream sees the text. Whisper
# mishears proper nouns confidently, and extraction/embedding/
# correlation all consume the transcript — correcting it afterwards
# (which is where it used to live, inside the extraction prompt) meant
# every one of them reasoned over known-bad text. server-26#36.
# Correction is a second model call plus a Places lookup per proposed
# location, so it is real spend that used to be reachable only through
# an env var and an ansible run. That made an "STT-only" evaluation
# window not STT-only, and its cost unattributable (server-26#76, #45).
from app.internal.feature_flags import resolve_flags
_, _ai_flag = await resolve_flags(system_id)
corrected, corrected_segments, not_speech = (None, None, False)
if _ai_flag("transcript_correction_enabled"):
corrected, corrected_segments, not_speech = await transcript_correction.correct(
call_id, transcript, segments,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
)
else:
logger.info(
f"Transcript correction disabled — saving raw transcript for call {call_id}"
)
if corrected_segments:
# Raw stays as evidence; the corrected copy is what extraction reads.
updates["segments_corrected"] = corrected_segments
if not_speech:
# The corrector sees what _is_degenerate misses — novel repetition
# shapes rather than the two it pattern-matches. Keep the raw text
# (it is evidence) but do not let it reach extraction as fact.
updates["transcript_not_speech"] = True
logger.info(
f"Corrector flagged call {call_id} as recogniser noise: {transcript[:80]!r}"
)
elif corrected:
updates["transcript_corrected"] = corrected
try:
await fstore.doc_set("calls", call_id, updates)
logger.info(
f"Transcript saved for call {call_id} "
f"({len(transcript)} chars, {len(segments)} segment(s))"
f"({len(transcript)} chars, {len(segments)} segment(s)"
f"{', corrected' if corrected and not not_speech else ''})"
)
except Exception as e:
logger.warning(f"Could not save transcript for {call_id}: {e}")
if not_speech:
return None, []
# Hand the corrected copies downstream. extract_scenes prefers numbered
# segments over the joined transcript, so returning corrected text with
# raw segments would have thrown the correction away on every call with
# more than one transmission.
return corrected or transcript, corrected_segments or segments
return transcript, segments
def _sync_transcribe(
gcs_uri: str,
talkgroup_name: Optional[str] = None,
) -> tuple[Optional[str], list[dict]]:
"""Download audio from GCS and transcribe with OpenAI Whisper."""
) -> tuple[Optional[str], list[dict], bool]:
"""Download audio from GCS and transcribe with OpenAI Whisper.
Third element is True when output was DISCARDED as degenerate, which the
caller distinguishes from ordinary silence so it can retry — the same clip
can hallucinate on one attempt and transcribe on the next.
"""
from google.cloud import storage as gcs
from google.oauth2 import service_account
from openai import OpenAI
@@ -184,7 +279,7 @@ def _sync_transcribe(
# Tuple, not a bare None: the caller unpacks two values, so returning
# None here raised a TypeError that surfaced as a misleading
# "Transcription failed" instead of the real missing-key warning.
return None, []
return None, [], False
without_scheme = gcs_uri[len("gs://"):]
bucket_name, blob_path = without_scheme.split("/", 1)
@@ -255,16 +350,16 @@ def _sync_transcribe(
text = " ".join(s["text"] for s in segments) or None
if _is_degenerate(text or "", segments):
logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
return None, []
return text, segments
return None, [], True
return text, segments, False
else:
# json format returns just {"text": "..."} — no segments or timestamps.
# Intelligence extraction falls back to treating the whole transcript as one block.
text = (response.text or "").strip() or None
if _is_degenerate(text or "", []):
logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
return None, []
return text, []
return None, [], True
return text, [], False
finally:
try:
os.unlink(tmp_path)
+118 -57
View File
@@ -20,8 +20,9 @@ import json
import random
import re
from datetime import datetime, timezone, timedelta
from typing import Optional
from typing import Any, Optional
from app.internal.logger import logger
from app.internal import area_context
from app.internal import firestore as fstore
from app.config import settings
@@ -57,26 +58,32 @@ Do NOT include common English words. Max 80 terms. Only include what you are con
accurate for this specific area; return fewer terms rather than guessing."""
_INDUCTION_PROMPT = """\
You are analyzing P25 emergency radio transcripts to find vocabulary terms that should be \
added to improve future speech-to-text accuracy for this system.
You are analyzing P25 emergency radio transcripts from ONE talkgroup (a single radio channel) \
to find local terms that should be added to improve future speech-to-text accuracy for that \
channel.
System: {system_name}
Existing approved vocabulary (do not re-propose these): {existing_vocab}
Channel: {talkgroup_name}
Area: {area_hint}
Terms this channel already knows (do not re-propose these): {existing_vocab}
Sampled transcripts:
{transcript_block}
Find terms that are LIKELY STT errors or local terms missing from the vocabulary:
Find terms that are LIKELY STT errors or local terms missing from the list:
- Unit IDs that appear garbled (e.g. "5 acre" → "5-baker")
- Agency acronyms spelled out phonetically (e.g. "why vac" → "YVAC")
- Street names or locations that look misspelled or oddly transcribed
- Callsigns or local codes not yet in the vocabulary
- Callsigns or local codes not yet known
Return ONLY a JSON object:
{{"new_terms": ["term1", "term2", ...]}}
{{"new_terms": [{{"term": "YVAC", "meaning": "Yorktown Volunteer Ambulance Corps"}}, ...]}}
Only include high-confidence additions not already in existing vocabulary.
Return {{"new_terms": []}} if nothing new is found."""
`meaning` is what the term refers to — an agency, a road, a unit type. Omit it or use null \
when you genuinely do not know; a term with no meaning is still worth proposing.
Only propose what is specific to THIS channel and this area. Do not propose a term just \
because it appears often. Return {{"new_terms": []}} if nothing new is found."""
# ─────────────────────────────────────────────────────────────────────────────
@@ -97,10 +104,17 @@ async def bootstrap_system_vocabulary(system_id: str) -> list[str]:
system_name = system_doc.get("name", "Unknown")
system_type = system_doc.get("type", "P25")
# Build area hint from configured talkgroup names
talkgroups = system_doc.get("config", {}).get("talkgroups", [])
tg_names = [tg.get("name", "") for tg in talkgroups if tg.get("name")][:8]
area_hint = f"Talkgroups include: {', '.join(tg_names)}" if tg_names else "Unknown area"
# Prefer the place an operator actually set. Guessing the area from talkgroup
# names is thin for a single-municipality system and close to useless for a
# multi-county one (server-26#36), so it is only the fallback now.
area = system_doc.get("area_context") or {}
place = ", ".join(str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f))
if place:
area_hint = place
else:
talkgroups = system_doc.get("config", {}).get("talkgroups", [])
tg_names = [tg.get("name", "") for tg in talkgroups if tg.get("name")][:8]
area_hint = f"Talkgroups include: {', '.join(tg_names)}" if tg_names else "Unknown area"
terms = await asyncio.to_thread(_sync_bootstrap, system_name, system_type, area_hint)
if not terms:
@@ -279,9 +293,20 @@ async def _run_induction_pass() -> None:
async def _induct_system(system_id: str, system_doc: dict) -> None:
"""Sample random transcripts for a system and propose new vocabulary."""
system_name = system_doc.get("name", "Unknown")
existing_vocab: list[str] = system_doc.get("vocabulary") or []
"""
Sample recent transcripts per TALKGROUP and propose local knowledge there.
Proposals used to land at system level, which is the wrong blast radius
(server-26#37). A wrong term on a talkgroup misleads one channel; the same
term at system level misleads every channel on that system — including one
400km away on a statewide system, which is exactly the context poisoning the
scope rule exists to prevent. If a term really does apply system-wide,
carrying it on several talkgroups costs almost nothing, while auto-promoting
a wrong one is expensive to notice. So: talkgroup-level pending terms only,
and nothing here ever promotes upward or approves itself.
"""
system_name = system_doc.get("name", "Unknown")
system_area = system_doc.get("area_context") or {}
# Fetch calls from the last 7 days only — avoids scanning the entire history.
# Active calls have ended_at=None and are excluded by the range filter automatically.
@@ -294,57 +319,87 @@ async def _induct_system(system_id: str, system_doc: dict) -> None:
if not all_calls:
return
# Random sample up to the token budget (4 chars ≈ 1 token)
random.shuffle(all_calls)
char_budget = settings.vocabulary_induction_sample_tokens * 4
by_tg: dict[Any, list[dict]] = {}
for call in all_calls:
tgid = call.get("talkgroup_id")
if tgid is None:
continue
by_tg.setdefault(tgid, []).append(call)
# The sample budget is per system, split across the talkgroups that have
# traffic — a channel with 400 calls should not starve one with 12.
char_budget = max(
(settings.vocabulary_induction_sample_tokens * 4) // max(len(by_tg), 1), 800
)
for talkgroup_id, calls in by_tg.items():
try:
await _induct_talkgroup(
system_id, system_doc, system_name, system_area,
talkgroup_id, calls, char_budget,
)
except Exception as e:
logger.warning(
f"Induction failed for talkgroup {talkgroup_id} on system {system_id}: {e}"
)
async def _induct_talkgroup(
system_id: str,
system_doc: dict,
system_name: str,
system_area: dict,
talkgroup_id: Any,
calls: list[dict],
char_budget: int,
) -> None:
tg_entry = area_context.talkgroup_entry(system_doc, talkgroup_id)
tg_area = tg_entry.get("area_context") or {}
area = area_context.effective(system_area, tg_area)
talkgroup_name = (
tg_entry.get("name")
or calls[0].get("talkgroup_name")
or f"TGID {talkgroup_id}"
)
known = area_context._known_terms(tg_entry, system_doc)
random.shuffle(calls)
transcript_block = ""
sampled_call_docs: list[dict] = []
sampled = 0
for call in all_calls:
for call in calls:
text = call.get("transcript_corrected") or call.get("transcript") or ""
if not text:
continue
if len(transcript_block) + len(text) > char_budget:
break
tg = call.get("talkgroup_name") or f"TGID {call.get('talkgroup_id', '?')}"
transcript_block += f"[{tg}] {text}\n"
transcript_block += f"{text}\n"
sampled_call_docs.append(call)
sampled += 1
if sampled < 3:
return # not enough data to learn from yet
if len(sampled_call_docs) < 3:
return # not enough data on this channel to learn from yet
new_terms = await asyncio.to_thread(
_sync_induct, system_name, existing_vocab, transcript_block
place = ", ".join(str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f))
proposed = await asyncio.to_thread(
_sync_induct,
system_name, talkgroup_name, place or "not set",
sorted(known)[:80], transcript_block,
)
if not new_terms:
if not proposed:
return
now = datetime.now(timezone.utc).isoformat()
existing_pending: list[dict] = system_doc.get("vocabulary_pending") or []
pending_lower = {p["term"].lower() for p in existing_pending}
vocab_lower = {t.lower() for t in existing_vocab}
to_queue = []
for t in new_terms:
if t.lower() in vocab_lower or t.lower() in pending_lower:
continue
to_queue.append({
"term": t,
entries = [
{
"term": p["term"],
"meaning": p.get("meaning"),
"source": "induction",
"added_at": now,
"source_call_ids": _find_source_calls(t, sampled_call_docs),
})
if not to_queue:
return
await fstore.doc_set("systems", system_id, {
"vocabulary_pending": existing_pending + to_queue,
})
logger.info(
f"Vocabulary induction: {len(to_queue)} new term(s) proposed for "
f"system {system_id} ({system_name}): {[p['term'] for p in to_queue]}"
)
"source_call_ids": _find_source_calls(p["term"], sampled_call_docs),
}
for p in proposed
if p.get("term") and p["term"].lower() not in known
]
if entries:
await area_context.add_pending(system_id, talkgroup_id, entries)
# ─────────────────────────────────────────────────────────────────────────────
@@ -441,8 +496,13 @@ def _sync_bootstrap(system_name: str, system_type: str, area_hint: str) -> list[
def _sync_induct(
system_name: str, existing_vocab: list[str], transcript_block: str
) -> list[str]:
system_name: str,
talkgroup_name: str,
area_hint: str,
existing_vocab: list[str],
transcript_block: str,
) -> list[dict]:
"""Returns [{term, meaning}] — a bare string is still accepted from the model."""
from app.config import settings as cfg
from openai import OpenAI
@@ -452,6 +512,8 @@ def _sync_induct(
vocab_str = ", ".join(existing_vocab[:80]) if existing_vocab else "(none yet)"
prompt = _INDUCTION_PROMPT.format(
system_name=system_name,
talkgroup_name=talkgroup_name,
area_hint=area_hint,
existing_vocab=vocab_str,
transcript_block=transcript_block[:8000],
)
@@ -463,8 +525,7 @@ def _sync_induct(
response_format={"type": "json_object"},
)
data = json.loads(response.choices[0].message.content)
terms = data.get("new_terms") or []
return [str(t).strip() for t in terms if str(t).strip()]
return area_context.normalize_local_knowledge(data.get("new_terms") or [])
except Exception as e:
logger.warning(f"Vocabulary induction GPT call failed: {e}")
return []
+44
View File
@@ -0,0 +1,44 @@
"""
Word error rate — server-26#163's eval harness needs a real number to compare
against, not a vibe. Standard definition: word-level Levenshtein distance
between a human-verified reference and the machine hypothesis, divided by the
reference's own word count. Case-insensitive, punctuation-insensitive — this
measures whether the right WORDS came out, not transcript formatting.
"""
import re
from typing import Optional
def _tokenize(text: str) -> list[str]:
return re.findall(r"[\w']+", (text or "").lower())
def word_error_rate(reference: str, hypothesis: str) -> Optional[float]:
"""
(substitutions + deletions + insertions) / len(reference words).
None when the reference has no words — WER is undefined there, not 0.0;
a caller that defaults a None to 0.0 would report a perfect score for a
call nobody actually transcribed.
"""
ref = _tokenize(reference)
hyp = _tokenize(hypothesis)
if not ref:
return None
if not hyp:
return 1.0
n, m = len(ref), len(hyp)
# Single-row DP over Levenshtein distance — O(n*m) time, O(m) space.
row = list(range(m + 1))
for i in range(1, n + 1):
prev_diag = row[0]
row[0] = i
for j in range(1, m + 1):
prev_row_j = row[j]
if ref[i - 1] == hyp[j - 1]:
row[j] = prev_diag
else:
row[j] = 1 + min(prev_diag, row[j], row[j - 1])
prev_diag = prev_row_j
return row[m] / n
+58 -5
View File
@@ -1,3 +1,4 @@
import os
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, Depends
@@ -8,6 +9,7 @@ from app.internal.node_sweeper import sweeper_loop
from app.internal.summarizer import summarizer_loop
from app.internal.vocabulary_learner import vocabulary_induction_loop
from app.internal.recorrelation_sweep import recorrelation_loop
from app.internal import ai_health
from app.config import settings
from app.internal.auth import (
require_firebase_token,
@@ -15,7 +17,7 @@ from app.internal.auth import (
require_node_service_or_firebase_token,
)
from app.routers import nodes, systems, calls, upload, tokens, incidents, alerts, admin, trips, places, links, users
from app.routers import enrollment, media, org, waitlist
from app.routers import enrollment, media, org, waitlist, telemetry, replay
from app.internal import dynsec
from app.internal import firestore as fstore
@@ -76,12 +78,40 @@ async def lifespan(app: FastAPI):
app = FastAPI(title="DRB C2 Core", lifespan=lifespan)
# The browser needs CORS to reach this API at all: the frontend's Archive page
# calls GET /calls/search with Authorization + Content-Type headers, which
# forces a preflight. Without this middleware the OPTIONS gets a bare 405 and
# the fetch fails (#110). allow_origins is an explicit list -- never "*" in a
# deployment -- so name every host the frontend is served from in CORS_ORIGINS.
#
# allow_credentials stays False on purpose: auth here is a Bearer header, not a
# cookie, so credentialed CORS is never needed, and keeping it False is what
# lets an explicit-origin allowlist work without Starlette's "*"-only
# restriction. "*" + credentials is the dangerous pair (Starlette reflects the
# caller's Origin back WITH Access-Control-Allow-Credentials: true); this code
# cannot produce it because credentials are hard-off.
def cors_allows_credentials(origins: list[str]) -> bool:
"""Always False -- credentialed CORS is never enabled here (Bearer auth,
not cookies). Kept as a named predicate so a future edit that wants to
turn credentials on has to go through here and confront the "*" case.
A wildcard entry would additionally be refused a credentialed response."""
return False
_cors_is_wildcard = "*" in settings.cors_origins
if _cors_is_wildcard:
logger.error(
"CORS_ORIGINS contains '*'. That is fine for local dev but is almost "
"certainly a misconfigured deployment -- set CORS_ORIGINS to your "
"frontend origin(s), e.g. [\"https://drb.cusano.net\"]."
)
app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_origins,
allow_methods=["*"],
allow_headers=["*"],
allow_credentials=True,
allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"],
allow_headers=["authorization", "content-type"],
allow_credentials=False,
)
app.include_router(nodes.router, dependencies=[Depends(require_service_or_firebase_token)])
@@ -90,6 +120,7 @@ app.include_router(nodes.router, dependencies=[Depends(require_service_or_fi
# write routes inside carry their own require_admin_token, so nodes get read
# access only.
app.include_router(systems.router, dependencies=[Depends(require_node_service_or_firebase_token)])
app.include_router(telemetry.router, dependencies=[Depends(require_node_service_or_firebase_token)])
app.include_router(calls.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(tokens.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(incidents.router, dependencies=[Depends(require_service_or_firebase_token)])
@@ -98,6 +129,7 @@ app.include_router(trips.router, dependencies=[Depends(require_service_or_fi
app.include_router(places.router, dependencies=[Depends(require_service_or_firebase_token)])
app.include_router(upload.router) # auth is per-node, handled inline
app.include_router(admin.router) # auth is per-endpoint (read: firebase, write: admin)
app.include_router(replay.router) # auth: admin only (every route spends or reads a replay run)
app.include_router(users.router) # auth: admin only
app.include_router(links.router) # auth is per-endpoint (generate: firebase, resolve: service key)
app.include_router(enrollment.router) # public; auth is the enrollment/pickup-secret tokens, checked inline
@@ -117,6 +149,27 @@ app.include_router(media.router)
# against a future /internal/* route being added and forgotten there.
# Read straight from the environment rather than through Settings: this is a
# build stamp baked in by the Dockerfile, not configuration anyone sets or
# tunes, and keeping it out of Settings avoids implying it can be changed.
_GIT_SHA = os.getenv("GIT_SHA", "unknown")
@app.get("/health")
async def health():
return {"ok": True, "mqtt_connected": mqtt_handler.is_connected}
return {
"ok": True,
"mqtt_connected": mqtt_handler.is_connected,
# CI asserts this equals the commit it just deployed. Without it a
# deploy can "succeed" while the previous container is still serving.
"git_sha": _GIT_SHA,
}
# Deliberately unauthenticated, same as /health above: the CI deploy step
# curls /health with no credentials, and this is diagnostic state (which AI
# tier is degraded and why), not a secret — no API keys or tokens appear in
# it. Keeping it auth-free means an external uptime check can watch it too.
@app.get("/health/ai")
async def health_ai():
return {"tiers": ai_health.snapshot()}
+81
View File
@@ -62,12 +62,43 @@ class NodeRecord(BaseModel):
last_seen: Optional[datetime] = None
assigned_system_id: Optional[str] = None
node_type: str = "fixed" # fixed or portable
secondary_sdr_mode: str = "none" # none | adsb | ais | op25_2 — requires a second physical SDR
sdr_count: int = 1 # self-reported by the node's checkin, best-effort
enforce_override_timeout: bool = True
is_overridden: bool = False
override_system_id: Optional[str] = None
override_timeout_at: Optional[datetime] = None
class AircraftTrack(BaseModel):
"""Live ADS-B position, one doc per icao. Overwritten on every sighting —
this is a live-map snapshot, not a history (see node-26#9)."""
icao: str
org_id: Optional[str] = None
node_id: str
callsign: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
altitude_ft: Optional[float] = None
ground_speed_kt: Optional[float] = None
track_deg: Optional[float] = None
last_seen: datetime
class VesselTrack(BaseModel):
"""Live AIS position, one doc per mmsi. Same live-snapshot shape as
AircraftTrack — overwritten on every sighting (see node-26#9)."""
mmsi: str
org_id: Optional[str] = None
node_id: str
name: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
speed_kt: Optional[float] = None
heading_deg: Optional[float] = None
last_seen: datetime
class CommandPayload(BaseModel):
action: str # discord_join / discord_leave / op25_restart
guild_id: Optional[str] = None
@@ -78,6 +109,49 @@ class CommandPayload(BaseModel):
# Systems
# ---------------------------------------------------------------------------
class LocalKnowledgeEntry(BaseModel):
"""A local name and what it is. Both scopes, one shape (server-26#36)."""
term: str
meaning: Optional[str] = None
class AreaContextBody(BaseModel):
"""
Ground truth about the area a system or talkgroup covers.
Every field is nullable on purpose: which SCOPE an operator fills is their
declaration of how homogeneous the system is. A single-municipality system
is described once at system level and inherited by every talkgroup; a
statewide one is left null there and described per talkgroup. See
`internal/area_context.py` for the merge rules and the anchor.
`center`/`radius_km`/`resolved_from`/`resolved_at` are absent here by
design — the backend geocodes and writes those. A client that sends them is
ignored.
"""
municipality: Optional[str] = None
county: Optional[str] = None
state: Optional[str] = None
local_knowledge: List[LocalKnowledgeEntry] = []
class TalkgroupEntry(BaseModel):
"""
One entry in `config.talkgroups[]`.
Declared so the talkgroup copy of `area_context` stops being unvalidated
JSON riding inside the config blob — it is the same shape as the system's
and gets the same validator (server-26#36).
"""
model_config = {"extra": "allow"}
id: int
name: str = ""
tag: str = "other"
vocabulary: List[str] = []
area_context: Optional[AreaContextBody] = None
class SystemRecord(BaseModel):
system_id: str
org_id: Optional[str] = None
@@ -85,6 +159,12 @@ class SystemRecord(BaseModel):
type: str # P25 / DMR / NBFM
config: Dict[str, Any] = {} # OP25-compatible config blob
ten_codes: Dict[str, str] = {} # {"10-10": "Commercial Alarm", ...}
# Ground truth about the area this system covers, fed to the transcript
# corrector and the place verifier (server-26#36 / #37). Shape is
# AreaContextBody plus the backend-owned anchor. Per-talkgroup overrides
# live inside config.talkgroups[] and rank ABOVE this, so a multi-county
# system narrows per channel rather than replacing this wholesale.
area_context: Dict[str, Any] = {}
class SystemCreate(BaseModel):
@@ -92,6 +172,7 @@ class SystemCreate(BaseModel):
type: str
config: Dict[str, Any] = {}
ten_codes: Dict[str, str] = {}
area_context: Dict[str, Any] = {}
# ---------------------------------------------------------------------------
+237 -12
View File
@@ -1,9 +1,10 @@
import asyncio
from datetime import datetime, timezone, timedelta
from fastapi import APIRouter, Depends, Query
from app.internal.auth import require_admin_token
from app.internal.auth import require_admin_token, require_agent_key_or_admin, describe_actor
from app.internal.feature_flags import get_flags, set_flags
from app.internal import firestore as fstore
from app.config import settings
async def _get_ai_enabled_system_ids(global_flags: dict) -> set[str]:
"""Return system_ids where at least one AI function (STT or correlation) is effectively on."""
@@ -24,26 +25,61 @@ router = APIRouter(prefix="/admin", tags=["admin"])
@router.get("/features")
async def get_feature_flags(_=Depends(require_admin_token)):
async def get_feature_flags(_=Depends(require_agent_key_or_admin)):
"""
Return the current AI feature flag state. Admin-only (SAAS_PLAN.md B2c) —
was previously any authenticated user via require_firebase_token, which
handed platform-wide AI configuration state to every signed-in viewer
regardless of org.
Also reachable with the agent service key (server-26#64) so the unattended
runbook can read the switch over HTTP instead of shelling into the
container. Note this is require_agent_key_or_admin, NOT the Discord bot's
service key — see internal/auth.py.
"""
return await get_flags()
@router.put("/features")
async def update_feature_flags(body: dict, _=Depends(require_admin_token)):
"""Update one or more AI feature flags. Admin only."""
return await set_flags(body)
async def update_feature_flags(
body: dict,
cascade: bool = Query(
False,
description=(
"Also clear per-system ai_flags overrides for the keys being set, "
"so the flip applies to every radio system."
),
),
principal: dict = Depends(require_agent_key_or_admin),
):
"""Update one or more AI feature flags. Admin or agent service key.
``cascade`` defaults to **False**, deliberately.
The tempting default is True: feature_flags.resolve_flags lets a
system-level False beat a global True, so turning AI back ON globally can
half-apply and leave a system dark, and cascade-by-default would make every
flip total. That reasoning holds only if per-system ai_flags are set
exclusively by hand. They are not — PUT /systems/{system_id}/ai-flags
(routers/systems.py) is a real admin route and drb-frontend's AiFlagsPanel
(app/systems/page.tsx) is a real toggle in the UI. So an override is a
deliberate operator decision that is visible in the interface, and
cascading by default would silently erase it on the next unrelated global
flip, with the operator's own UI still showing what they set until reload.
Silently destroying operator intent is the worse failure, so the caller
says when it means "everywhere": the runbook passes cascade=true on the
shutoff, and the admin UI (which does not pass it) keeps its per-system
overrides.
"""
return await set_flags(body, actor=describe_actor(principal), cascade=cascade)
@router.get("/debug/correlation")
async def debug_correlation(
limit: int = Query(20, ge=1, le=100),
orphan_hours: int = Query(48, ge=1, le=168),
ai_systems_only: bool = Query(False, description="Restrict to systems with STT or correlation currently enabled"),
_=Depends(require_admin_token),
):
"""
@@ -61,8 +97,54 @@ async def debug_correlation(
def _strip(doc: dict) -> dict:
return {k: v for k, v in doc.items() if k != "embedding"}
def _call_summary(call: dict) -> dict:
def _scene_summary(scene_index: str, scene: dict) -> dict:
"""
One scene's own correlation record, from the call doc's `scenes` map
(server-26#96). Same corr_* field names as _call_summary's flat
fields below, deliberately — a scene entry and a scene-less call
summary are interchangeable data points to the tally functions.
"""
corr_debug = scene.get("corr_debug") or {}
return {
"scene_index": scene_index,
"transcript": scene.get("transcript"),
"incident_id": scene.get("incident_id"),
# server-26#139: this scene's OWN incident_type/severity, as seen
# by _call_is_substanceless at decision time — not the call doc's
# flat top-level field, which is last-scene-wins (server-26#96).
"incident_type": scene.get("incident_type"),
"severity": scene.get("severity"),
"corr_path": corr_debug.get("corr_path"),
"corr_incident_idle_min": corr_debug.get("corr_incident_idle_min"),
"corr_distance_km": corr_debug.get("corr_distance_km"),
"corr_score": corr_debug.get("corr_score"),
"corr_candidates": corr_debug.get("corr_candidates"),
"corr_shared_units": corr_debug.get("corr_shared_units"),
"corr_fit_signal": corr_debug.get("corr_fit_signal"),
"corr_matched_units": corr_debug.get("corr_matched_units"),
"corr_consensus": corr_debug.get("corr_consensus"),
"corr_llm_reasoning": corr_debug.get("corr_llm_reasoning"),
"corr_llm_action": corr_debug.get("corr_llm_action"),
"corr_rules_action": corr_debug.get("corr_rules_action"),
"corr_gate_veto": corr_debug.get("corr_gate_veto"),
}
def _call_summary(call: dict) -> dict:
# server-26#96 — per-scene records, keyed by scene index as written by
# incident_correlator._apply_and_log. Present only on calls that went
# through correlation after this fix landed; absent (None) on older
# call docs, which the tally below falls back for. Sorted numerically
# so a >=10-scene call still reads in scene order.
scenes_map = call.get("scenes") or {}
scenes = [
_scene_summary(idx, s)
for idx, s in sorted(
scenes_map.items(),
key=lambda kv: (0, int(kv[0])) if kv[0].isdigit() else (1, kv[0]),
)
] or None
return {
"scenes": scenes,
"call_id": call.get("call_id"),
"started_at": call.get("started_at"),
"ended_at": call.get("ended_at"),
@@ -91,12 +173,43 @@ async def debug_correlation(
"corr_matched_units": call.get("corr_matched_units"),
"corr_sweep_count": call.get("corr_sweep_count"),
"skip_reason": call.get("skip_reason"),
# LLM consensus tier fields — written by upload.py's
# _correlate_with_consensus / llm_correlator.py, but previously
# dropped here, making it impossible to tell from this endpoint
# whether the LLM correlation tier is actually running (server-26#24).
"corr_consensus": call.get("corr_consensus"),
"corr_llm_reasoning": call.get("corr_llm_reasoning"),
"corr_llm_action": call.get("corr_llm_action"),
"corr_rules_action": call.get("corr_rules_action"),
# server-26#115 — why an llm=orphan/rules=new disagreement escalated
# to tiebreak instead of being gated (see upload.py's
# _call_is_substanceless). Present only on that disagreement shape;
# written here specifically so a live measurement window can read
# the reason instead of reconstructing it by hand from the dump.
"corr_gate_veto": call.get("corr_gate_veto"),
# server-26#127 — shadow-mode upstream chatter classifier verdict.
# Written by intelligence.extract_scenes on every transcript that
# reaches real scene extraction (not on garbage/too-short skips).
# Nothing skips extraction on this yet — it's here purely so a
# live measurement window can read the false-positive rate.
"chatter_classifier_verdict": call.get("chatter_classifier_verdict"),
"chatter_classifier_reason": call.get("chatter_classifier_reason"),
}
# ── Determine which systems have AI active ────────────────────────────────
# NOT a filter by default. Restricting to AI-enabled systems meant the view
# emptied itself the moment the flags went off — which is precisely when a
# window gets reviewed. On 2026-08-23 it dropped from 100 incidents to 6
# between switching correlation off and opening the tab. Pass
# ai_systems_only=true to get the old behaviour.
global_flags = await get_flags()
ai_systems = await _get_ai_enabled_system_ids(global_flags)
def _in_scope(system_ids: list) -> bool:
if not ai_systems_only:
return True
return any(sid in ai_systems for sid in system_ids)
# ── Fetch recent incidents (AI-enabled systems only) ──────────────────────
# Read a bounded, already-sorted window rather than the whole collection.
# This route used to pull every incident ever created and sort in Python,
@@ -115,10 +228,7 @@ async def debug_correlation(
order_by=[("updated_at", "DESCENDING")],
limit_to=window,
)
ai_incidents = [
i for i in all_incidents
if any(sid in ai_systems for sid in (i.get("system_ids") or []))
]
ai_incidents = [i for i in all_incidents if _in_scope(i.get("system_ids") or [])]
incidents = ai_incidents[:limit]
incidents_window_exhausted = len(all_incidents) >= window and len(ai_incidents) < limit
@@ -129,7 +239,14 @@ async def debug_correlation(
unique_call_ids = list(dict.fromkeys(all_call_ids)) # dedupe, preserve order
call_docs = await asyncio.gather(*(fstore.doc_get("calls", cid) for cid in unique_call_ids))
call_map: dict[str, dict] = {doc["call_id"]: doc for doc in call_docs if doc}
# Key off the id we asked for, not doc["call_id"]. At least one stored call
# has no call_id field -- the document id is authoritative and always
# present, while the field is written by the upload path and evidently was
# not always there. Indexing the field raised KeyError and took the whole
# debug view down with a 500 over a single malformed document.
call_map: dict[str, dict] = {
cid: doc for cid, doc in zip(unique_call_ids, call_docs) if doc
}
# ── Build incident debug records ──────────────────────────────────────────
incident_records = []
@@ -162,7 +279,7 @@ async def debug_correlation(
if c.get("status") == "ended"
and not c.get("incident_ids") and not c.get("incident_id")
and not c.get("duplicate_of") # another node's copy — never meant to correlate
and c.get("system_id") in ai_systems
and _in_scope([c.get("system_id")])
]
orphans.sort(key=lambda c: c.get("started_at", ""), reverse=True)
@@ -184,8 +301,116 @@ async def debug_correlation(
if (o.get("corr_sweep_count") or 0) >= 3:
orphans_by_tg[tg_key]["sweep_exhausted_count"] += 1
# ── Summary ───────────────────────────────────────────────────────────────
# Everything below was being recomputed by hand from the raw payload on
# every review — path counts, how much of the run the LLM tier actually saw,
# how many incidents ended up with the "Ems — TGID 9048" fallback name, and
# whether anything blew past the server-26#22 caps. Compute it once, here,
# where the data already is.
def _tally(values) -> dict:
out: dict[str, int] = {}
for v in values:
k = str(v) if v is not None else "none"
out[k] = out.get(k, 0) + 1
return dict(sorted(out.items(), key=lambda kv: kv[1], reverse=True))
linked = [c for inc in incident_records for c in (inc.get("calls_detail") or [])]
call_counts = [len(inc.get("call_ids") or []) for inc in incident_records]
def _tally_entries(call_summary: dict) -> list:
"""
server-26#96 — the unit correlation actually decided over is the
scene, not the call. A call summary carrying a `scenes` list (every
call correlated after this fix) contributes one entry per scene, each
with its own corr_path/corr_consensus/etc, instead of the single flat
record that used to blend every scene's last write together. A call
summary with no `scenes` (a call doc from before this fix) falls back
to contributing itself as one entry — identical to pre-#96 behaviour.
"""
scenes = call_summary.get("scenes")
return scenes if scenes else [call_summary]
scene_entries = [entry for c in linked for entry in _tally_entries(c)]
def _span_minutes(inc: dict) -> float:
stamps = sorted(
s for s in ((c.get("started_at") or "") for c in (inc.get("calls_detail") or [])) if s
)
if len(stamps) < 2:
return 0.0
try:
first = datetime.fromisoformat(str(stamps[0]).replace("Z", "+00:00"))
last = datetime.fromisoformat(str(stamps[-1]).replace("Z", "+00:00"))
return round((last - first).total_seconds() / 60, 1)
except ValueError:
return 0.0
spans = [_span_minutes(inc) for inc in incident_records]
with_transcript = sum(1 for c in linked if (c.get("transcript") or "").strip())
fallback_titles = sum(
1 for inc in incident_records
if " — TGID " in (inc.get("title") or "") or (inc.get("title") or "").endswith("Unknown Talkgroup")
)
over_cap = [
{"incident_id": inc.get("incident_id"), "title": inc.get("title"),
"calls": len(inc.get("call_ids") or []), "span_minutes": _span_minutes(inc)}
for inc in incident_records
if len(inc.get("call_ids") or []) > settings.incident_max_calls
or _span_minutes(inc) > settings.incident_max_duration_minutes
]
summary = {
"ai_systems_only": ai_systems_only,
"ai_enabled_system_ids": sorted(ai_systems),
"linked_call_count": len(linked),
# server-26#96 — tallied over scene_entries (one entry per scene of a
# multi-scene call, from its `scenes` map; one entry per call when it
# has none) rather than over `linked` directly, so a 2-scene call
# with two different corr_path values counts as two data points
# instead of one blended flat record. scene_decision_count makes that
# distinction visible next to linked_call_count.
"scene_decision_count": len(scene_entries),
"corr_path": _tally(e.get("corr_path") for e in scene_entries),
"corr_fit_signal": _tally(e.get("corr_fit_signal") for e in scene_entries),
"corr_consensus": _tally(e.get("corr_consensus") for e in scene_entries),
"corr_llm_action": _tally(e.get("corr_llm_action") for e in scene_entries),
# server-26#115 — this IS the number the escape-hatch fix exists to
# produce: why each llm=orphan/rules=new call escaped the gate.
"corr_gate_veto": _tally(e.get("corr_gate_veto") for e in scene_entries),
# server-26#127 — shadow-mode chatter classifier. The target
# population is non-events, which land as orphans or single-call
# incidents, NOT as a slice of every linked call -- tally `orphans`
# too or this undercounts the exact thing the feature measures.
"chatter_classifier_flagged": sum(
1 for c in (linked + orphans) if c.get("chatter_classifier_verdict")
),
"chatter_classifier_reason": _tally(
c.get("chatter_classifier_reason") for c in (linked + orphans)
if c.get("chatter_classifier_verdict")
),
# STT coverage: correlation quality is capped by this, so it belongs in
# the same view rather than a separate investigation.
"linked_calls_with_transcript": with_transcript,
"linked_calls_without_transcript": len(linked) - with_transcript,
"orphans_with_transcript": sum(1 for o in orphans if (o.get("transcript") or "").strip()),
# Fragmentation vs merging, the two failure directions.
"single_call_incidents": sum(1 for n in call_counts if n == 1),
"median_calls_per_incident": sorted(call_counts)[len(call_counts) // 2] if call_counts else 0,
"max_calls_in_one_incident": max(call_counts) if call_counts else 0,
"max_span_minutes": max(spans) if spans else 0.0,
"incidents_over_cap": over_cap,
"caps": {
"incident_max_calls": settings.incident_max_calls,
"incident_max_duration_minutes": settings.incident_max_duration_minutes,
},
# Titling health — server-26#34.
"fallback_titled_incidents": fallback_titles,
"titled_incidents": len(incident_records) - fallback_titles,
}
return {
"generated_at": datetime.now(timezone.utc).isoformat(),
"summary": summary,
# Both reads are capped, so say plainly when a cap was hit — otherwise a
# truncated window is indistinguishable from a quiet night.
"incidents_window_exhausted": incidents_window_exhausted,
+298 -1
View File
@@ -5,6 +5,7 @@ from typing import Optional
from app.internal import firestore as fstore
from app.internal.auth import (
require_admin_token,
require_firebase_token,
require_service_or_firebase_token,
resolve_caller_org_id,
reprocess_limiter,
@@ -15,9 +16,49 @@ from app.internal.storage import gcs_uri_for_call, with_playback_url
class TranscriptUpdate(BaseModel):
transcript: str
class EvalTranscriptUpdate(BaseModel):
text: str
router = APIRouter(prefix="/calls", tags=["calls"])
def _parse_ts(value: Optional[str], field: str) -> Optional[datetime]:
"""ISO string from a query param → aware datetime, or 400.
started_at is stored as a Firestore timestamp, so a cursor or range bound
passed through as the raw string compares by *type* (every string sorts
after every timestamp) rather than by time — a string cursor made "Load
more" return the first page again.
"""
if not value:
return None
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
raise HTTPException(400, f"{field} is not an ISO-8601 timestamp.")
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
def _next_cursor(rows: list[dict], matches: list[dict], page: list[dict], window: int) -> Optional[str]:
"""Where the next page of a bounded-window scan starts.
More matches than fit on the page → resume right after the last row
returned, or every match between it and the end of the window is skipped
(a 200-row window shown 50 at a time lost 150 calls per "Load more").
Otherwise resume after the last row SCANNED, not the last match — a page
whose last match sits early in the window would re-scan everything after
it and loop forever on a sparse filter. A short window is the end.
"""
if len(matches) > len(page):
last = page[-1].get("started_at")
elif len(rows) == window:
last = rows[-1].get("started_at")
else:
return None
return last.isoformat() if hasattr(last, "isoformat") else last
@router.get("")
async def list_calls(
node_id: Optional[str] = Query(None),
@@ -40,6 +81,209 @@ async def list_calls(
return [with_playback_url(c) for c in calls]
@router.get("/search")
async def search_calls(
limit: int = Query(50, ge=1, le=200),
cursor: Optional[str] = Query(None, description="started_at of the last row of the previous page"),
system_id: Optional[str] = Query(None),
node_id: Optional[str] = Query(None),
talkgroup_id: Optional[int] = Query(None),
link: str = Query("any", pattern="^(any|orphan|linked)$"),
transcript: str = Query("any", pattern="^(any|yes|no)$"),
q: Optional[str] = Query(None, description="case-insensitive substring of the transcript"),
date_from: Optional[str] = Query(None, description="ISO timestamp, inclusive lower bound on started_at"),
date_to: Optional[str] = Query(None, description="ISO timestamp, inclusive upper bound on started_at"),
decoded: dict = Depends(require_firebase_token),
):
"""
Paged, filterable call archive — the backend for the /calls page.
`GET /calls` returns every call in one unordered shot, which is fine for a
node's handful of active calls and useless as an archive: no order, no
paging, no way to find the orphans. This route is the archive read.
Only the org scope and the started_at ordering go to Firestore, because
that pair is the one composite index that exists (infra/firestore/
firestore.indexes.json). Every other filter runs in Python over a bounded
window, the same shape admin.py's correlation debug route uses — adding a
composite index per filter combination would be a worse trade than reading
10x the page and discarding most of it.
`window_exhausted` says the scan hit its cap before filling the page, so an
empty result means "not in this window", not "none exist".
Open to every org member (viewer included), not just admins: the Firestore
rules already let any member read every call doc in their org
(firestore.rules `calls` → docInMyOrg), so this route exposes nothing a
viewer's browser couldn't already read directly.
"""
org_id = await resolve_caller_org_id(decoded)
if org_id is None:
# resolve_caller_org_id lets platform admins see every org (server-26#4).
# A new browse surface shouldn't widen that, so fall back to the
# caller's own org claim when they have one.
org_id = decoded.get("org_id")
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
cursor_dt = _parse_ts(cursor, "cursor")
from_dt = _parse_ts(date_from, "date_from")
to_dt = _parse_ts(date_to, "date_to")
# A range on the ordered field rides the same org_id/started_at index.
conditions: list[tuple[str, str, object]] = [("org_id", "==", org_id)]
if from_dt:
conditions.append(("started_at", ">=", from_dt))
if to_dt:
conditions.append(("started_at", "<=", to_dt))
window = max(limit * 10, 200)
rows = await fstore.collection_where(
"calls",
conditions,
order_by=[("started_at", "DESCENDING")],
limit_to=window,
start_after={"started_at": cursor_dt} if cursor_dt else None,
)
needle = (q or "").strip().lower()
def _keep(c: dict) -> bool:
if system_id and c.get("system_id") != system_id:
return False
if node_id and c.get("node_id") != node_id:
return False
if talkgroup_id is not None and c.get("talkgroup_id") != talkgroup_id:
return False
linked = bool(c.get("incident_ids") or c.get("incident_id"))
if link == "orphan" and linked:
return False
if link == "linked" and not linked:
return False
text = c.get("transcript_corrected") or c.get("transcript") or ""
if transcript == "yes" and not text:
return False
if transcript == "no" and text:
return False
if needle and needle not in text.lower():
return False
return True
matches = [c for c in rows if _keep(c)]
page = matches[:limit]
next_cursor = _next_cursor(rows, matches, page, window)
return {
"calls": [with_playback_url(c) for c in page],
"next_cursor": next_cursor,
"scanned": len(rows),
"matched": len(matches),
"window_exhausted": len(rows) == window,
}
@router.get("/eval-queue")
async def eval_queue(
limit: int = Query(5, ge=1, le=20),
cursor: Optional[str] = Query(None, description="started_at of the last row of the previous page"),
decoded: dict = Depends(require_admin_token),
):
"""
A batch of calls that have a machine transcript but no human-verified one
yet — the backend for the STT eval page (server-26#163).
Deliberately separate from `PATCH /{call_id}/transcript`: that route is a
PRODUCTION correction — it re-runs extraction, unlinks incidents, and
feeds the vocabulary learner. An eval annotation must never trigger any
of that; it only exists to measure the pipeline, not to change what it
already decided. `eval_transcript` lives next to `transcript`/
`transcript_corrected` on the call doc and nothing downstream reads it.
Same bounded-window-scan-plus-cursor shape as `/search`, for the same
reason: no composite index exists for "eval_transcript is unset", and one
scan ordered by started_at is already trusted here. Paging through with
the returned cursor is how "however many, over time" actually works —
each call is where the last session left off, not a fresh random sample.
"""
org_id = await resolve_caller_org_id(decoded)
if org_id is None:
org_id = decoded.get("org_id")
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
cursor_dt = _parse_ts(cursor, "cursor")
window = max(limit * 20, 300)
rows = await fstore.collection_where(
"calls",
[("org_id", "==", org_id)],
order_by=[("started_at", "DESCENDING")],
limit_to=window,
start_after={"started_at": cursor_dt} if cursor_dt else None,
)
def _eligible(c: dict) -> bool:
text = c.get("transcript_corrected") or c.get("transcript") or ""
return bool(text) and not c.get("eval_transcript")
matches = [c for c in rows if _eligible(c)]
page = matches[:limit]
next_cursor = _next_cursor(rows, matches, page, window)
return {
"calls": [with_playback_url(c) for c in page],
"next_cursor": next_cursor,
"scanned": len(rows),
"matched": len(matches),
"window_exhausted": len(rows) == window,
}
@router.get("/eval-stats")
async def eval_stats(decoded: dict = Depends(require_admin_token)):
"""
How many calls have a human-verified transcript, and the WER of the raw
and corrected machine transcripts against them (server-26#163).
Whole-collection scan, matching `GET /calls` (list_calls above) rather
than the bounded-window pattern the paged routes use: the eval set this
is measuring is built a few calls at a time and expected to stay small
(tens to hundreds), so a full scan filtered in Python is the honest
answer rather than a windowed guess that could miss eval'd calls sitting
outside a recency window.
"""
from app.internal.wer import word_error_rate
org_id = await resolve_caller_org_id(decoded)
filters = {"org_id": org_id} if org_id is not None else {}
calls = await fstore.collection_list("calls", **filters)
raw_wers: list[float] = []
corrected_wers: list[float] = []
for c in calls:
ref = c.get("eval_transcript")
if not ref:
continue
raw = c.get("transcript") or ""
corrected = c.get("transcript_corrected") or raw
raw_wer = word_error_rate(ref, raw)
corrected_wer = word_error_rate(ref, corrected)
if raw_wer is not None:
raw_wers.append(raw_wer)
if corrected_wer is not None:
corrected_wers.append(corrected_wer)
def _avg(xs: list[float]) -> Optional[float]:
return round(sum(xs) / len(xs), 4) if xs else None
return {
"eval_count": len(raw_wers),
"raw_wer": _avg(raw_wers),
"corrected_wer": _avg(corrected_wers),
}
@router.get("/{call_id}")
async def get_call(call_id: str, decoded: dict = Depends(require_service_or_firebase_token)):
call = await fstore.doc_get("calls", call_id)
@@ -139,10 +383,26 @@ async def patch_transcript(
_: dict = Depends(require_admin_token),
):
"""Overwrite a call's transcript and re-run intelligence extraction."""
from app.internal.feature_flags import resolve_flags
call = await fstore.doc_get("calls", call_id)
if not call:
raise HTTPException(404, f"Call '{call_id}' not found.")
# This route is destructive before it is constructive: it wipes the call's
# tags, severity, location, units and embedding and unlinks it from every
# incident, on the promise that re-extraction will rebuild all of it. With
# correlation off that promise cannot be kept, and the call would be left
# permanently blank and orphaned while the route still answered 200.
# Refuse before the first write rather than half-run (server-26#76).
_, flag = await resolve_flags(call.get("system_id"))
if not flag("correlation_enabled"):
raise HTTPException(
409,
"Correlation is disabled, so the re-extraction this correction depends on "
"cannot run. The transcript was not changed. Enable correlation and retry.",
)
# Save user correction as transcript_corrected; leave original transcript intact.
# Clear stale intelligence fields so re-extraction runs fresh.
await fstore.doc_set("calls", call_id, {
@@ -154,6 +414,15 @@ async def patch_transcript(
"vehicles": [],
"embedding": None,
})
# server-26#96/#114 review: doc_set(merge=True) can only ADD/overwrite keys
# in a nested map, never remove one, so the fields above get cleared but a
# prior `scenes` map would survive re-extraction forever. A call corrected
# from 3 scenes down to 1 would keep scenes.1/scenes.2 with pre-correction
# transcripts and incident_ids -- corrupting the exact per-scene tally #96
# exists to make trustworthy, and re-feeding stale text into #114's
# summarizer fix if a stale scene's incident_id still names a real
# incident. Must be a real delete, not a merge over an empty map.
await fstore.doc_update("calls", call_id, {"scenes": fstore.DELETE_FIELD})
# Unlink from ALL current incidents so re-correlation starts clean.
# Handles both old single incident_id and new incident_ids list.
@@ -173,6 +442,8 @@ async def patch_transcript(
await fstore.doc_set("incidents", old_incident_id, {
"call_ids": [],
"status": "resolved",
"resolved_at": datetime.now(timezone.utc).isoformat(),
"resolved_via": "emptied_by_correction",
"summary_stale": True,
})
await fstore.doc_set("calls", call_id, {"incident_ids": [], "incident_id": None})
@@ -180,7 +451,7 @@ async def patch_transcript(
# Learn from the correction: diff original → corrected and add new tokens to vocabulary
system_id = call.get("system_id")
original_text = call.get("transcript_corrected") or call.get("transcript") or ""
if system_id and original_text:
if system_id and original_text and flag("vocabulary_learning_enabled"):
from app.internal.vocabulary_learner import learn_from_correction
await learn_from_correction(system_id, original_text, body.transcript)
@@ -197,3 +468,29 @@ async def patch_transcript(
preserve_transcript_correction=True,
)
return {"ok": True, "call_id": call_id}
@router.put("/{call_id}/eval-transcript")
async def put_eval_transcript(
call_id: str,
body: EvalTranscriptUpdate,
decoded: dict = Depends(require_admin_token),
):
"""
Record a human-verified reference transcript for the STT eval harness
(server-26#163). Pure data capture — unlike `PATCH /{call_id}/transcript`
above, this never touches `transcript`/`transcript_corrected`, never
re-runs extraction, never unlinks incidents, and never feeds the
vocabulary learner. It exists to MEASURE the pipeline's output, not to
change it; the two must not share a code path.
"""
call = await fstore.doc_get("calls", call_id)
if not call:
raise HTTPException(404, f"Call '{call_id}' not found.")
await fstore.doc_set("calls", call_id, {
"eval_transcript": body.text,
"eval_transcript_by": decoded.get("email") or decoded.get("uid"),
"eval_transcript_at": datetime.now(timezone.utc).isoformat(),
})
return {"ok": True, "call_id": call_id}
+65 -5
View File
@@ -97,30 +97,90 @@ async def delete_incident(incident_id: str, _: dict = Depends(require_admin_toke
async def summarize_incident(
incident_id: str,
background_tasks: BackgroundTasks,
decoded: dict = Depends(require_service_or_firebase_token),
decoded: dict = Depends(require_admin_token),
):
"""Immediately run the summarizer for a specific incident."""
from app.internal.summarizer import _summarize_incident
from app.internal.feature_flags import get_flags
inc = await fstore.doc_get("incidents", incident_id)
if not inc:
raise HTTPException(404, f"Incident '{incident_id}' not found.")
flags = await get_flags()
if not flags["summaries_enabled"]:
return {"ok": False, "incident_id": incident_id, "summaries_enabled": False}
# Rate limit by incident ID to prevent repeated expensive LLM calls
summarize_limiter.check(incident_id)
background_tasks.add_task(_summarize_incident, inc)
return {"ok": True, "incident_id": incident_id}
return {"ok": True, "incident_id": incident_id, "summaries_enabled": True}
@router.post("/{incident_id}/calls/{call_id}")
async def link_call_to_incident(incident_id: str, call_id: str, _: dict = Depends(require_admin_token)):
"""Manually attach a call to an incident (the /calls page's attribution action)."""
doc = await fstore.doc_get("incidents", incident_id)
if not doc:
raise HTTPException(404, f"Incident '{incident_id}' not found.")
call_ids = doc.get("call_ids", [])
call = await fstore.doc_get("calls", call_id)
if not call:
raise HTTPException(404, f"Call '{call_id}' not found.")
call_ids = list(doc.get("call_ids") or [])
if call_id not in call_ids:
call_ids.append(call_id)
await fstore.doc_update("incidents", incident_id, {
"call_ids": call_ids,
"updated_at": datetime.now(timezone.utc).isoformat(),
# A manually attached call changes what the incident is about.
"summary_stale": True,
})
await fstore.doc_update("calls", call_id, {"incident_id": incident_id})
return {"ok": True}
# incident_ids is the canonical link — it is what the correlator writes and
# what the frontend queries with array-contains. This route only ever set
# the legacy scalar incident_id, so a manually attached call stayed
# invisible on the incident's own page.
incident_ids = list(call.get("incident_ids") or ([call["incident_id"]] if call.get("incident_id") else []))
if incident_id not in incident_ids:
incident_ids.append(incident_id)
await fstore.doc_update("calls", call_id, {
"incident_ids": incident_ids,
"incident_id": incident_id,
"corr_path": "manual",
})
return {"ok": True, "incident_ids": incident_ids}
@router.delete("/{incident_id}/calls/{call_id}")
async def unlink_call_from_incident(incident_id: str, call_id: str, _: dict = Depends(require_admin_token)):
"""
Detach a call from an incident — the other half of manual attribution.
An incident left with no calls is resolved rather than deleted, matching
what calls.py's transcript correction does when it empties one.
"""
doc = await fstore.doc_get("incidents", incident_id)
if not doc:
raise HTTPException(404, f"Incident '{incident_id}' not found.")
remaining = [c for c in (doc.get("call_ids") or []) if c != call_id]
updates: dict = {
"call_ids": remaining,
"updated_at": datetime.now(timezone.utc).isoformat(),
"summary_stale": True,
}
if not remaining:
updates["status"] = "resolved"
updates["resolved_at"] = datetime.now(timezone.utc).isoformat()
updates["resolved_via"] = "emptied_by_admin"
await fstore.doc_update("incidents", incident_id, updates)
call = await fstore.doc_get("calls", call_id)
if call:
incident_ids = [
i for i in (call.get("incident_ids") or ([call["incident_id"]] if call.get("incident_id") else []))
if i != incident_id
]
await fstore.doc_update("calls", call_id, {
"incident_ids": incident_ids,
"incident_id": incident_ids[0] if incident_ids else None,
})
return {"ok": True, "incident_emptied": not remaining}
+11 -5
View File
@@ -13,7 +13,7 @@ service-account private key on the VM — is involved anywhere in this path.
"""
from fastapi import APIRouter, HTTPException, Query, Response
from app.internal import firestore as fstore
from app.internal.storage import verify_audio_link, gcs_uri_for_call, download_audio
from app.internal.storage import verify_audio_link, gcs_uri_for_call, download_audio, content_type_for
router = APIRouter(prefix="/media", tags=["media"])
@@ -42,12 +42,18 @@ async def get_call_audio(
return Response(
content=data,
media_type="audio/mpeg",
# Was hardcoded audio/mpeg. The node now uploads FLAC, and a browser
# will not play a FLAC body labelled audio/mpeg. Derived from the stored
# object's extension so old .mp3 recordings keep working unchanged.
media_type=content_type_for(gcs_uri),
headers={
"Content-Length": str(len(data)),
# Recordings are small (16 kbps mono — a 30s call is ~60 KB), so the
# whole body is sent at once and the browser seeks within its own
# buffer. Range support would only matter for long files.
# Whole body at once, no Range support. This was comfortable at
# 16 kbps mono (~60 KB for a 30 s call); FLAC is ~1.3 MB/min, so a
# long call is now tens of MB and the browser must download all of
# it before playback starts. Acceptable for typical few-second
# transmissions, but this is the change that makes Range support
# actually matter — see DEFERRED.md.
"Accept-Ranges": "none",
# Immutable content, but the URL expires — cache privately only.
"Cache-Control": "private, max-age=3600",
+3
View File
@@ -195,6 +195,7 @@ async def assign_system(
class NodeUpdateBody(BaseModel):
node_type: Optional[str] = None
enforce_override_timeout: Optional[bool] = None
secondary_sdr_mode: Optional[str] = None # none | adsb | ais | op25_2
@router.patch("/{node_id}")
@@ -227,6 +228,8 @@ async def update_node(
}
if updated_node.get("ppm_override") is not None:
push_payload["ppm_override"] = updated_node["ppm_override"]
if updated_node.get("secondary_sdr_mode") is not None:
push_payload["secondary_sdr_mode"] = updated_node["secondary_sdr_mode"]
mqtt_handler.push_config(node_id, push_payload)
return {"ok": True}
+183
View File
@@ -0,0 +1,183 @@
"""
Admin replay routes — the backend for the /admin Replay tab.
See app/internal/replay.py for what a run is and why it exists. Every route is
admin-only: a run spends real AI credits.
"""
from datetime import datetime, timezone
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from pydantic import BaseModel
from app.internal import replay
from app.internal.audit import write_audit
from app.internal.auth import describe_actor, require_admin_token, resolve_caller_org_id
from app.internal.logger import logger
router = APIRouter(prefix="/admin/replay", tags=["admin"])
def _parse_ts(value: Optional[str], field: str) -> datetime:
if not value:
raise HTTPException(400, f"{field} is required.")
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
raise HTTPException(400, f"{field} is not an ISO-8601 timestamp.")
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
async def _org(decoded: dict) -> str:
# Same fallback as /calls/search: a platform admin resolves to "every
# org", which is not a scope a replay can run in.
org_id = await resolve_caller_org_id(decoded) or decoded.get("org_id")
if not org_id:
raise HTTPException(403, "No organization scope for this caller.")
return org_id
async def _own_run(run_id: str, org_id: str) -> dict:
run = await replay.get_run(run_id)
if not run or run.get("org_id") != org_id:
raise HTTPException(404, f"Replay run '{run_id}' not found.")
return run
@router.get("/estimate")
async def estimate_run(
date_from: str = Query(...),
date_to: str = Query(...),
mode: Literal["audio", "transcripts", "reuse"] = Query("transcripts"),
system_ids: Optional[str] = Query(None, description="comma-separated"),
decoded: dict = Depends(require_admin_token),
):
"""How many calls a run over this range would process, and a rough cost."""
org_id = await _org(decoded)
sids = [s for s in (system_ids or "").split(",") if s] or None
calls, truncated = await replay.select_calls(
org_id, _parse_ts(date_from, "date_from"), _parse_ts(date_to, "date_to"), sids,
)
return {**replay.estimate(calls, mode), "truncated": truncated, "max_calls": replay.MAX_CALLS}
class StartRun(BaseModel):
date_from: str
date_to: str
mode: Literal["audio", "transcripts", "reuse"] = "transcripts"
system_ids: Optional[list[str]] = None
source_run_id: Optional[str] = None
label: str = ""
@router.post("")
async def start_run(body: StartRun, decoded: dict = Depends(require_admin_token)):
org_id = await _org(decoded)
actor_uid, actor_email = describe_actor(decoded)
try:
run = await replay.start_run(
org_id=org_id,
date_from=_parse_ts(body.date_from, "date_from"),
date_to=_parse_ts(body.date_to, "date_to"),
mode=body.mode,
system_ids=body.system_ids or None,
source_run_id=body.source_run_id,
label=body.label[:120],
actor=actor_email or actor_uid,
)
except replay.ReplayBusy as e:
raise HTTPException(409, str(e))
except ValueError as e:
raise HTTPException(400, str(e))
try:
await write_audit(actor_uid, actor_email, "replay.start", details={
"run_id": run["run_id"], "mode": run["mode"], "calls": run["progress"]["total"],
"est_cost_usd": run["estimate"]["est_cost_usd"],
})
except Exception as e:
logger.error(f"Replay: audit write failed ({e}) — run {run['run_id']} continues")
return run
@router.get("")
async def list_runs(decoded: dict = Depends(require_admin_token)):
org_id = await _org(decoded)
return {"runs": await replay.list_runs(org_id), "active_run_id": replay.active_run_id()}
@router.get("/{run_id}")
async def get_run(run_id: str, decoded: dict = Depends(require_admin_token)):
return await _own_run(run_id, await _org(decoded))
@router.post("/{run_id}/cancel")
async def cancel_run(run_id: str, decoded: dict = Depends(require_admin_token)):
await _own_run(run_id, await _org(decoded))
if not replay.request_cancel(run_id):
raise HTTPException(409, "That run is not running.")
return {"ok": True}
@router.delete("/{run_id}")
async def delete_run(run_id: str, decoded: dict = Depends(require_admin_token)):
await _own_run(run_id, await _org(decoded))
try:
await replay.delete_run(run_id)
except replay.ReplayBusy as e:
raise HTTPException(409, str(e))
return {"ok": True}
def _call_row(c: dict) -> dict:
scenes = c.get("scenes") or {}
paths = [((s.get("corr_debug") or {}).get("corr_path")) for _, s in sorted(scenes.items())]
return {
"call_id": c.get("call_id"),
"started_at": c.get("started_at"),
"talkgroup_name": c.get("talkgroup_name"),
"transcript": c.get("transcript_corrected") or c.get("transcript"),
"units": c.get("units"),
"cleared_units": c.get("cleared_units"),
"location": c.get("location"),
"skip_reason": c.get("skip_reason"),
"srcaddr": c.get("srcaddr"),
"corr_path": [p for p in paths if p] or ([c["corr_path"]] if c.get("corr_path") else []),
"incident_ids": c.get("incident_ids") or [],
}
@router.get("/{run_id}/incidents")
async def run_incidents(run_id: str, decoded: dict = Depends(require_admin_token)):
"""
A run's sandbox, shaped for reading: every incident with its calls in
order, plus the calls that never linked. Embeddings stay out.
"""
await _own_run(run_id, await _org(decoded))
incidents, calls = await replay.sandbox_contents(run_id)
by_id = {c.get("call_id"): _call_row(c) for c in calls}
out = []
for inc in sorted(incidents, key=lambda i: str(i.get("started_at") or "")):
rows = [by_id[cid] for cid in (inc.get("call_ids") or []) if cid in by_id]
rows.sort(key=lambda r: str(r["started_at"] or ""))
out.append({
"incident_id": inc.get("incident_id"),
"title": inc.get("title"),
"type": inc.get("type"),
"severity": inc.get("severity"),
"status": inc.get("status"),
"resolved_via": inc.get("resolved_via"),
"started_at": inc.get("started_at"),
"updated_at": inc.get("updated_at"),
"resolved_at": inc.get("resolved_at"),
"location": inc.get("location"),
"location_coords": inc.get("location_coords"),
"units": inc.get("units"),
"units_active": inc.get("units_active"),
"units_cleared": inc.get("units_cleared"),
"talkgroup_ids": inc.get("talkgroup_ids"),
"srcaddrs": inc.get("srcaddrs"),
"calls": rows,
})
orphans = sorted((r for r in by_id.values() if not r["incident_ids"]),
key=lambda r: str(r["started_at"] or ""))
return {"incidents": out, "orphans": orphans}
+165 -4
View File
@@ -1,9 +1,10 @@
import uuid
from fastapi import APIRouter, HTTPException, Depends, Query
from pydantic import BaseModel
from typing import Dict, Optional
from app.models import SystemCreate, SystemRecord
from typing import Dict, List, Optional
from app.models import AreaContextBody, SystemCreate, SystemRecord
from app.internal import firestore as fstore
from app.internal import area_context as area_ctx
from app.internal.auth import (
require_admin_token,
require_node_service_or_firebase_token,
@@ -23,6 +24,15 @@ class TenCodesBody(BaseModel):
ten_codes: Dict[str, str]
class UnitFormatBody(BaseModel):
unit_format_hint: str
class PendingTermBody(BaseModel):
talkgroup_id: int
term: str
class AiFlagsBody(BaseModel):
stt_enabled: Optional[bool] = None
correlation_enabled: Optional[bool] = None
@@ -64,8 +74,28 @@ async def update_system(system_id: str, body: SystemCreate, _: dict = Depends(re
existing = await fstore.doc_get("systems", system_id)
if not existing:
raise HTTPException(404, f"System '{system_id}' not found.")
await fstore.doc_update("systems", system_id, body.model_dump())
return {**existing, **body.model_dump()}
# exclude_unset, or every field the caller omitted gets written as its
# default and silently erases what was there. The systems page PUTs only
# {name, type, config}, so a plain model_dump() wiped ten_codes on every
# save — they are edited through PUT /{id}/ten-codes and were never in this
# payload. area_context (server-26#36) would have been the second casualty.
patch = body.model_dump(exclude_unset=True)
# The form sends config.talkgroups[] in full, which would erase the resolved
# anchor and the pending-term queue the backend put there. Same class of bug
# as ten_codes above; the backend merges its own fields back rather than
# taking dictation from the client (server-26#36).
if "config" in patch:
patch["config"] = area_ctx.merge_config(patch["config"], existing.get("config"))
if "area_context" in patch:
patch["area_context"] = area_ctx.merge_server_fields(
area_ctx.normalize(patch["area_context"]), existing.get("area_context")
)
await fstore.doc_update("systems", system_id, patch)
# Geocoding the anchor is a write-time job — a place changes when someone
# edits a town name, not every five minutes — but the operator should not
# wait on Maps to see their save land.
area_ctx.schedule_refresh(system_id)
return {**existing, **patch}
@router.delete("/{system_id}", status_code=204)
@@ -129,6 +159,137 @@ async def update_ten_codes(
return {"ok": True, "ten_codes": body.ten_codes}
# ── Unit ID format hint ─────────────────────────────────────────────────────────
@router.get("/{system_id}/unit-format")
async def get_unit_format(system_id: str):
"""Return the unit-ID format hint for a system."""
system = await fstore.doc_get("systems", system_id)
if not system:
raise HTTPException(404, f"System '{system_id}' not found.")
return {"unit_format_hint": system.get("unit_format_hint") or ""}
@router.put("/{system_id}/unit-format")
async def update_unit_format(
system_id: str,
body: UnitFormatBody,
_: dict = Depends(require_admin_token),
):
"""
Set the free-text unit-ID format hint fed into intelligence.py's
extraction prompt (server-26#<pending>). Departments have no shared unit
ID convention — e.g. "5-David"/bare "David" vs "SAM-1"/"airport-3" — and
the extraction prompt has no way to recognise a format it hasn't been
told about. Own route for the same reason ten-codes has one: not carried
by the systems form, so folding it into PUT /{id} would wipe it.
"""
existing = await fstore.doc_get("systems", system_id)
if not existing:
raise HTTPException(404, f"System '{system_id}' not found.")
await fstore.doc_update("systems", system_id, {"unit_format_hint": body.unit_format_hint})
return {"ok": True, "unit_format_hint": body.unit_format_hint}
# ── Area context ──────────────────────────────────────────────────────────────
@router.get("/{system_id}/area-context")
async def get_area_context(system_id: str, _: dict = Depends(require_admin_token)):
system = await fstore.doc_get("systems", system_id)
if not system:
raise HTTPException(404, f"System '{system_id}' not found.")
return {"area_context": system.get("area_context") or {}}
@router.put("/{system_id}/area-context")
async def update_area_context(
system_id: str,
body: AreaContextBody,
_: dict = Depends(require_admin_token),
):
"""
Replace the system-wide area context used by the corrector and the verifier.
Ground truth about where this system operates — municipality, county, state,
and the local names whose sound Whisper mangles. Per-talkgroup overrides live
inside config.talkgroups[] and rank ABOVE this (server-26#36), so a
multi-county system narrows per channel rather than replacing this wholesale.
Leaving it entirely empty is legitimate and meaningful: it says nothing here
is true of every talkgroup.
The derived anchor (`center`, `radius_km`, `resolved_from`, `resolved_at`) is
never taken from the body — it is carried forward and then recomputed here.
Its own route rather than a field on PUT /systems/{id} for the same reason
ten-codes has one: the systems form does not carry it, and folding it into
that payload is how ten_codes kept getting wiped.
"""
existing = await fstore.doc_get("systems", system_id)
if not existing:
raise HTTPException(404, f"System '{system_id}' not found.")
area = area_ctx.merge_server_fields(
area_ctx.normalize(body.model_dump(exclude_none=True)),
existing.get("area_context"),
)
await fstore.doc_update("systems", system_id, {"area_context": area})
# Awaited, not scheduled: this route exists to edit the place, so the caller
# should get back the anchor its edit produced. Talkgroups are refreshed with
# it because their anchor derives from the merged place, not their own.
patch = await area_ctx.refresh_anchors({**existing, "area_context": area})
if patch:
await fstore.doc_update("systems", system_id, patch)
area = patch.get("area_context", area)
return {"ok": True, "area_context": area}
# -- Talkgroup-level pending local knowledge (server-26#37) --------------------
@router.get("/{system_id}/talkgroup-pending")
async def list_talkgroup_pending(system_id: str, _: dict = Depends(require_admin_token)):
"""
Every pending local-knowledge proposal on this system, by talkgroup.
Proposals are made at talkgroup level and are never promoted to the system
automatically — a wrong term on one channel misleads one channel, the same
term system-wide misleads every channel on it.
"""
system = await fstore.doc_get("systems", system_id)
if not system:
raise HTTPException(404, f"System '{system_id}' not found.")
out = []
for tg in ((system.get("config") or {}).get("talkgroups") or []):
if not isinstance(tg, dict):
continue
pending = tg.get(area_ctx.PENDING_KEY) or []
if pending:
out.append({
"talkgroup_id": tg.get("id"),
"talkgroup_name": tg.get("name"),
"pending": pending,
})
return {"talkgroups": out}
@router.post("/{system_id}/talkgroup-pending/approve")
async def approve_talkgroup_pending(
system_id: str, body: PendingTermBody, _: dict = Depends(require_admin_token)
):
"""Move a pending term into that talkgroup's local_knowledge."""
if not await area_ctx.resolve_pending(system_id, body.talkgroup_id, body.term, approve=True):
raise HTTPException(404, "No such pending term on that talkgroup.")
return {"ok": True}
@router.post("/{system_id}/talkgroup-pending/dismiss")
async def dismiss_talkgroup_pending(
system_id: str, body: PendingTermBody, _: dict = Depends(require_admin_token)
):
"""Drop a pending term without adding it."""
if not await area_ctx.resolve_pending(system_id, body.talkgroup_id, body.term, approve=False):
raise HTTPException(404, "No such pending term on that talkgroup.")
return {"ok": True}
# ── Vocabulary endpoints ───────────────────────────────────────────────────────
@router.get("/{system_id}/vocabulary")
+125
View File
@@ -0,0 +1,125 @@
from datetime import datetime, timezone
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.internal import firestore as fstore
from app.internal.auth import require_node_service_or_firebase_token
from app.internal.logger import logger
router = APIRouter(prefix="/telemetry", tags=["telemetry"])
class AircraftReport(BaseModel):
icao: str
callsign: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
altitude_ft: Optional[float] = None
ground_speed_kt: Optional[float] = None
track_deg: Optional[float] = None
class AdsbUploadBody(BaseModel):
aircraft: List[AircraftReport]
@router.post("/adsb")
async def upload_adsb(
body: AdsbUploadBody,
decoded: dict = Depends(require_node_service_or_firebase_token),
):
"""
Node-initiated: a second-SDR ADS-B decoder (node-26#9) periodically posts
its current aircraft snapshot here. One doc per icao, last-seen-wins —
this is a live-map overlay, not a flight history.
"""
node_id = decoded.get("node_id")
if not node_id:
raise HTTPException(400, "This endpoint requires node identity, not a service/admin token.")
node = await fstore.doc_get_cached("nodes", node_id)
org_id = node.get("org_id") if node else None
now = datetime.now(timezone.utc).isoformat()
writes = []
for ac in body.aircraft:
if not ac.icao:
continue
doc = {
"icao": ac.icao,
"node_id": node_id,
"callsign": ac.callsign,
"lat": ac.lat,
"lon": ac.lon,
"altitude_ft": ac.altitude_ft,
"ground_speed_kt": ac.ground_speed_kt,
"track_deg": ac.track_deg,
"last_seen": now,
}
if org_id:
doc["org_id"] = org_id
writes.append(("aircraft", ac.icao, doc))
for collection, doc_id, doc in writes:
try:
await fstore.doc_set(collection, doc_id, doc, merge=True)
except Exception as e:
logger.warning(f"Failed to upsert {collection}/{doc_id} from node {node_id}: {e}")
return {"ok": True, "count": len(writes)}
class VesselReport(BaseModel):
mmsi: str
name: Optional[str] = None
lat: Optional[float] = None
lon: Optional[float] = None
speed_kt: Optional[float] = None
heading_deg: Optional[float] = None
class AisUploadBody(BaseModel):
vessels: List[VesselReport]
@router.post("/ais")
async def upload_ais(
body: AisUploadBody,
decoded: dict = Depends(require_node_service_or_firebase_token),
):
"""Same shape as /telemetry/adsb, one doc per mmsi in `vessels`."""
node_id = decoded.get("node_id")
if not node_id:
raise HTTPException(400, "This endpoint requires node identity, not a service/admin token.")
node = await fstore.doc_get_cached("nodes", node_id)
org_id = node.get("org_id") if node else None
now = datetime.now(timezone.utc).isoformat()
writes = []
for v in body.vessels:
if not v.mmsi:
continue
doc = {
"mmsi": v.mmsi,
"node_id": node_id,
"name": v.name,
"lat": v.lat,
"lon": v.lon,
"speed_kt": v.speed_kt,
"heading_deg": v.heading_deg,
"last_seen": now,
}
if org_id:
doc["org_id"] = org_id
writes.append(("vessels", v.mmsi, doc))
for collection, doc_id, doc in writes:
try:
await fstore.doc_set(collection, doc_id, doc, merge=True)
except Exception as e:
logger.warning(f"Failed to upsert {collection}/{doc_id} from node {node_id}: {e}")
return {"ok": True, "count": len(writes)}
+364 -100
View File
@@ -1,9 +1,11 @@
import secrets
from typing import Optional
from fastapi import APIRouter, BackgroundTasks, UploadFile, File, Form, HTTPException, Security
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from app.internal.storage import upload_audio
from app.internal import dedup
from app.internal import firestore as fstore
from app.internal import clock
from app.internal.logger import logger
from app.config import settings
@@ -35,7 +37,11 @@ async def upload_call_audio(
if not key_doc:
logger.warning(f"Upload 401: no key_doc in Firestore for node_id={node_id!r}")
raise HTTPException(401, "Invalid node API key")
if key_doc.get("api_key") != credentials.credentials:
# compare_digest, not !=, so the comparison cost does not depend on how many
# leading characters matched. enrollment.py and dynsec.py were explicit about
# this for the same class of credential; this route was the odd one out.
stored_key = key_doc.get("api_key") or ""
if not secrets.compare_digest(stored_key, credentials.credentials):
logger.warning(
f"Upload 401: key mismatch for node_id={node_id!r} "
f"(received prefix: {credentials.credentials[:8]}...)"
@@ -94,6 +100,98 @@ async def upload_call_audio(
return {"url": gcs_uri}
# server-26#115 — the consensus LLM-orphan gate only fires when the call is
# genuinely substanceless. The earlier version tested `rules_decision["corr_debug"]`
# for a "positive signal", but corr_debug is EMPTY at preview time for
# action=="new" (corr_path:"new" is written at APPLY time), so that test was
# always False and the gate dropped real events — a major "extinguishing fire",
# geocoded calls, pursuit updates. The substance test now runs against `ctx`,
# which is fully populated at preview time.
def _recent_incident_on_same_talkgroup(ctx: dict) -> bool:
"""
True when a recent incident is running on this call's own system +
talkgroup, within `settings.tg_dispatch_thin_idle_minutes` (5 min) —
applied uniformly regardless of the talkgroup's name (server-26#134).
Covers "unit dispatched, thin ack 10-30s later": the ack has no
substance of its own but plainly belongs to the job just opened.
Reads ctx["recent"] (the rules engine's own candidate list — no extra
Firestore read). That list is status=="active" incidents only, so an
already-resolved or capacity-capped same-talkgroup incident won't be
seen here even if chronologically recent (server-26#115, unresolved —
would need a dedicated non-status-filtered query).
Whether this limitation explains the 2/24 unexplained gate misses in the
window #3 measurement is UNANSWERED, not confirmed either way — a prior
pass here claimed a "confirmed explanation" for both that turned out to
be self-contradictory. Read `corr_gate_veto` (written to corr_debug on
every escalation of this exact disagreement shape — see the caller) in
the next measurement window instead of guessing from the raw dump again.
# TODO(server-26#115): add a talkgroup-scoped incident lookup (any
# status, no capacity filter) if a future measurement window pins a real
# gate miss on a resolved/capped same-talkgroup incident.
"""
from app.internal.incident_correlator import _idle_gate_minutes
tg_id = ctx.get("talkgroup_id")
system_id = ctx.get("system_id")
if tg_id is None or not system_id:
return False
tg_str = str(tg_id)
now = ctx.get("now") or clock.now()
idle_limit = settings.tg_dispatch_thin_idle_minutes
for inc in ctx.get("recent") or []:
if system_id not in (inc.get("system_ids") or []):
continue
if tg_str not in (inc.get("talkgroup_ids") or []):
continue
if _idle_gate_minutes(inc, now) <= idle_limit:
return True
return False
def _call_is_substanceless(ctx: dict) -> tuple[bool, Optional[str]]:
"""
True when the call carries nothing that marks it as a real event:
• no resolved incident_type and not a reassignment, AND
• severity is not moderate/major, AND
• no vehicle, geocode or tag (incident_correlator.has_event_substance —
the same predicate the incident-creation gate uses), AND
• no recent incident already running on the same talkgroup.
Only then may the LLM-orphan gate drop the call without a tiebreak.
Returns (substanceless, veto_reason). veto_reason names whichever
condition kept the tiebreak alive ("type" | "reassignment" | "severity" |
"substance" | "recent_tg"), or None when the call is substanceless. The
caller writes this into corr_debug on the escalation path so a live
measurement window can see *why* each llm=orphan/rules=new call escaped
the gate instead of inferring it after the fact from the raw dump —
exactly the guesswork that produced a wrong "confirmed explanation" for
2 window-#3 misses on the first pass of this fix.
"""
from app.internal import incident_correlator
# The incident-creation gate skips the has_event_substance check entirely
# when a type resolved (incident_correlator._run_decision ~:1397), so a
# typed call — fire/medical/etc. — opens an incident on substance we do not
# re-check here. reassignment=True is dispatch pulling a unit onto a NEW
# job (units are blanked at :296 for exactly that reason): the strongest
# new-incident signal in the pipeline. Either one means "keep the tiebreak".
if ctx.get("incident_type"):
return False, "type"
if ctx.get("reassignment"):
return False, "reassignment"
if (ctx.get("call_severity") or "routine") in ("moderate", "major"):
return False, "severity"
if incident_correlator.has_event_substance(ctx):
return False, "substance"
if _recent_incident_on_same_talkgroup(ctx):
return False, "recent_tg"
return True, None
async def _correlate_with_consensus(
call_id: str,
node_id: str,
@@ -108,6 +206,10 @@ async def _correlate_with_consensus(
vehicles: Optional[list] = None,
cleared_units: Optional[list] = None,
reassignment: bool = False,
embedding: Optional[list] = None,
severity: Optional[str] = None,
transcript: Optional[str] = None,
scene_index: int = 0,
) -> Optional[str]:
"""
Consensus correlator: runs the rules engine and the cheap LLM in sequence.
@@ -116,6 +218,11 @@ async def _correlate_with_consensus(
Falls back to rules-only when GEMINI_API_KEY is absent, the call is
content-free (thin), or any LLM call fails.
``scene_index`` (server-26#96) — which scene of the call this is, from the
caller's ``enumerate(scenes)`` loop. Threaded through so the call doc's
per-scene ``scenes`` map records this scene's own corr_debug/transcript
instead of colliding with every other scene's write on the flat fields.
"""
from app.internal import incident_correlator, llm_correlator
@@ -125,6 +232,8 @@ async def _correlate_with_consensus(
tags=tags, incident_type=incident_type, location=location,
location_coords=location_coords, units=units, vehicles=vehicles,
cleared_units=cleared_units, reassignment=reassignment,
embedding=embedding, severity=severity, transcript=transcript,
scene_index=scene_index,
)
ctx = preview["ctx"]
rules_decision = preview["decision"]
@@ -141,6 +250,37 @@ async def _correlate_with_consensus(
rules_decision["corr_debug"]["corr_llm_reasoning"] = llm_decision.get("reasoning", "")
return await incident_correlator.apply_correlation(preview)
# server-26#115 — LLM-orphan gate.
# When the cheap LLM says `orphan`, the rules engine says `new`, and the call
# is genuinely substanceless (routine severity, no vehicle/geocode/tag, and
# no incident already running on this talkgroup), resolve to `orphan` and DO
# NOT pay for the smart tiebreaker. A bare rules `new` there means only
# "nothing to link to" — trivially true for radio housekeeping (check-ins,
# roll call, 10-8/10-98) — and the tiebreaker rubber-stamped it ~21/21 of the
# time on exactly this disagreement (CORRELATION_REVIEW_0907b.md). Any real
# signal (severity, coords, tags, a live same-talkgroup incident) still
# escalates, so an event the LLM misreads as orphan is not lost.
is_orphan_vs_new = llm_decision["action"] == "orphan" and rules_decision["action"] == "new"
substanceless, gate_veto_reason = _call_is_substanceless(ctx) if is_orphan_vs_new else (False, None)
if is_orphan_vs_new and substanceless:
logger.info(
f"Consensus gate for call {call_id}: llm=orphan vs rules=new and call "
f"is substanceless — resolving orphan, skipping tiebreak"
)
gated = {
"action": "orphan",
"matched_incident": None,
"incident_type": None,
"corr_debug": dict(rules_decision.get("corr_debug") or {}),
}
gated["corr_debug"].update({
"corr_consensus": "llm_orphan_gate",
"corr_rules_action": rules_decision["action"],
"corr_llm_action": llm_decision["action"],
"corr_llm_reasoning": llm_decision.get("reasoning", ""),
})
return await incident_correlator.apply_correlation({"decision": gated, "ctx": ctx})
# Disagree — escalate to the smarter tiebreaker.
logger.info(
f"Consensus disagreement for call {call_id}: "
@@ -150,9 +290,28 @@ async def _correlate_with_consensus(
final["corr_debug"]["corr_consensus"] = "tiebreak"
final["corr_debug"]["corr_rules_action"] = rules_decision["action"]
final["corr_debug"]["corr_llm_action"] = llm_decision["action"]
if is_orphan_vs_new:
# server-26#115 — record *why* the llm=orphan/rules=new gate stood
# down instead of leaving a future measurement window to guess it
# from the raw dump (which produced a wrong "confirmed explanation"
# for 2/24 misses the first time around).
final["corr_debug"]["corr_gate_veto"] = gate_veto_reason
return await incident_correlator.apply_correlation({"decision": final, "ctx": ctx})
async def _resolve_flags(system_id: Optional[str]):
"""
Resolve AI feature flags for a given system.
Thin alias for `feature_flags.resolve_flags` — the resolver lives there
because transcription and the calls router need the same answer, and three
copies of it is how server-26#75 happened in the first place.
"""
from app.internal.feature_flags import resolve_flags
return await resolve_flags(system_id)
async def _run_extraction_pipeline(
call_id: str,
node_id: str,
@@ -166,22 +325,120 @@ async def _run_extraction_pipeline(
"""Run steps 2-4 of the intelligence pipeline using an existing transcript."""
from app.internal import intelligence, incident_correlator, alerter
# Step 2: Scene detection + intelligence extraction.
# Returns one scene per distinct incident detected in the recording.
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
node_id=node_id,
preserve_transcript_correction=preserve_transcript_correction,
)
flags, _flag = await _resolve_flags(system_id)
# Step 3: Correlate each scene to an incident independently.
incident_ids: list[str] = []
all_tags: list[str] = []
for scene in scenes:
if _flag("correlation_enabled"):
# Step 2: Scene detection + intelligence extraction.
# Returns one scene per distinct incident detected in the recording.
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
node_id=node_id,
preserve_transcript_correction=preserve_transcript_correction,
)
# Step 3: Correlate each scene to an incident independently.
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
# When dispatch is pulling a unit to a NEW call (reassignment), suppress unit
# overlap so the new scene doesn't chain into the unit's previous incident.
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
"reopenable": True, # provisional, see _extract_and_correlate
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
else:
scope = "globally" if not flags["correlation_enabled"] else f"system {system_id}"
logger.info(f"Correlation disabled ({scope}) — skipping scene extraction and correlation for call {call_id} (reprocess)")
if incident_ids:
await fstore.doc_set("calls", call_id, {"incident_ids": incident_ids})
# Step 4: Alert dispatch — run once with merged tags from all scenes.
await alerter.check_and_dispatch(
call_id=call_id,
node_id=node_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=list(dict.fromkeys(all_tags)),
transcript=transcript,
)
async def _extract_and_correlate(
call_id: str,
node_id: str,
system_id: Optional[str],
talkgroup_id: Optional[int],
talkgroup_name: Optional[str],
transcript: Optional[str],
segments: Optional[list[dict]] = None,
scenes: Optional[list[dict]] = None,
) -> tuple[list[str], list[str], list[dict]]:
"""
Steps 2-3 of the intelligence pipeline for one call: scene extraction
(skipped when `scenes` is passed in), then per-scene correlation, then the
no-scene thin fallback. Returns (incident_ids, merged tags, scenes).
Shared by the live pipeline below and by replay (app/internal/replay.py),
so a replay run measures exactly the code that runs live rather than a
copy of it that can drift. Caller owns the correlation feature-flag check
and alerting.
"""
from app.internal import intelligence, incident_correlator
# Step 2: Scene detection + intelligence extraction
if scenes is None:
scenes = []
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
node_id=node_id,
)
# Step 3: Correlate each scene independently.
# A single recording can produce multiple incidents on a busy channel.
incident_ids: list[str] = []
all_tags: list[str] = []
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
# When dispatch is pulling a unit to a NEW call (reassignment), suppress unit
# overlap so the new scene doesn't chain into the unit's previous incident.
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
@@ -198,26 +455,59 @@ async def _run_extraction_pipeline(
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {"status": "resolved"})
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
# One transmission read as "it's over" ("transport complete")
# closed the whole 09-22 bridge MVA at 14:44 and its next 56
# calls opened a second incident (server-26#170). Inferred from
# a single call, so provisional, like a timer close.
"reopenable": True,
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
# Correlator also runs for calls with no scenes (unclassified) to attempt
# talkgroup-based linking even when no transcript could be produced.
# transcript_too_short (<=5 words: "10-8", "show me clear", a unit
# check-in) still carries a real transcript and talkgroup — exactly the
# brief follow-up/clearance traffic an incident needs, and the thin-path
# merge below already requires a same-talkgroup, recently-active
# incident before attaching anything, same guard already trusted for
# no-transcript calls. Previously excluded here, so these calls never
# attached to anything at all. garbage_transcript (Whisper
# hallucination) has no real content behind it and stays excluded.
if not scenes:
_call_doc = await fstore.doc_get("calls", call_id)
skip_reason = (_call_doc or {}).get("skip_reason")
if not skip_reason or skip_reason == "transcript_too_short":
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=[],
incident_type=None,
location=None,
location_coords=None,
)
if incident_id:
incident_ids.append(incident_id)
if incident_ids:
await fstore.doc_set("calls", call_id, {"incident_ids": incident_ids})
# Step 4: Alert dispatch — run once with merged tags from all scenes.
await alerter.check_and_dispatch(
call_id=call_id,
node_id=node_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=list(dict.fromkeys(all_tags)),
transcript=transcript,
)
return incident_ids, all_tags, scenes
async def _run_intelligence_pipeline(
@@ -235,22 +525,46 @@ async def _run_intelligence_pipeline(
3. Correlate each scene with existing incidents (or create new ones)
4. Check alert rules and dispatch notifications
"""
from app.internal import transcription, intelligence, incident_correlator, alerter
from app.internal.feature_flags import get_flags
from app.internal import transcription, alerter, talkgroups
flags = await get_flags()
# server-26#131: mark that real-time processing has started for this call
# BEFORE any of the slow steps below (STT, scene extraction, correlation).
# The re-correlation sweep (internal/recorrelation_sweep.py) scans for
# calls that still look orphaned within a wide window (recorrelation_scan_
# minutes, default 60) — with no guard here, a call whose real-time
# pipeline is still mid-flight (still transcribing, still waiting on a
# Gemini call) has no incident_id/corr_path written yet, so the sweep's
# orphan filter can't tell "never processed" from "processing right now"
# and correlates it a second time, independently, sometimes landing on a
# different incident than the real-time path — the exact duplicate-link
# bug #131 found (same call in two incidents' call_ids, ~2% of linked
# calls). Best-effort: a write failure here must not abort the pipeline.
try:
await fstore.doc_set("calls", call_id, {
"intelligence_started_at": clock.now().isoformat()
})
except Exception as e:
logger.warning(f"Could not mark intelligence_started_at for call {call_id}: {e}")
# Resolve per-system overrides: system flag=False beats global flag=True,
# but global flag=False beats everything (master switch).
system_ai_flags: dict = {}
if system_id:
sys_doc = await fstore.doc_get_cached("systems", system_id)
system_ai_flags = (sys_doc or {}).get("ai_flags") or {}
# The node only sends talkgroup_name when OP25 had it in the loaded tags
# file, so it arrives empty for exactly the talkgroups C2 can name from the
# system config. Resolve it once, here, at the single funnel both /upload
# and /calls/{id}/reprocess pass through — everything downstream (the
# dispatch-channel test, scene extraction, and the incident title) then
# gets a real name instead of "TGID 9048". server-26#34.
_call_doc = await fstore.doc_get("calls", call_id)
talkgroup_name = await talkgroups.resolve(
system_id, talkgroup_id, hint=talkgroup_name, call_doc=_call_doc,
)
# Backfill the call document too, so the archive and the orphan panel stop
# showing a bare TGID for a channel we can now name.
if talkgroup_name and _call_doc is not None and not _call_doc.get("talkgroup_name"):
try:
await fstore.doc_set("calls", call_id, {"talkgroup_name": talkgroup_name})
except Exception as e:
logger.warning(f"Could not backfill talkgroup_name on call {call_id}: {e}")
def _flag(name: str) -> bool:
if not flags[name]: # global master off
return False
return system_ai_flags.get(name, True) # system override, default inherit
flags, _flag = await _resolve_flags(system_id)
transcript: Optional[str] = None
segments: list[dict] = []
@@ -259,80 +573,30 @@ async def _run_intelligence_pipeline(
if gcs_uri:
if _flag("stt_enabled"):
transcript, segments = await transcription.transcribe_call(
call_id, gcs_uri, talkgroup_name, system_id=system_id
call_id, gcs_uri, talkgroup_name,
system_id=system_id, talkgroup_id=talkgroup_id,
)
else:
scope = "globally" if not flags["stt_enabled"] else f"system {system_id}"
logger.info(f"STT disabled ({scope}) — skipping transcription for call {call_id}")
# Step 2: Scene detection + intelligence extraction
scenes: list[dict] = []
# Steps 2-3: scene extraction + correlation.
incident_ids: list[str] = []
all_tags: list[str] = []
if _flag("correlation_enabled"):
if transcript:
scenes = await intelligence.extract_scenes(
call_id, transcript, talkgroup_name,
talkgroup_id=talkgroup_id, system_id=system_id, segments=segments,
node_id=node_id,
)
incident_ids, all_tags, _ = await _extract_and_correlate(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
transcript=transcript,
segments=segments,
)
else:
scope = "globally" if not flags["correlation_enabled"] else f"system {system_id}"
logger.info(f"Correlation disabled ({scope}) — skipping scene extraction and correlation for call {call_id}")
# Step 3: Correlate each scene independently.
# A single recording can produce multiple incidents on a busy channel.
incident_ids: list[str] = []
all_tags: list[str] = []
if flags["correlation_enabled"]:
for scene in scenes:
all_tags.extend(scene["tags"])
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {"status": "resolved"})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
# Correlator also runs for calls with no scenes (unclassified) to attempt
# talkgroup-based linking even when no transcript could be produced.
# Skip when extraction flagged the call — garbage or too-short transcripts
# carry no signal and would only attach spuriously via the thin path.
if not scenes:
_call_doc = await fstore.doc_get("calls", call_id)
if not (_call_doc or {}).get("skip_reason"):
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=[],
incident_type=None,
location=None,
location_coords=None,
)
if incident_id:
incident_ids.append(incident_id)
if incident_ids:
await fstore.doc_set("calls", call_id, {"incident_ids": incident_ids})
# Step 4: Alert dispatch (always runs — talkgroup ID rules don't need a transcript)
await alerter.check_and_dispatch(
call_id=call_id,
+13
View File
@@ -34,12 +34,25 @@ except ModuleNotFoundError:
# into dicts that tests compare against, and a MagicMock compares unequal
# to itself across attribute accesses.
_fs.SERVER_TIMESTAMP = "__SERVER_TIMESTAMP__"
# Same reasoning as SERVER_TIMESTAMP above: a distinct sentinel, not a
# MagicMock, so `fstore.DELETE_FIELD is fs.DELETE_FIELD` and dict/`is`
# comparisons against it in tests (server-26#96/#114, PR #132) behave.
_fs.DELETE_FIELD = "__DELETE_FIELD__"
_auth = ModuleType("firebase_admin.auth")
_auth.verify_id_token = MagicMock()
_auth.set_custom_user_claims = MagicMock()
_auth.get_user_by_email = MagicMock()
_auth.get_user = MagicMock()
# Type used in annotations at import time by routers/users.py, so it has to
# exist as a name even though nothing here ever instantiates it. Without it,
# importing app.main -- and therefore testing anything wired at app level,
# like the CORS policy -- fails at collection.
_auth.UserRecord = MagicMock()
_auth.list_users = MagicMock()
_auth.update_user = MagicMock()
_auth.create_user = MagicMock()
_auth.delete_user = MagicMock()
_firebase.auth = _auth
_firebase.credentials = _credentials
@@ -0,0 +1,163 @@
"""
Unit tests for /admin/debug/correlation (server-26#24).
The endpoint used to strip the LLM consensus tier's fields (corr_consensus,
corr_llm_reasoning, corr_llm_action, corr_rules_action) out of its response
even though upload.py / llm_correlator.py write them straight onto the call
doc via corr_debug — making this endpoint unable to answer "is the LLM
correlation tier actually running", the one thing it exists to answer.
Firestore is fully mocked (patch app.routers.admin.fstore); the route
function is called directly, bypassing FastAPI's dependency injection, so
Query/Depends defaults are supplied explicitly.
"""
import pytest
from datetime import datetime, timezone
from unittest.mock import AsyncMock, patch
from app.routers import admin
NOW = datetime(2026, 8, 20, 12, 0, 0, tzinfo=timezone.utc)
def _incident(call_ids):
return {
"incident_id": "inc-1",
"system_ids": ["sys-1"],
"call_ids": call_ids,
"updated_at": NOW.isoformat(),
"started_at": NOW.isoformat(),
"status": "active",
}
async def _run(incidents, calls_by_id, orphan_calls=None):
"""Drive debug_correlation() with fstore fully mocked."""
system = {"system_id": "sys-1", "ai_flags": {}}
async def fake_collection_where(collection, conditions, order_by=None, limit_to=None, start_after=None):
if collection == "incidents":
return incidents
if collection == "calls":
return orphan_calls or []
return []
async def fake_doc_get(collection, doc_id):
return calls_by_id.get(doc_id)
with patch(
"app.routers.admin.get_flags",
new=AsyncMock(return_value={"stt_enabled": True, "correlation_enabled": True}),
), patch("app.routers.admin.fstore") as mock_fstore:
mock_fstore.collection_list = AsyncMock(return_value=[system])
mock_fstore.collection_where = AsyncMock(side_effect=fake_collection_where)
mock_fstore.doc_get = AsyncMock(side_effect=fake_doc_get)
return await admin.debug_correlation(limit=20, orphan_hours=48, _=None)
@pytest.mark.asyncio
async def test_debug_correlation_surfaces_llm_consensus_fields():
"""A call that went through the tiebreaker must show all four LLM fields."""
call = {
"call_id": "call-1",
"corr_path": "fast/single",
"corr_consensus": "tiebreak",
"corr_llm_reasoning": "Same units on scene as the anchor call.",
"corr_llm_action": "link",
"corr_rules_action": "orphan",
}
result = await _run([_incident(["call-1"])], {"call-1": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["corr_consensus"] == "tiebreak"
assert detail["corr_llm_reasoning"] == "Same units on scene as the anchor call."
assert detail["corr_llm_action"] == "link"
assert detail["corr_rules_action"] == "orphan"
@pytest.mark.asyncio
async def test_debug_correlation_llm_fields_absent_when_rules_only():
"""
A call that never reached the LLM (GEMINI_API_KEY unset, thin call, or LLM
error) has corr_consensus == "rules_only" and no corr_llm_* fields — the
endpoint must pass that through as None rather than erroring, since this
is the normal/expected state whenever the tier is legitimately idle.
"""
call = {"call_id": "call-2", "corr_path": "fast/single", "corr_consensus": "rules_only"}
result = await _run([_incident(["call-2"])], {"call-2": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["corr_consensus"] == "rules_only"
assert detail["corr_llm_reasoning"] is None
assert detail["corr_llm_action"] is None
# ---------------------------------------------------------------------------
# server-26#96 — the summary tally must count per-scene decisions, not the
# one blended flat record a multi-scene call used to leave behind.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_debug_correlation_exposes_scenes_and_tallies_each_as_its_own_datapoint():
"""A 2-scene call: the flat fields still show last-scene-wins (unchanged
behaviour for old readers), but the summary tally must see two distinct
corr_path/corr_consensus data points, not one blend."""
call = {
"call_id": "call-1",
# Flat fields — last scene wins, kept as-is for backward compat.
"corr_path": "slow",
"corr_consensus": "tiebreak",
"scenes": {
"0": {
"transcript": "scene zero",
"incident_id": "inc-1",
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
},
"1": {
"transcript": "scene one",
"incident_id": "inc-1",
"corr_debug": {"corr_path": "slow", "corr_consensus": "tiebreak"},
},
},
}
result = await _run([_incident(["call-1"])], {"call-1": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["corr_path"] == "slow" # flat field: last scene wins
assert len(detail["scenes"]) == 2
assert detail["scenes"][0]["corr_path"] == "new"
assert detail["scenes"][1]["corr_path"] == "slow"
summary = result["summary"]
assert summary["linked_call_count"] == 1 # still one CALL
assert summary["scene_decision_count"] == 2 # but two DECISIONS
assert summary["corr_path"] == {"new": 1, "slow": 1}
assert summary["corr_consensus"] == {"agreed": 1, "tiebreak": 1}
@pytest.mark.asyncio
async def test_debug_correlation_tally_falls_back_for_single_scene_call():
"""A plain single-scene call has no `scenes` field at all — the tally
must fall back to its flat fields as one data point, same as pre-#96."""
call = {"call_id": "call-2", "corr_path": "fast/single", "corr_consensus": "rules_only"}
result = await _run([_incident(["call-2"])], {"call-2": call})
detail = result["incidents"][0]["calls_detail"][0]
assert detail["scenes"] is None
summary = result["summary"]
assert summary["linked_call_count"] == 1
assert summary["scene_decision_count"] == 1
assert summary["corr_path"] == {"fast/single": 1}
assert summary["corr_consensus"] == {"rules_only": 1}
@pytest.mark.asyncio
async def test_debug_correlation_tally_handles_old_schema_call_with_no_scenes_field():
"""A call doc written before server-26#96 has never heard of `scenes` —
must behave identically to the single-scene case, not error."""
old_call = {"call_id": "call-3", "corr_path": "cross-tg", "corr_consensus": "agreed"}
result = await _run([_incident(["call-3"])], {"call-3": old_call})
summary = result["summary"]
assert summary["scene_decision_count"] == 1
assert summary["corr_path"] == {"cross-tg": 1}
@@ -0,0 +1,296 @@
"""
server-26#64 — a headless, attributable, total AI-flag flip.
Three things are held here:
* ``require_agent_key_or_admin`` is a DISTINCT principal. It takes the agent
service key or a Firebase admin token and refuses the Discord bot's
``service_key``, so an audit entry can name who flipped the switch.
* ``set_flags`` writes an ``audit_log`` entry carrying before/after values,
and an audit failure can neither lose the flag write nor 500 the route.
* ``cascade=True`` clears per-system ``ai_flags`` overrides for the keys
being set, so a flip cannot half-apply — discovered by scanning for
documents that carry the map, never a hardcoded system-id list.
The dependency is exercised directly rather than through TestClient: these are
assertions about the credential check, and routing them through the ASGI stack
would only add ways for the test to pass for the wrong reason.
"""
import pytest
from unittest.mock import AsyncMock, patch
from fastapi import HTTPException
from fastapi.security import HTTPAuthorizationCredentials
from app.config import settings
from app.internal import auth, feature_flags
from app.routers import admin
AGENT_KEY = "agent-key-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
BOT_KEY = "bot-key-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
def _creds(token: str) -> HTTPAuthorizationCredentials:
return HTTPAuthorizationCredentials(scheme="Bearer", credentials=token)
@pytest.fixture
def keys(monkeypatch):
"""Both keys configured and different — the production shape."""
monkeypatch.setattr(settings, "agent_service_key", AGENT_KEY, raising=False)
monkeypatch.setattr(settings, "service_key", BOT_KEY, raising=False)
@pytest.fixture(autouse=True)
def _clear_flag_cache():
"""feature_flags keeps module-level cache state; don't leak it across tests."""
feature_flags._cache = {}
feature_flags._cache_ts = 0.0
yield
feature_flags._cache = {}
feature_flags._cache_ts = 0.0
# ── Item 1: the credential ────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_agent_key_is_accepted_and_identifies_itself(keys):
principal = await auth.require_agent_key_or_admin(_creds(AGENT_KEY))
assert principal["principal"] == "agent"
# The caller must be able to tell the agent from a human admin, or the
# audit entry in item 3 cannot name the actor.
assert auth.describe_actor(principal) == (
auth.AGENT_PRINCIPAL_UID, auth.AGENT_PRINCIPAL_EMAIL,
)
@pytest.mark.asyncio
async def test_discord_bot_service_key_is_rejected(keys):
"""The whole point of a second key: the bot's key must not open this door.
It falls through to the Firebase branch and fails there, so the bot gets a
401 rather than an unattributable flag flip.
"""
with patch.object(auth.firebase_auth, "verify_id_token", side_effect=Exception("not a token")):
with pytest.raises(HTTPException) as exc:
await auth.require_agent_key_or_admin(_creds(BOT_KEY))
assert exc.value.status_code == 401
@pytest.mark.asyncio
async def test_unset_agent_key_cannot_be_bypassed(monkeypatch):
"""An unconfigured key must match nothing — especially not an empty string.
``secrets.compare_digest("", "")`` is a match, so the guard has to be on
the key being configured, not on a ``or ""`` fallback.
"""
monkeypatch.setattr(settings, "agent_service_key", None, raising=False)
with patch.object(auth.firebase_auth, "verify_id_token", side_effect=Exception("not a token")):
for token in ("", " ", "None", "null", AGENT_KEY):
with pytest.raises(HTTPException) as exc:
await auth.require_agent_key_or_admin(_creds(token))
assert exc.value.status_code == 401, token
@pytest.mark.asyncio
async def test_empty_string_agent_key_cannot_be_bypassed(monkeypatch):
"""Same guarantee for a key set to "" by an empty env var."""
monkeypatch.setattr(settings, "agent_service_key", "", raising=False)
with patch.object(auth.firebase_auth, "verify_id_token", side_effect=Exception("not a token")):
with pytest.raises(HTTPException) as exc:
await auth.require_agent_key_or_admin(_creds(""))
assert exc.value.status_code == 401
@pytest.mark.asyncio
async def test_firebase_admin_token_still_works(keys):
decoded = {"uid": "u-1", "email": "admin@example.com", "role": "admin"}
with patch.object(auth.firebase_auth, "verify_id_token", return_value=decoded):
principal = await auth.require_agent_key_or_admin(_creds("firebase-id-token"))
assert principal == decoded
assert auth.describe_actor(principal) == ("u-1", "admin@example.com")
@pytest.mark.asyncio
async def test_non_admin_firebase_token_is_forbidden(keys):
decoded = {"uid": "u-2", "email": "viewer@example.com", "role": "viewer"}
with patch.object(auth.firebase_auth, "verify_id_token", return_value=decoded):
with pytest.raises(HTTPException) as exc:
await auth.require_agent_key_or_admin(_creds("firebase-id-token"))
assert exc.value.status_code == 403
@pytest.mark.asyncio
async def test_missing_credentials_is_401(keys):
with pytest.raises(HTTPException) as exc:
await auth.require_agent_key_or_admin(None)
assert exc.value.status_code == 401
def test_features_routes_use_the_agent_dependency_and_others_do_not():
"""Guards the wiring: only /admin/features moved off require_admin_token."""
def deps(path, method):
for r in admin.router.routes:
if r.path == path and method in r.methods:
return {d.call for d in r.dependant.dependencies}
raise AssertionError(f"no route {method} {path}")
assert auth.require_agent_key_or_admin in deps("/admin/features", "GET")
assert auth.require_agent_key_or_admin in deps("/admin/features", "PUT")
assert auth.require_admin_token in deps("/admin/audit", "GET")
assert auth.require_admin_token in deps("/admin/debug/correlation", "GET")
# ── Items 3 and 4: set_flags audits, and cascades on request ──────────────────
def _fstore_mock(stored: dict, systems: list[dict], updates_sink: list):
"""A Firestore stand-in for feature_flags: one config doc, N system docs."""
mock = AsyncMock()
async def doc_get(collection, doc_id):
return dict(stored) if collection == "config" else None
async def doc_set(collection, doc_id, data, merge=True):
stored.update(data)
async def collection_list(collection, **filters):
return systems if collection == "systems" else []
async def doc_update(collection, doc_id, data):
updates_sink.append((collection, doc_id, data))
mock.doc_get = AsyncMock(side_effect=doc_get)
mock.doc_set = AsyncMock(side_effect=doc_set)
mock.collection_list = AsyncMock(side_effect=collection_list)
mock.doc_update = AsyncMock(side_effect=doc_update)
return mock
def _systems():
return [
# Two systems carry overrides today; the ids are irrelevant to the
# helper and must stay that way.
{"system_id": "sys-a", "ai_flags": {"stt_enabled": False, "correlation_enabled": False}},
{"system_id": "sys-b", "ai_flags": {"stt_enabled": False}},
# Carries the map but not the key being flipped — must be left alone.
{"system_id": "sys-c", "ai_flags": {"summaries_enabled": False}},
# No overrides at all: already inherits, nothing to cascade to.
{"system_id": "sys-d"},
{"system_id": "sys-e", "ai_flags": {}},
]
async def _run_set_flags(updates, *, stored=None, systems=None, cascade=False, actor=None,
audit_side_effect=None):
stored = stored if stored is not None else {"stt_enabled": True, "correlation_enabled": True}
systems = systems if systems is not None else _systems()
updates_sink: list = []
audit_mock = AsyncMock(side_effect=audit_side_effect)
with patch.object(feature_flags, "fstore", _fstore_mock(stored, systems, updates_sink)), \
patch("app.internal.audit.write_audit", new=audit_mock):
result = await feature_flags.set_flags(updates, actor=actor, cascade=cascade)
return result, stored, updates_sink, audit_mock
@pytest.mark.asyncio
async def test_set_flags_is_backward_compatible_without_actor_or_cascade():
"""Existing call shape — set_flags({...}) — must keep working."""
result, stored, updates_sink, audit_mock = await _run_set_flags({"stt_enabled": False})
assert result["stt_enabled"] is False
assert stored["stt_enabled"] is False
assert updates_sink == [] # no cascade unless asked
assert audit_mock.await_count == 1 # but still audited
@pytest.mark.asyncio
async def test_audit_records_before_and_after_and_the_actor():
_, _, _, audit_mock = await _run_set_flags(
{"stt_enabled": False},
actor=(auth.AGENT_PRINCIPAL_UID, auth.AGENT_PRINCIPAL_EMAIL),
)
kwargs = audit_mock.await_args.kwargs
assert kwargs["action"] == "feature_flags.update"
assert kwargs["actor_uid"] == auth.AGENT_PRINCIPAL_UID
assert kwargs["actor_email"] == auth.AGENT_PRINCIPAL_EMAIL
details = kwargs["details"]
assert details["changed"]["stt_enabled"] == {"from": True, "to": False}
assert details["before"]["stt_enabled"] is True
assert details["after"]["stt_enabled"] is False
@pytest.mark.asyncio
async def test_audit_failure_neither_loses_the_write_nor_raises():
"""audit_log is a record OF the write, never a precondition for it."""
result, stored, _, audit_mock = await _run_set_flags(
{"stt_enabled": False},
audit_side_effect=RuntimeError("firestore down"),
)
assert audit_mock.await_count == 1
assert stored["stt_enabled"] is False # flag write survived
assert result["stt_enabled"] is False # and the route returns normally
@pytest.mark.asyncio
async def test_cascade_clears_matching_system_overrides_at_both_levels():
result, stored, updates_sink, audit_mock = await _run_set_flags(
{"stt_enabled": True}, cascade=True,
)
# Global level.
assert stored["stt_enabled"] is True
assert result["stt_enabled"] is True
# System level: only the two documents whose ai_flags carry stt_enabled.
written = {sid: data["ai_flags"] for _, sid, data in updates_sink}
assert set(written) == {"sys-a", "sys-b"}
# The flipped key is removed so the system inherits; unrelated overrides stay.
assert written["sys-a"] == {"correlation_enabled": False}
assert written["sys-b"] == {}
# And the cascade is recorded, per system, in the audit entry.
cascaded = audit_mock.await_args.kwargs["details"]["cascaded_systems"]
assert {c["system_id"] for c in cascaded} == {"sys-a", "sys-b"}
assert cascaded[0]["cleared_overrides"] == {"stt_enabled": False}
@pytest.mark.asyncio
async def test_cascade_finds_systems_by_shape_not_by_hardcoded_id():
"""A newly added system carrying an override must not defeat a flip."""
systems = _systems() + [{"system_id": "sys-new", "ai_flags": {"stt_enabled": False}}]
_, _, updates_sink, _ = await _run_set_flags(
{"stt_enabled": True}, systems=systems, cascade=True,
)
assert "sys-new" in {sid for _, sid, _ in updates_sink}
@pytest.mark.asyncio
async def test_cascade_off_leaves_every_system_override_intact():
"""The default path must not silently erase a deliberate per-system value."""
_, _, updates_sink, _ = await _run_set_flags({"stt_enabled": True}, cascade=False)
assert updates_sink == []
@pytest.mark.asyncio
async def test_cascade_error_on_one_system_does_not_stop_the_others():
systems = _systems()
stored = {"stt_enabled": True, "correlation_enabled": True}
updates_sink: list = []
fs = _fstore_mock(stored, systems, updates_sink)
real_update = fs.doc_update.side_effect
async def flaky(collection, doc_id, data):
if doc_id == "sys-a":
raise RuntimeError("write conflict")
return await real_update(collection, doc_id, data)
fs.doc_update = AsyncMock(side_effect=flaky)
audit_mock = AsyncMock()
with patch.object(feature_flags, "fstore", fs), \
patch("app.internal.audit.write_audit", new=audit_mock):
await feature_flags.set_flags({"stt_enabled": True}, cascade=True)
assert [sid for _, sid, _ in updates_sink] == ["sys-b"]
details = audit_mock.await_args.kwargs["details"]
assert [e["system_id"] for e in details["cascade_errors"]] == ["sys-a"]
@pytest.mark.asyncio
async def test_unrecognised_keys_still_raise():
with pytest.raises(ValueError):
await _run_set_flags({"not_a_flag": True})
+252
View File
@@ -0,0 +1,252 @@
"""
The AI feature flags have to be an enforceable statement about the system,
not just about the ingest path (server-26#75, server-26#76).
Three defects motivate these tests:
#75 Correlation read the raw global config/ai_features flag instead of the
per-system resolution, so a system that had opted out via its own
ai_flags still correlated -- with empty tags, down the thin/recency
path, blindly attaching to whatever incident was most recent.
#76 Transcript correction and the transcript-PATCH extraction path checked
no Firestore flag at all, so "AI is off" still spent money.
Plus the destructive half of PATCH /calls/{id}/transcript, which wipes a
call's intelligence fields on the promise that re-extraction rebuilds them.
Firestore and the lazily-imported pipeline modules are fully mocked; the
functions are called directly rather than through FastAPI.
"""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from fastapi import HTTPException
from app.routers import upload, calls
from app.internal import summarizer, transcription
ALL_ON = {
"stt_enabled": True,
"correlation_enabled": True,
"summaries_enabled": True,
"vocabulary_learning_enabled": True,
"transcript_correction_enabled": True,
}
def _flags(**overrides):
return {**ALL_ON, **overrides}
def _system(ai_flags):
return {"system_id": "sys-1", "ai_flags": ai_flags or {}}
def _patch_flags(global_flags, system_ai_flags):
"""Patch the two reads resolve_flags() makes: the global doc and the system doc."""
return (
patch("app.internal.feature_flags.get_flags", AsyncMock(return_value=global_flags)),
patch("app.internal.firestore.doc_get_cached",
AsyncMock(return_value=_system(system_ai_flags))),
)
# --------------------------------------------------------------------------
# resolve_flags: global master off beats everything, system false beats
# global true, absent system key inherits global.
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"global_on, system_ai_flags, expected",
[
(True, {"correlation_enabled": False}, False), # #75: system opt-out holds
(True, {}, True), # absent -> inherit global
(True, {"correlation_enabled": True}, True),
(False, {"correlation_enabled": True}, False), # global is the master switch
(False, {}, False),
],
)
@pytest.mark.asyncio
async def test_resolve_flags_precedence(global_on, system_ai_flags, expected):
g, sysdoc = _patch_flags(_flags(correlation_enabled=global_on), system_ai_flags)
with g, sysdoc:
_, flag = await upload._resolve_flags("sys-1")
assert flag("correlation_enabled") is expected
@pytest.mark.asyncio
async def test_resolve_flags_without_a_system_id_does_not_read_the_system_doc():
with patch("app.internal.feature_flags.get_flags", AsyncMock(return_value=_flags())), \
patch("app.internal.firestore.doc_get_cached", AsyncMock()) as cached:
_, flag = await upload._resolve_flags(None)
assert flag("correlation_enabled") is True
cached.assert_not_awaited()
# --------------------------------------------------------------------------
# The ingest path. This is the exact shape of #75: with the global on and the
# system opted out, extraction was skipped but the empty-scenes fallback still
# ran, correlating the call with no tags and attaching it to whatever incident
# was most recent on that system.
# --------------------------------------------------------------------------
async def _run_ingest(global_correlation, system_ai_flags):
g, sysdoc = _patch_flags(
_flags(correlation_enabled=global_correlation), system_ai_flags
)
with g, sysdoc, \
patch.object(upload, "fstore") as fs, \
patch.object(upload, "_correlate_with_consensus", AsyncMock(return_value=None)) as corr, \
patch("app.internal.transcription.transcribe_call",
AsyncMock(return_value=("units respond to main street", []))), \
patch("app.internal.intelligence.extract_scenes", AsyncMock(return_value=[])) as scenes, \
patch("app.internal.alerter.check_and_dispatch", AsyncMock()):
fs.doc_get = AsyncMock(return_value={})
fs.doc_set = AsyncMock()
await upload._run_intelligence_pipeline(
call_id="call-1",
node_id="node-1",
system_id="sys-1",
talkgroup_id=101,
talkgroup_name="PD Dispatch",
gcs_uri="gs://bucket/call-1.mp3",
)
return scenes, corr
@pytest.mark.asyncio
async def test_per_system_opt_out_blocks_the_blind_recency_fallback_too():
scenes, corr = await _run_ingest(True, {"correlation_enabled": False})
scenes.assert_not_awaited()
# The regression that mattered: the no-scenes fallback correlating on empty tags.
corr.assert_not_awaited()
@pytest.mark.asyncio
async def test_ingest_correlates_when_the_system_has_not_opted_out():
scenes, corr = await _run_ingest(True, {})
scenes.assert_awaited_once()
corr.assert_awaited_once()
# --------------------------------------------------------------------------
# _run_extraction_pipeline -- the transcript-PATCH path (#76).
# --------------------------------------------------------------------------
async def _run_extraction(global_correlation, system_ai_flags=None):
g, sysdoc = _patch_flags(
_flags(correlation_enabled=global_correlation), system_ai_flags
)
with g, sysdoc, \
patch.object(upload, "fstore") as fs, \
patch("app.internal.intelligence.extract_scenes", AsyncMock(return_value=[])) as scenes, \
patch("app.internal.alerter.check_and_dispatch", AsyncMock()) as alert:
fs.doc_set = AsyncMock()
await upload._run_extraction_pipeline(
call_id="call-1",
node_id="node-1",
system_id="sys-1",
talkgroup_id=101,
talkgroup_name="PD Dispatch",
transcript="units respond to main street",
)
return scenes, alert, fs
@pytest.mark.asyncio
async def test_extraction_does_not_spend_when_correlation_is_off():
scenes, alert, fs = await _run_extraction(False)
scenes.assert_not_awaited()
# No incidents produced, so nothing may be stamped onto the call doc.
fs.doc_set.assert_not_awaited()
# Alerting is rule-based and free -- it still runs.
alert.assert_awaited_once()
@pytest.mark.asyncio
async def test_extraction_respects_a_per_system_opt_out():
scenes, _alert, _fs = await _run_extraction(True, {"correlation_enabled": False})
scenes.assert_not_awaited()
@pytest.mark.asyncio
async def test_extraction_runs_when_the_flag_is_on():
scenes, _alert, _fs = await _run_extraction(True)
scenes.assert_awaited_once()
# --------------------------------------------------------------------------
# PATCH /calls/{id}/transcript is destructive before it is constructive.
# With correlation off it must refuse rather than blank the call out.
# --------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_transcript_patch_refuses_when_correlation_is_off():
g, sysdoc = _patch_flags(_flags(correlation_enabled=False), {})
with g, sysdoc, patch.object(calls, "fstore") as fs:
fs.doc_get = AsyncMock(return_value={"call_id": "call-1", "system_id": "sys-1"})
fs.doc_set = AsyncMock()
with pytest.raises(HTTPException) as exc:
await calls.patch_transcript(
call_id="call-1",
body=MagicMock(transcript="corrected text"),
background_tasks=MagicMock(),
_={},
)
assert exc.value.status_code == 409
# The refusal has to land before the first write, or the call is already ruined.
fs.doc_set.assert_not_awaited()
# --------------------------------------------------------------------------
# Transcript correction is a second model call plus a Places lookup (#76).
# --------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_transcript_correction_is_skipped_when_its_flag_is_off():
g, sysdoc = _patch_flags(_flags(transcript_correction_enabled=False), {})
with g, sysdoc, \
patch.object(transcription, "fstore") as fs, \
patch.object(transcription, "ai_health") as health, \
patch.object(transcription, "transcript_correction") as tc, \
patch("asyncio.to_thread", AsyncMock(return_value=("units respond", [], False))):
fs.doc_set = AsyncMock()
health.report_healthy = AsyncMock()
health.report_failure = AsyncMock()
tc.correct = AsyncMock()
await transcription.transcribe_call(
"call-1", "gs://bucket/call-1.mp3", "PD Dispatch", system_id="sys-1"
)
tc.correct.assert_not_awaited()
# --------------------------------------------------------------------------
# Summarizer: the flag guards model spend, not the free Firestore sweep.
# --------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_summarize_incident_is_a_no_op_when_summaries_are_off():
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value=_flags(summaries_enabled=False))), \
patch.object(summarizer, "fstore") as fs, \
patch.object(summarizer, "_sync_summarize") as sync:
fs.doc_get = AsyncMock()
fs.doc_set = AsyncMock()
await summarizer._summarize_incident(
{"incident_id": "inc-1", "call_ids": ["call-1"]}
)
sync.assert_not_called()
fs.doc_get.assert_not_awaited()
fs.doc_set.assert_not_awaited()
+201
View File
@@ -0,0 +1,201 @@
"""
Unit tests for app.internal.ai_health — the shared AI-provider degradation
registry added for logan/server-26#14 (no alerting when a provider account
runs dry or a model is retired).
Covers:
* classify() telling a permanent condition (dead model, depleted billing —
both of which can arrive as the same HTTP status a rate limit uses) apart
from a transient one.
* report_degraded() alerting immediately for a permanent condition but only
after TRANSIENT_ALERT_THRESHOLD consecutive failures for a transient one.
* Alerting exactly once per episode, not once per call, and again exactly
once on recovery.
* report_healthy() clearing degraded state so a later re-degradation can
alert again (a fresh episode, not a continuation of the old one).
The Discord webhook is patched at ai_health._post_webhook so no real HTTP is
made; settings.ai_alert_webhook_url is irrelevant to these tests since
_post_webhook itself is replaced.
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import ai_health
@pytest.fixture(autouse=True)
def _reset_state():
"""Every test gets a clean registry — module-level state persists otherwise."""
ai_health._state = {t: ai_health._default_state() for t in ai_health.TIERS}
yield
ai_health._state = {t: ai_health._default_state() for t in ai_health.TIERS}
# ---------------------------------------------------------------------------
# classify()
# ---------------------------------------------------------------------------
def test_classify_dead_model_404():
assert ai_health.classify("404 models/gemini-2.0-flash is not found") == "dead_model"
def test_classify_dead_model_no_longer_available():
assert ai_health.classify("this model is no longer available") == "dead_model"
def test_classify_billing_depleted_credits():
# The exact wording that bit the Gemini correlator on 2026-08-18.
assert ai_health.classify("429 prepayment credits are depleted") == "billing"
def test_classify_billing_openai_insufficient_quota():
assert ai_health.classify("Error: insufficient_quota — exceeded your current quota") == "billing"
def test_classify_ordinary_rate_limit_is_transient():
# Same HTTP status (429) as the depleted-balance case, but no billing
# wording — this must NOT be classified as billing.
assert ai_health.classify("429 Too Many Requests, please retry later") == "transient"
def test_classify_network_error_is_transient():
assert ai_health.classify("Connection reset by peer") == "transient"
# ---------------------------------------------------------------------------
# report_degraded — permanent alerts immediately
# ---------------------------------------------------------------------------
async def test_permanent_failure_alerts_on_first_occurrence():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
await ai_health.report_degraded(
"correlation_cheap", "gemini", "gemini-2.0-flash",
"model is unavailable", "update the model ID", permanent=True,
)
webhook.assert_awaited_once()
state = ai_health.snapshot()["correlation_cheap"]
assert state["degraded"] is True
assert state["alerted"] is True
assert state["permanent"] is True
async def test_permanent_failure_alerts_only_once_per_episode():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
for _ in range(5):
await ai_health.report_degraded(
"correlation_cheap", "gemini", "gemini-2.0-flash",
"model is unavailable", "update the model ID", permanent=True,
)
# Once per episode, not once per call — this runs at radio-traffic volume.
webhook.assert_awaited_once()
assert ai_health.snapshot()["correlation_cheap"]["consecutive_failures"] == 5
# ---------------------------------------------------------------------------
# report_degraded — transient only alerts once it persists
# ---------------------------------------------------------------------------
async def test_transient_failure_does_not_alert_below_threshold():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
for _ in range(ai_health.TRANSIENT_ALERT_THRESHOLD - 1):
await ai_health.report_degraded(
"transcription", "openai", "whisper-1",
"transient API error", "no action needed unless this persists",
permanent=False,
)
webhook.assert_not_awaited()
state = ai_health.snapshot()["transcription"]
assert state["degraded"] is False
assert state["alerted"] is False
async def test_transient_failure_alerts_once_threshold_crossed():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
for _ in range(ai_health.TRANSIENT_ALERT_THRESHOLD):
await ai_health.report_degraded(
"transcription", "openai", "whisper-1",
"transient API error", "no action needed unless this persists",
permanent=False,
)
webhook.assert_awaited_once()
# Further failures in the same episode must not re-alert.
await ai_health.report_degraded(
"transcription", "openai", "whisper-1",
"transient API error", "no action needed unless this persists",
permanent=False,
)
webhook.assert_awaited_once()
# ---------------------------------------------------------------------------
# report_healthy — recovery
# ---------------------------------------------------------------------------
async def test_recovery_alerts_once_after_an_alerted_episode():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
await ai_health.report_degraded(
"correlation_smart", "gemini", "gemini-1.5-pro",
"the Gemini account is out of credit", "top up billing", permanent=True,
)
webhook.reset_mock()
await ai_health.report_healthy("correlation_smart")
webhook.assert_awaited_once()
state = ai_health.snapshot()["correlation_smart"]
assert state["degraded"] is False
assert state["alerted"] is False
assert state["consecutive_failures"] == 0
async def test_recovery_from_never_alerted_transient_state_is_silent():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
# Two failures — below the transient threshold, never alerted.
await ai_health.report_degraded(
"extraction", "gemini", "gemini-3.6-flash",
"transient API error", "no action needed unless this persists",
permanent=False,
)
await ai_health.report_degraded(
"extraction", "gemini", "gemini-3.6-flash",
"transient API error", "no action needed unless this persists",
permanent=False,
)
webhook.reset_mock()
await ai_health.report_healthy("extraction")
# Nothing was ever posted for this episode, so recovery posts nothing either.
webhook.assert_not_awaited()
async def test_recovery_then_re_degradation_alerts_again_as_a_new_episode():
with patch.object(ai_health, "_post_webhook", new=AsyncMock()) as webhook:
await ai_health.report_degraded(
"correlation_cheap", "gemini", "gemini-2.0-flash",
"model is unavailable", "update the model ID", permanent=True,
)
await ai_health.report_healthy("correlation_cheap")
webhook.reset_mock()
# A second, later episode must alert on its own first occurrence.
await ai_health.report_degraded(
"correlation_cheap", "gemini", "gemini-2.0-flash",
"model is unavailable", "update the model ID", permanent=True,
)
webhook.assert_awaited_once()
# ---------------------------------------------------------------------------
# snapshot()
# ---------------------------------------------------------------------------
async def test_snapshot_reports_all_tiers_healthy_by_default():
state = ai_health.snapshot()
assert set(state.keys()) == set(ai_health.TIERS)
for tier_state in state.values():
assert tier_state["degraded"] is False
assert tier_state["consecutive_failures"] == 0
+178
View File
@@ -0,0 +1,178 @@
"""
Alert payload redaction — server-26#85.
Board minutes #42 suppress person names on every surface until E&O is bound.
A Discord webhook is the least recoverable surface the system has: once the
text is in a channel we do not own it, cannot unsend it, and cannot audit who
read it. These tests pin the default-closed behaviour so it cannot regress
quietly the way it shipped.
The transcript below deliberately contains a person name; every assertion is
"this string did not leave the process", not "some flag was set".
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.config import settings
from app.internal import alerter
TRANSCRIPT = "Units respond, subject identified as Michael Brennan, 42 Elm Street"
ORG = "org-1"
RULE = {
"rule_id": "r1",
"name": "Structure fire",
"enabled": True,
"keywords": ["respond"],
"discord_webhook": "https://discord.example/webhook",
}
@pytest.fixture
def captured(monkeypatch):
"""Capture what alerter would write to Firestore and POST outbound."""
saved: list[dict] = []
posted: list[dict] = []
async def _doc_set(collection, doc_id, data, merge=False):
saved.append(data)
async def _post(url, json=None, **kwargs):
posted.append(json or {})
class _R:
status_code = 204
return _R()
monkeypatch.setattr(alerter.fstore, "doc_set", _doc_set)
monkeypatch.setattr(
alerter.fstore, "collection_list", AsyncMock(return_value=[dict(RULE)])
)
return saved, posted, _post
async def _run(captured, org_doc):
saved, posted, _post = captured
with patch.object(
alerter.fstore,
"doc_get",
AsyncMock(side_effect=lambda c, i: {"org_id": ORG} if c == "calls" else org_doc),
):
client = AsyncMock()
client.post = _post
with patch("httpx.AsyncClient") as ac:
ac.return_value.__aenter__.return_value = client
await alerter.check_and_dispatch(
call_id="c1",
node_id="n1",
talkgroup_id=1,
talkgroup_name="Fire Dispatch",
tags=[],
transcript=TRANSCRIPT,
)
return saved, posted
def _blob(payloads) -> str:
return " ".join(str(p) for p in payloads)
@pytest.mark.asyncio
async def test_webhook_carries_no_transcript_by_default(captured):
"""The shipped default must not put raw transcript text on the wire."""
saved, posted = await _run(captured, {})
assert posted, "the webhook should still fire — alerting is not disabled, only the text is"
assert "Michael Brennan" not in _blob(posted)
assert "Elm Street" not in _blob(posted)
# The alert is still useful: it names the rule and the talkgroup.
assert "Structure fire" in _blob(posted)
@pytest.mark.asyncio
async def test_alert_event_stores_no_transcript_by_default(captured):
"""Firestore is a surface too — the frontend reads it directly."""
saved, _ = await _run(captured, {})
assert saved, "the alert event should still be recorded"
assert saved[0]["transcript_snippet"] is None
assert "Michael Brennan" not in _blob(saved)
@pytest.mark.asyncio
async def test_org_opt_in_alone_does_not_open_the_gate(captured):
"""
An org owner writing their own org document must not be able to opt
themselves into receiving somebody else's PII. The operator switch is
the control; the org flag is only consent.
"""
assert settings.alert_transcript_snippet_enabled is False
saved, posted = await _run(captured, {"alert_snippet_opt_in": True})
assert "Michael Brennan" not in _blob(posted)
assert "Michael Brennan" not in _blob(saved)
@pytest.mark.asyncio
async def test_both_gates_open_emits_the_snippet(monkeypatch, captured):
"""The opt-in path still works, so this is a gate and not a deletion."""
monkeypatch.setattr(settings, "alert_transcript_snippet_enabled", True)
saved, posted = await _run(captured, {"alert_snippet_opt_in": True})
assert "Michael Brennan" in _blob(posted)
assert saved[0]["transcript_snippet"] is not None
@pytest.mark.asyncio
async def test_operator_switch_alone_does_not_open_the_gate(monkeypatch, captured):
"""Consent is required as well as capability."""
monkeypatch.setattr(settings, "alert_transcript_snippet_enabled", True)
saved, posted = await _run(captured, {})
assert "Michael Brennan" not in _blob(posted)
assert saved[0]["transcript_snippet"] is None
@pytest.mark.asyncio
async def test_unreadable_org_fails_closed(monkeypatch, captured):
"""A Firestore error must withhold the transcript, not default to sending it."""
monkeypatch.setattr(settings, "alert_transcript_snippet_enabled", True)
saved, posted, _post = captured
async def _doc_get(collection, doc_id):
if collection == "calls":
return {"org_id": ORG}
raise RuntimeError("firestore unavailable")
with patch.object(alerter.fstore, "doc_get", _doc_get):
client = AsyncMock()
client.post = _post
with patch("httpx.AsyncClient") as ac:
ac.return_value.__aenter__.return_value = client
await alerter.check_and_dispatch(
call_id="c1", node_id="n1", talkgroup_id=1,
talkgroup_name="Fire Dispatch", tags=[], transcript=TRANSCRIPT,
)
assert "Michael Brennan" not in _blob(posted)
assert saved[0]["transcript_snippet"] is None
@pytest.mark.asyncio
async def test_pre_tenancy_call_with_no_org_fails_closed(monkeypatch, captured):
"""A call with no org_id has nobody who could have consented to anything."""
monkeypatch.setattr(settings, "alert_transcript_snippet_enabled", True)
saved, posted, _post = captured
with patch.object(alerter.fstore, "doc_get", AsyncMock(return_value={})):
client = AsyncMock()
client.post = _post
with patch("httpx.AsyncClient") as ac:
ac.return_value.__aenter__.return_value = client
await alerter.check_and_dispatch(
call_id="c1", node_id="n1", talkgroup_id=1,
talkgroup_name="Fire Dispatch", tags=[], transcript=TRANSCRIPT,
)
assert "Michael Brennan" not in _blob(posted)
assert saved[0]["transcript_snippet"] is None
+305
View File
@@ -0,0 +1,305 @@
"""
Unit tests for the area_context schema and anchor (server-26#36).
Three properties carry the real risk:
* NULLABILITY IS THE MECHANISM. Which scope an operator fills is their
declaration of how homogeneous the system is. Merging must let a talkgroup
narrow the system without dropping what the system already said — a
talkgroup that sets only a town must still inherit the state, or "Ossining"
is nationally ambiguous again.
* NO ANCHOR IS BETTER THAN A USELESS ONE. An anchor wider than
area_anchor_max_radius_km, or one whose resolved_from no longer matches the
place it came from, must read as ABSENT. Verification then skips. Treating
either as usable would rubber-stamp any location while looking like a check.
* THE CLIENT DOES NOT WRITE SERVER FIELDS. The systems form sends
config.talkgroups[] in full; taking it verbatim destroys the resolved anchor
and the pending queue, which is the same bug as the ten_codes wipe.
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import area_context as ac
SYSTEM_AREA = {
"county": "Westchester",
"state": "New York",
"local_knowledge": [{"term": "Route 9", "meaning": "state highway"}],
"center": {"lat": 41.1, "lng": -73.8},
"radius_km": 30.0,
"resolved_from": "|westchester|new york",
"resolved_at": "2026-08-23T00:00:00+00:00",
}
TG_AREA = {
"municipality": "Ossining",
"local_knowledge": [{"term": "Sing Sing", "meaning": "state prison"}],
"center": {"lat": 41.16, "lng": -73.86},
"radius_km": 6.0,
"resolved_from": "ossining|westchester|new york",
"resolved_at": "2026-08-23T00:00:00+00:00",
}
# -- Merging -------------------------------------------------------------------
def test_talkgroup_narrows_without_dropping_the_system():
merged = ac.effective(SYSTEM_AREA, TG_AREA)
assert merged["municipality"] == "Ossining"
assert merged["county"] == "Westchester"
assert merged["state"] == "New York", "the state must survive the narrowing"
def test_talkgroup_knowledge_ranks_first_and_dedupes():
system = {"local_knowledge": [{"term": "Route 9"}, {"term": "Metro-North"}]}
tg = {"local_knowledge": [{"term": "route 9", "meaning": "the local name"}]}
merged = ac.effective(system, tg)
assert [e["term"] for e in merged["local_knowledge"]] == ["route 9", "Metro-North"]
assert merged["local_knowledge"][0]["meaning"] == "the local name"
def test_empty_at_both_scopes_is_legal():
assert ac.effective(None, None) == {}
assert ac.effective({}, {}) == {}
def test_bare_strings_are_accepted_as_terms():
"""roads[]/landmarks[] from the old shape, and anything a model returns."""
assert ac.normalize_local_knowledge(["Route 9", "", "Route 9", 7]) == [{"term": "Route 9"}]
def test_pre_36_roads_and_landmarks_are_read_forward():
"""
Real systems still have the old shape stored. Dropping it the day this
shipped would silently discard ground truth an operator already entered.
"""
legacy = {"county": "Westchester", "roads": ["Route 9"], "landmarks": ["Sing Sing"]}
merged = ac.effective(legacy, None)
assert [e["term"] for e in merged["local_knowledge"]] == ["Route 9", "Sing Sing"]
assert ac.normalize(legacy) == {
"county": "Westchester",
"local_knowledge": [{"term": "Route 9"}, {"term": "Sing Sing"}],
}, "and the next save writes them in the new shape"
def test_normalize_drops_client_sent_server_fields():
out = ac.normalize({"municipality": " Ossining ", "radius_km": 5000, "center": {"lat": 0}})
assert out == {"municipality": "Ossining"}
# -- Anchor selection ----------------------------------------------------------
def test_talkgroup_anchor_wins():
anchor = ac.anchor_for(SYSTEM_AREA, TG_AREA)
assert anchor == {"lat": 41.16, "lng": -73.86, "radius_km": 6.0}
def test_system_anchor_used_when_talkgroup_sets_no_place():
anchor = ac.anchor_for(SYSTEM_AREA, {"local_knowledge": [{"term": "Post 4"}]})
assert anchor == {"lat": 41.1, "lng": -73.8, "radius_km": 30.0}
def test_no_anchor_when_nothing_is_configured():
assert ac.anchor_for({}, {}) is None
def test_stale_anchor_reads_as_absent():
"""
Someone edited the town and the refresh has not run yet. The stored centre
is for the OLD place, so using it would validate locations against an area
the channel no longer covers.
"""
stale = {**TG_AREA, "municipality": "Croton"}
assert ac.anchor_for(SYSTEM_AREA, stale) is None
def test_anchor_key_ignores_case_and_padding():
assert ac.anchor_key({"municipality": " OSSINING "}) == ac.anchor_key({"municipality": "ossining"})
# -- Anchor resolution ---------------------------------------------------------
def _maps(viewport_span_deg: float):
"""A geocode response whose viewport spans roughly the given degrees."""
payload = {
"status": "OK",
"results": [{
"geometry": {
"location": {"lat": 41.0, "lng": -73.0},
"viewport": {
"northeast": {"lat": 41.0 + viewport_span_deg, "lng": -73.0 + viewport_span_deg},
"southwest": {"lat": 41.0 - viewport_span_deg, "lng": -73.0 - viewport_span_deg},
},
}
}],
}
class _Resp:
def raise_for_status(self): pass
def json(self): return payload
class _Client:
async def __aenter__(self): return self
async def __aexit__(self, *a): return False
async def get(self, *a, **k): return _Resp()
return patch("httpx.AsyncClient", lambda *a, **k: _Client())
@pytest.fixture(autouse=True)
def _clear_cache():
ac._anchor_cache.clear()
with patch.object(ac.settings, "google_maps_api_key", "test-key"):
yield
ac._anchor_cache.clear()
@pytest.mark.asyncio
async def test_small_place_produces_an_anchor():
with _maps(0.05):
anchor = await ac.resolve_anchor({"municipality": "Ossining", "state": "New York"})
assert anchor is not None
assert anchor["radius_km"] < 10
assert anchor["resolved_from"] == "ossining||new york"
@pytest.mark.asyncio
async def test_statewide_place_produces_no_anchor():
"""
A radius that covers a state would confirm any location inside it. Storing
it would make the geocode check worse than useless — it would look like
verification and pass everything.
"""
with _maps(4.0), patch.object(ac.settings, "area_anchor_max_radius_km", 60.0):
assert await ac.resolve_anchor({"state": "Colorado"}) is None
@pytest.mark.asyncio
async def test_no_place_never_calls_maps():
with patch("httpx.AsyncClient") as client:
assert await ac.resolve_anchor({"local_knowledge": [{"term": "Post 4"}]}) is None
client.assert_not_called()
@pytest.mark.asyncio
async def test_refresh_skips_scopes_whose_place_is_unchanged():
doc = {"area_context": SYSTEM_AREA, "config": {"talkgroups": [{"id": 1, "area_context": TG_AREA}]}}
with patch("httpx.AsyncClient") as client:
assert await ac.refresh_anchors(doc) == {}
client.assert_not_called()
@pytest.mark.asyncio
async def test_editing_the_system_place_re_anchors_its_talkgroups():
"""
A talkgroup's anchor derives from its EFFECTIVE place, so changing the
system's county silently changes what every talkgroup should be anchored to.
"""
doc = {
"area_context": {"county": "Putnam", "state": "New York"},
"config": {"talkgroups": [{"id": 1, "area_context": {"municipality": "Ossining"}}]},
}
with _maps(0.05):
patch_out = await ac.refresh_anchors(doc)
tg = patch_out["config"]["talkgroups"][0]
assert tg["area_context"]["resolved_from"] == "ossining|putnam|new york"
assert tg["area_context"]["center"]["lat"] == 41.0
# -- Client writes -------------------------------------------------------------
def test_merge_config_preserves_the_anchor_and_the_pending_queue():
existing = {"talkgroups": [{
"id": 9048,
"area_context": TG_AREA,
ac.PENDING_KEY: [{"term": "Snowden Avenue"}],
}]}
# What the systems form actually sends: no anchor, no pending queue.
incoming = {"talkgroups": [{"id": 9048, "name": "Ossining PD",
"area_context": {"municipality": "Ossining"}}]}
merged = ac.merge_config(incoming, existing)
tg = merged["talkgroups"][0]
assert tg["area_context"]["center"] == TG_AREA["center"]
assert tg[ac.PENDING_KEY] == [{"term": "Snowden Avenue"}]
assert tg["name"] == "Ossining PD", "the client still owns the fields it owns"
def test_merge_config_drops_an_emptied_area():
existing = {"talkgroups": [{"id": 1, "area_context": TG_AREA}]}
merged = ac.merge_config({"talkgroups": [{"id": 1}]}, existing)
assert "area_context" not in merged["talkgroups"][0]
# -- Pending terms -------------------------------------------------------------
def _store(doc):
saved = {}
async def _get(_col, _id):
return doc
async def _update(_col, _id, patch):
saved.update(patch)
return saved, patch.multiple(
"app.internal.firestore", doc_get=AsyncMock(side_effect=_get),
doc_update=AsyncMock(side_effect=_update),
)
@pytest.mark.asyncio
async def test_pending_terms_land_on_the_talkgroup():
doc = {"config": {"talkgroups": [{"id": 9048}]}, "vocabulary": []}
saved, store = _store(doc)
with store:
assert await ac.add_pending("sys-1", 9048, [{"term": "Snowden Avenue"}]) == 1
assert saved["config"]["talkgroups"][0][ac.PENDING_KEY][0]["term"] == "Snowden Avenue"
assert "vocabulary" not in saved, "nothing writes to the system"
@pytest.mark.asyncio
async def test_already_known_terms_are_not_re_proposed():
doc = {
"vocabulary": ["Metro-North"],
"area_context": {"local_knowledge": [{"term": "Route 9"}]},
"config": {"talkgroups": [{"id": 9048, "local_knowledge_pending": [{"term": "Sing Sing"}]}]},
}
saved, store = _store(doc)
with store:
queued = await ac.add_pending("sys-1", 9048, [
{"term": "route 9"}, {"term": "Metro-North"}, {"term": "sing sing"},
])
assert queued == 0
assert saved == {}
@pytest.mark.asyncio
async def test_approving_writes_to_the_talkgroup_and_never_the_system():
"""
Blast radius: the same term at system level misleads every channel on the
system, including one 400km away on a statewide system.
"""
doc = {"config": {"talkgroups": [{"id": 9048, ac.PENDING_KEY: [
{"term": "Snowden Avenue", "meaning": "residential street"}]}]}}
saved, store = _store(doc)
with store:
assert await ac.resolve_pending("sys-1", 9048, "snowden avenue", approve=True) is True
tg = saved["config"]["talkgroups"][0]
assert tg["area_context"]["local_knowledge"] == [
{"term": "Snowden Avenue", "meaning": "residential street"}
]
assert tg[ac.PENDING_KEY] == []
assert "vocabulary" not in saved and "area_context" not in saved
@pytest.mark.asyncio
async def test_dismissing_adds_nothing():
doc = {"config": {"talkgroups": [{"id": 9048, ac.PENDING_KEY: [{"term": "Optum"}]}]}}
saved, store = _store(doc)
with store:
assert await ac.resolve_pending("sys-1", 9048, "Optum", approve=False) is True
tg = saved["config"]["talkgroups"][0]
assert tg[ac.PENDING_KEY] == []
assert not (tg.get("area_context") or {}).get("local_knowledge")
+59
View File
@@ -0,0 +1,59 @@
"""calls._parse_ts — cursor/date bounds must reach Firestore as datetimes.
A raw ISO string compared against a timestamp field sorts by type, not time,
which made the Archive's "Load more" return the first page again.
"""
from datetime import datetime, timezone
import pytest
from fastapi import HTTPException
from app.routers.calls import _next_cursor, _parse_ts
def test_empty_is_none():
assert _parse_ts(None, "cursor") is None
assert _parse_ts("", "cursor") is None
def test_z_suffix_parses_as_utc():
assert _parse_ts("2026-09-20T12:00:00Z", "date_from") == datetime(2026, 9, 20, 12, tzinfo=timezone.utc)
def test_naive_is_assumed_utc():
assert _parse_ts("2026-09-20T12:00:00", "date_to").tzinfo == timezone.utc
def test_round_trips_isoformat_cursor():
dt = datetime(2026, 9, 20, 12, 30, 5, 123456, tzinfo=timezone.utc)
assert _parse_ts(dt.isoformat(), "cursor") == dt
def test_garbage_is_400():
with pytest.raises(HTTPException) as exc:
_parse_ts("yesterday", "date_from")
assert exc.value.status_code == 400
# ── _next_cursor ──────────────────────────────────────────────────────────
def _rows(n):
return [{"started_at": datetime(2026, 9, 20, 12, i // 60, i % 60, tzinfo=timezone.utc)} for i in range(n)]
def test_cursor_resumes_after_last_returned_row_when_matches_overflow():
rows = _rows(200)
page = rows[:50]
assert _next_cursor(rows, rows, page, 200) == page[-1]["started_at"].isoformat()
def test_cursor_resumes_after_window_when_page_holds_every_match():
rows = _rows(200)
matches = rows[:3]
assert _next_cursor(rows, matches, matches, 200) == rows[-1]["started_at"].isoformat()
def test_short_window_is_the_end():
rows = _rows(20)
assert _next_cursor(rows, rows[:5], rows[:5], 200) is None
@@ -0,0 +1,144 @@
"""
server-26#127 — upstream dispatch-vs-chatter classifier, shadow mode.
Fixtures are real transcripts, not invented ones: pulled from
`corr_dump_9-7_0437am.json`, `corr_dump_9-7_pm.json`, `corr_dump_9-12.json`
and the hand-labeled examples in `CORRELATION_REVIEW_0907b.md` /
`CORRELATION_REVIEW_0912.md`. The "must classify False" set specifically
includes every transcript those review docs flagged as dangerous to drop —
a false positive here is a real event silently losing its scene once this
classifier ever goes live, which is a much worse failure than a missed
chatter call staying in the existing (already-working) pipeline.
"""
import pytest
from app.internal.chatter_classifier import classify_chatter
# ─────────────────────────────────────────────────────────────────────────────
# Must classify as chatter
# ─────────────────────────────────────────────────────────────────────────────
CHATTER_EXAMPLES = [
# Bare acknowledgements / unit check-ins (CORRELATION_REVIEW_0907b.md)
("114 Paul.\n114 Paul, Metro Central.\n10-4.", "bare_acknowledgement"),
("Affirmative, in charge of 10-8. 10-8, 10-4.", "bare_acknowledgement"),
("6-8, you can show me 98. 10-4.", "bare_acknowledgement"),
("10-4, 10-4 Central, 98. 10-4, 98.", "bare_acknowledgement"),
("7 for Post 1 and 2, 98. Affirm.", "bare_acknowledgement"),
("11-Victor to Central. 11-Victor. 72-Holland, 1-5. Central.", "bare_acknowledgement"),
# Roll call (CORRELATION_REVIEW_0907b.md / _0912.md)
("Post 4, Ossining. And to volunteer patrol, stand by for roll call.", "roll_call"),
("Headquarters to all cars, stand by for roll call.", "roll_call"),
("All Troop NYC Patrols, stand by for roll call.", "roll_call"),
(
"Car 100, roll call.\nHenry 1.\nHenry 1.\nSam 1.\nSam 1.\n45 Baker.\n"
"45 Baker.\n11 Adam.\nAdam.\n11 Baker.\nBaker.\nStaff 1.\n1.\nStaff 2.",
"roll_call",
),
(
"Headquarters, all cars on a roll call. Baker 1? Baker 1. Henry 1? "
"Henry 1. Sam 2? Sam 2. 11 Adam? 11. 11 Baker? 11 Baker.",
"roll_call",
),
("Because all cars came out for roll call.", "roll_call"),
("10-1. KL Cars, that concludes roll call, time is 3-31.", "roll_call"),
# Minimal single-word / bare-code transmissions (orphan pool, all 3 dumps)
("10-4.", "bare_acknowledgement"),
("Roger.", "bare_acknowledgement"),
("Clear.", "bare_acknowledgement"),
("Affirmative.", "bare_acknowledgement"),
("Received.", "bare_acknowledgement"),
("10-8, clear. 10-4.", "bare_acknowledgement"),
("Post 4, 10-8. 10-4.", "bare_acknowledgement"),
("Central to 6 Henry.", "bare_acknowledgement"),
]
@pytest.mark.parametrize("transcript,expected_reason", CHATTER_EXAMPLES)
def test_classifies_chatter(transcript, expected_reason):
is_chatter, reason = classify_chatter(transcript)
assert is_chatter is True
assert reason == expected_reason
# ─────────────────────────────────────────────────────────────────────────────
# Must NOT classify as chatter — real events, including every transcript the
# review docs specifically named as dangerous to drop.
# ─────────────────────────────────────────────────────────────────────────────
REAL_EVENT_EXAMPLES = [
# The major "extinguishing fire" call (severity=major, tags=[extinguishing-fire])
("Dispatch, this is 7-4, extinguishing fire.", "extinguishing_fire"),
# Geocoded 911-hangup call (has location_coords)
(
"7, Charlie. Charlie, check and advise, we've got a call for service "
"coming over, it's going to be a 9-1-1 hangout, no voice contact. "
"Looks like it was an automated message saying it's the Doral Hat Company.",
"geocoded_911_hangup",
),
# Pursuit updates (severity=major, tags include pursuit / low-speed-pursuit)
("I'm aware of that one. It's a low-speed pursuit. It's refusing to pull over.", "low_speed_pursuit"),
(
"1. Headquarters to 5-charlie. I'm going to say the last thing to anyone.\n"
"2. Info, Sgt. Repeat.\n"
"3. The SP is on a pursuit southbound on I-684. It's approaching the airport.\n"
"4. Okay, thank you.\n5. 23-59.",
"pursuit_i684",
),
# "6 Alpha ... Pelham Station" subject check (CORRELATION_REVIEW_0907b.md's
# own "genuinely distinct events" list) — looks like a bare check-in but
# dispatches a unit to a specific location.
(
"6 Alpha, this is Central. 7 Alpha here.\n"
"6 Alpha, can you show me on scene at Pelham Station? Stand by.",
"pelham_station_subject_check",
),
# Property-retrieval call (tags=[property-retrieval])
(
"Property was retrieved with a 911. Can I get a phone number? 10-4. "
"Phone number is 214792. 214792.",
"property_retrieval",
),
# Subject check south of Maronex Station (tags=[subject-check])
(
"Proceed. Show me on a subject south of Maronex Station. Can I get a "
"15 check by New York client ID?",
"maronex_subject_check",
),
# Trespassing at milepost 13.7 (tags=[trespassing])
(
"Can you just 10-5 that job? You came over real muffled.\n"
"10-4, there's going to be a trespass on the tracks.\n"
"Train 8755 reports two juveniles, one male, one female, both wearing "
"white shirts, track three side, at milepost 13.7.",
"trespass_milepost_13_7",
),
# MVA (severity=moderate, tags=[traffic-accident])
("1. Train patrol 9.\n2. MVA 4, how close is it?\n3. 10-4.", "mva"),
]
@pytest.mark.parametrize("transcript,label", REAL_EVENT_EXAMPLES, ids=[l for _, l in REAL_EVENT_EXAMPLES])
def test_does_not_classify_real_events_as_chatter(transcript, label):
is_chatter, reason = classify_chatter(transcript)
assert is_chatter is False, f"{label}: false positive, reason={reason!r}"
assert reason is None
# ─────────────────────────────────────────────────────────────────────────────
# Edge cases
# ─────────────────────────────────────────────────────────────────────────────
def test_empty_transcript_not_chatter():
assert classify_chatter("") == (False, None)
assert classify_chatter(None) == (False, None)
assert classify_chatter(" ") == (False, None)
def test_unrecognized_content_defaults_to_not_chatter():
# Anything containing real descriptive words the classifier doesn't
# recognize must fall through to (False, None), not guess.
is_chatter, reason = classify_chatter("Shots fired, officer down, requesting backup immediately.")
assert is_chatter is False
assert reason is None
@@ -0,0 +1,142 @@
"""
Dispatch→10-8 lifecycle, as measured by the first replay (server-26#170):
0 of 19 incidents resolved on a clear although 25 transmissions said one.
Three independent breaks, each pinned here.
"""
from app.internal import incident_correlator as ic
from app.internal.intelligence import _clearance_scene, _short_clearance_unit
def test_short_clearance_names_the_unit_that_cleared():
assert _short_clearance_unit("45-9, I'm clear.") == "45-9"
assert _short_clearance_unit("Vehicle 1, clear.") == "Vehicle 1"
assert _short_clearance_unit("11 Adam, clear") == "11 Adam"
assert _short_clearance_unit("Car 12 10-8") == "Car 12"
assert _short_clearance_unit("45 9 clear") == "45-9"
assert _short_clearance_unit("Warrant 4, clear from the jail") is None or True # >5 words: GPT's job
assert _short_clearance_unit("45-9 clear, thank you") == "45-9"
def test_short_clearance_never_guesses():
for t in ("10-8, 10-8.", "CMT clear.", "10-8, I'm back now. Clear.",
"10-8, thank you.", "Show us 10-8, post 4.", "7, Charlie Central.", "10-4.",
# review findings: questions, negations, orders, places, times
"45-9, are you clear?", "Is 45-9 clear", "45-9, not clear yet.",
"Engine 5 not available.", "45-9, clear the scene.", "Medic 3, clear to transport.",
"Room 2 clear.", "Route 9 is clear.", "1400 hours, clear.", "you clear 45-9"):
assert _short_clearance_unit(t) is None, t
def test_clearance_scene_cannot_open_an_incident():
scene = _clearance_scene("45-9, I'm clear.", "45-9")
ctx = {"call_vehicles": scene["vehicles"], "coords": scene["location_coords"], "tags": scene["tags"]}
assert not ic.has_event_substance(ctx)
assert scene["severity"] == "routine" and scene["incident_type"] is None
def test_clearance_matches_a_differently_spoken_unit():
inc = {"units_active": ["11 Adam", "45-9"], "units_cleared": []}
active, cleared, resolved = ic._apply_unit_clearance(inc, ["11-Adam"])
assert active == ["45-9"]
active, cleared, resolved = ic._apply_unit_clearance(
{"units_active": active, "units_cleared": cleared}, ["45 9"])
assert active == [] and resolved
def test_a_unit_never_on_the_incident_cannot_close_it():
inc = {"units_active": ["45-9"], "units_cleared": []}
active, cleared, resolved = ic._apply_unit_clearance(inc, ["22-1"])
assert active == ["45-9"] and cleared == [] and not resolved
# an incident with no numbered unit ever active never resolves on a clear
active, cleared, resolved = ic._apply_unit_clearance({"units_active": [], "units_cleared": []}, ["22-1"])
assert not resolved
def test_clearance_only_call_skips_the_llm():
from app.internal import llm_correlator
ctx = {"call_units": ["45-9"], "call_cleared": ["45-9"], "tags": [], "location": None,
"call_vehicles": [], "incident_type": None}
assert llm_correlator._is_clearance_only(ctx)
assert not llm_correlator._is_clearance_only({**ctx, "tags": ["mva"]})
assert not llm_correlator._is_clearance_only({**ctx, "call_cleared": []})
def test_only_numbered_units_hold_an_incident_open():
for junk in ("Desk", "Central", "Division", "sergeant", "unknown", "John", "Zebra", "10-8", "10 4"):
assert not ic._is_trackable_unit(junk), junk
for real in ("45-9", "11-Adam", "Whitestone 1", "E-14", "Highway 3-4", "7"):
assert ic._is_trackable_unit(real), real
# ---------------------------------------------------------------------------
# Provisional timer close + reopen, substantive cap (server-26#170 replay)
# ---------------------------------------------------------------------------
from datetime import datetime, timedelta, timezone # noqa: E402
from app.config import settings # noqa: E402
from app.internal import summarizer # noqa: E402
def test_quiet_timer_scales_with_severity():
assert summarizer._auto_resolve_minutes({"severity": "routine"}) == settings.incident_auto_resolve_minutes_routine
assert summarizer._auto_resolve_minutes({"severity": "minor"}) == settings.incident_auto_resolve_minutes_routine
assert summarizer._auto_resolve_minutes({"severity": "moderate"}) == settings.incident_auto_resolve_minutes_moderate
assert summarizer._auto_resolve_minutes({"severity": "major"}) == settings.incident_auto_resolve_minutes
assert summarizer._auto_resolve_minutes({}) == settings.incident_auto_resolve_minutes
def test_thin_calls_do_not_fill_the_call_cap():
now = datetime(2026, 9, 22, 15, 0, tzinfo=timezone.utc)
inc = {"call_ids": [f"c{i}" for i in range(60)], "substantive_call_count": 12,
"started_at": (now - timedelta(minutes=40)).isoformat(),
"updated_at": now.isoformat()}
assert ic._incident_at_capacity(inc, now) is None
legacy = {k: v for k, v in inc.items() if k != "substantive_call_count"}
assert ic._incident_at_capacity(legacy, now).startswith("call_cap")
def test_reopen_only_for_a_call_after_the_close():
closed = {"resolved_at": "2026-09-22T15:00:00+00:00"}
assert ic._after_close(closed, datetime(2026, 9, 22, 15, 5, tzinfo=timezone.utc))
assert not ic._after_close(closed, datetime(2026, 9, 22, 14, 55, tzinfo=timezone.utc))
def test_traffic_stops_become_events():
from app.internal.intelligence import _self_initiated_backstop as b
for t in ("45 Adam on a stop, Eastbound Central Express.",
"11-0. CM2 on the stop, southbound, KFLA on the right.",
"Car 7 on a traffic stop, Route 9 at Main"):
tags, typ, sev = b(t, [], None, "routine")
assert "traffic-stop" in tags and typ == "police" and sev == "minor", t
tags, typ, sev = b("Charlie 1. You put me out with a pedestrian on a parkway", [], None, "routine")
assert "self-initiated" in tags
# negation and unrelated chatter stay untouched
assert b("Do you want me to not pull the car over", [], None, "routine") == ([], None, "routine")
assert b("45-8, go ahead.", [], None, "routine") == ([], None, "routine")
# an existing type/severity is never downgraded
assert b("on a stop", ["dwi"], "police", "moderate") == (["dwi", "traffic-stop"], "police", "moderate")
def test_stop_backstop_stays_off_rail_bridge_and_ems_channels():
from app.internal.intelligence import _self_initiated_backstop as b
none = ([], None, "routine")
assert b("Train 4 holding on the stop at Grand Central", [], None, "routine",
"MTA PD Districts 6/7/11 - Police Dispatch") == none
assert b("out with a disabled on the bridge, toll plaza", [], None, "routine",
"MTA Bridges and Tunnels - Whitestone/Throgs Neck Bridge Patrols") == none
assert b("Medic 2 pull over to the side and wait", [], None, "routine") == none
assert b("ran a plate for a car stop", [], None, "routine") == none
assert b("I dont think he is on a stop", [], None, "routine") == none
assert b("45 Adam on a stop", [], None, "routine", "Ch 1 (Patched with 155.310)")[0] == ["traffic-stop"]
def test_short_stop_report_opens_a_scene():
import asyncio
from unittest.mock import patch
from app.internal import firestore as fstore, intelligence
async def run():
with patch.object(fstore, "doc_set"), patch.object(fstore, "doc_get_cached", return_value=None):
return await intelligence.extract_scenes("c1", "Adam 3 on a stop.", "Ch 1 (Patched with 155.310)")
scenes = asyncio.run(run())
assert len(scenes) == 1 and scenes[0]["tags"] == ["traffic-stop"] and scenes[0]["incident_type"] == "police"
+430
View File
@@ -0,0 +1,430 @@
"""
server-26#115 — two consensus-quality fixes.
Fix 1 (routers/upload.py): when the cheap LLM says `orphan`, the rules engine
says `new`, and the call is genuinely SUBSTANCELESS (routine severity, no
vehicle/geocode/tag, and no incident already running on the same talkgroup),
resolve to `orphan` and DO NOT pay for the smart tiebreaker. Radio housekeeping
(unit check-ins, roll call, 10-8/10-98) was being promoted to incidents because
the tiebreaker rubber-stamped the rules `new` ~21/21 of the time
(CORRELATION_REVIEW_0907b.md).
The substance test runs against `ctx` (fully populated at preview time), NOT
against `rules_decision["corr_debug"]` — that dict is EMPTY at preview time for
action=="new" (corr_path:"new" is written at APPLY time), so the first version of
this gate fired on real events (a `major` "extinguishing fire", geocoded calls,
pursuit updates).
Fix 2 (incident_correlator.py): the `location` correlation path linked on a bare
sub-`location_proximity_km` (0.5 km) distance alone, taking whichever incident
came first in an unsorted `recent`. In a dense village two unrelated events
routinely geocode that close. A `location` link now needs unit overlap with the
candidate OR a distance under a tighter bar, and picks the NEAREST qualifying
candidate. A unit-overlap location link is tagged `location_unit_overlap` so it
does not merge into the fast path's bucket in the admin fit-signal histogram.
"""
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
import pytest
from app.routers import upload
from app.internal.incident_correlator import _run_decision, has_event_substance
NOW = datetime(2026, 9, 7, 21, 30, 0, tzinfo=timezone.utc)
# ─────────────────────────────────────────────────────────────────────────────
# Fix 1 — the LLM-orphan gate in _correlate_with_consensus
# ─────────────────────────────────────────────────────────────────────────────
def _preview(action, corr_debug=None, ctx=None):
base_ctx = {"call_id": "call-1"}
if ctx:
base_ctx.update(ctx)
return {
"decision": {
"action": action,
"matched_incident": None,
"incident_type": "other" if action == "new" else None,
"corr_debug": {} if corr_debug is None else dict(corr_debug),
},
"ctx": base_ctx,
}
def _llm(action, reasoning="—"):
md = {"incident_id": "inc-1"} if action == "link" else None
return {"action": action, "matched_incident": md, "reasoning": reasoning}
async def _run_consensus(preview, llm_decision):
tiebreak_result = {
"action": "new", "matched_incident": None, "incident_type": "other",
"corr_debug": {}, "reasoning": "tb",
}
with patch("app.internal.incident_correlator.preview_correlation",
new=AsyncMock(return_value=preview)), \
patch("app.internal.incident_correlator.apply_correlation",
new=AsyncMock(return_value="incident-x")) as m_apply, \
patch("app.internal.llm_correlator.decide",
new=AsyncMock(return_value=llm_decision)), \
patch("app.internal.llm_correlator.tiebreak",
new=AsyncMock(return_value=tiebreak_result)) as m_tiebreak:
await upload._correlate_with_consensus(
call_id="call-1", node_id="n1", system_id="sys-1",
talkgroup_id=9048, talkgroup_name="Dispatch", tags=[],
incident_type=None, location=None, location_coords=None,
)
return m_apply, m_tiebreak
async def test_substanceless_no_recent_same_tg_incident_gates_without_tiebreak():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}), _llm("orphan", "unit check-in, not an incident"),
)
m_tiebreak.assert_not_called()
m_apply.assert_called_once()
gated = m_apply.call_args[0][0]["decision"]
assert gated["action"] == "orphan"
dbg = gated["corr_debug"]
assert dbg["corr_consensus"] == "llm_orphan_gate"
assert dbg["corr_consensus"] != "tiebreak"
assert dbg["corr_rules_action"] == "new"
assert dbg["corr_llm_action"] == "orphan"
assert dbg["corr_llm_reasoning"] == "unit check-in, not an incident"
@pytest.mark.parametrize("severity", ["moderate", "major"])
async def test_moderate_or_major_severity_is_not_gated(severity):
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_severity": severity}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_routine_severity_alone_still_gates():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_severity": "routine"}), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_call_with_coords_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"coords": {"lat": 41.15, "lng": -73.86}}),
_llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_tags_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"tags": ["structure-fire"]}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_vehicles_is_not_gated():
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"call_vehicles": ["red sedan"]}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_call_with_resolved_incident_type_is_not_gated():
# The creation gate skips has_event_substance when a type resolved, so a
# typed call (fire/medical/…) opens an incident on substance the gate does
# not re-check — it must keep the tiebreak, not be dropped.
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"incident_type": "fire"}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_reassignment_call_is_not_gated():
# reassignment=True is dispatch pulling a unit onto a NEW job (units are
# blanked for exactly that reason) — the strongest new-incident signal.
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx={"reassignment": True}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_recent_incident_on_same_talkgroup_is_not_gated():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-live",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
"updated_at": (NOW - timedelta(minutes=1)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
# server-26#115 window #3 (CORRELATION_REVIEW_0912.md): the escape hatch used
# to treat ANY same-talkgroup incident inside the 2h correlation_window_hours
# as "recent", which on a busy dispatch channel (3-13 incidents/2h) was
# satisfied almost unconditionally — the gate fired 0/24 times against its own
# target shape. It now only counts an incident as recent within
# settings.tg_dispatch_thin_idle_minutes (5 min), applied uniformly regardless
# of the talkgroup's name (owner correction, 2026-09-13 — see
# test_channel_name_does_not_affect_the_window below for why the dichotomy
# this originally had with incident_correlator's fast/thin idle selection was
# removed here).
async def test_recent_same_tg_incident_inside_new_short_window_still_escapes_gate():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-live",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
# 3 min ago — inside tg_dispatch_thin_idle_minutes (5).
"updated_at": (NOW - timedelta(minutes=3)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
async def test_recent_same_tg_incident_older_than_short_window_now_gates():
# Regression test for the fix: 8 minutes is past the 5-minute bound but
# still inside the OLD 2-hour correlation_window_hours lookback. Before
# the fix this escaped the gate on any channel; after the fix it gates.
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"talkgroup_name": "Dispatch",
"now": NOW,
"recent": [{
"incident_id": "inc-stale",
"system_ids": ["sys-1"],
"talkgroup_ids": ["9048"],
"updated_at": (NOW - timedelta(minutes=8)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_channel_name_does_not_affect_the_window():
# Owner correction, 2026-09-13 (direct scanning experience): a talkgroup
# named "tac"/"tactical" only sees materially different traffic during a
# real incident, and that's rare -- the bulk of traffic on any monitored
# channel, including high-risk stops and pursuits, runs on the main
# channel regardless of what it's named. An earlier version of this used
# a longer 15-minute window on anything not literally named "dispatch"/
# "patched"/"primary" (mirroring incident_correlator's fast/thin idle
# selection); that meant a busy single-channel department not literally
# named "dispatch" silently got the more permissive window and could
# reproduce #115's original bug. Same 8-minute age as the dispatch-named
# test above, but on a channel named "Tac 3" -- must gate identically,
# not escape into a longer window just because of the name.
ctx = {
"system_id": "sys-1",
"talkgroup_id": 383,
"talkgroup_name": "Tac 3",
"now": NOW,
"recent": [{
"incident_id": "inc-tac",
"system_ids": ["sys-1"],
"talkgroup_ids": ["383"],
"updated_at": (NOW - timedelta(minutes=8)).isoformat(),
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_gate_veto_reason_is_recorded_on_the_escalation_path():
# server-26#115: a live measurement window must be able to see *why* an
# llm=orphan/rules=new call escaped the gate without guessing from the raw
# dump (which produced a wrong "confirmed explanation" for 2 window-#3
# misses the first time). corr_gate_veto names the surviving condition.
ctx = {"call_severity": "major"}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_called_once()
final = m_apply.call_args[0][0]["decision"]
assert final["corr_debug"]["corr_gate_veto"] == "severity"
async def test_gate_veto_reason_is_absent_when_the_disagreement_is_not_orphan_vs_new():
# corr_gate_veto is only meaningful for the llm=orphan/rules=new shape the
# gate targets — it must not appear (or be misleadingly None-vs-absent) on
# an unrelated disagreement shape.
m_apply, m_tiebreak = await _run_consensus(
_preview("link", {}), _llm("orphan"),
)
m_tiebreak.assert_called_once()
final = m_apply.call_args[0][0]["decision"]
assert "corr_gate_veto" not in final["corr_debug"]
async def test_recent_incident_on_a_different_talkgroup_still_gates():
ctx = {
"system_id": "sys-1",
"talkgroup_id": 9048,
"recent": [{
"incident_id": "inc-other",
"system_ids": ["sys-1"],
"talkgroup_ids": ["1200"],
}],
}
m_apply, m_tiebreak = await _run_consensus(
_preview("new", {}, ctx=ctx), _llm("orphan"),
)
m_tiebreak.assert_not_called()
assert m_apply.call_args[0][0]["decision"]["action"] == "orphan"
async def test_llm_link_vs_rules_new_still_escalates():
m_apply, m_tiebreak = await _run_consensus(_preview("new", {}), _llm("link", "same job"))
m_tiebreak.assert_called_once()
async def test_llm_orphan_vs_rules_link_still_escalates():
# Not the gate condition (gate needs rules=="new"); must fall through.
m_apply, m_tiebreak = await _run_consensus(_preview("link", {}), _llm("orphan"))
m_tiebreak.assert_called_once()
def test_has_event_substance_predicate():
assert has_event_substance({"coords": {"lat": 1, "lng": 2}})
assert has_event_substance({"tags": ["fire"]})
assert has_event_substance({"call_vehicles": ["sedan"]})
assert not has_event_substance({})
assert not has_event_substance({"coords": None, "tags": [], "call_vehicles": []})
# units and location are NOT substance — nearly every transmission has them.
assert not has_event_substance({"call_units": ["7-Adam"], "location": "Main St"})
# ─────────────────────────────────────────────────────────────────────────────
# Fix 2 — tighten corr_path=location
# ─────────────────────────────────────────────────────────────────────────────
CALL_COORDS = {"lat": 41.150000, "lng": -73.860000}
# ~0.39 km north of the call — inside location_proximity_km (0.5) but well
# outside the tight bar (_LOCATION_TIGHT_PROXIMITY_KM, 0.2).
FAR_INC_COORDS = {"lat": 41.153500, "lng": -73.860000}
# ~0.13 km north of the call — inside the tight bar.
NEAR_INC_COORDS = {"lat": 41.151200, "lng": -73.860000}
# ~0.28 km north — inside the 0.5 radius, outside the 0.2 tight bar; used as a
# second candidate that must lose the nearest-wins sort to NEAR_INC_COORDS.
MID_INC_COORDS = {"lat": 41.152500, "lng": -73.860000}
def _inc(incident_id, coords, units):
return {
"incident_id": incident_id,
"system_ids": ["sys-1"],
"talkgroup_ids": ["100"], # different TGID → fast path is a no-op
"location_coords": coords,
"units": units,
"tags": [],
"type": "police",
"updated_at": (NOW - timedelta(minutes=6)).isoformat(),
"started_at": (NOW - timedelta(minutes=20)).isoformat(),
"status": "active",
"call_ids": ["c0"],
}
def _loc_ctx(*, incidents, call_units):
return {
"call_id": "call-loc",
"all_active": list(incidents),
"recent": list(incidents),
"call_doc": {},
"call_embedding": None,
"call_units": call_units,
"call_vehicles": [],
"call_cleared": [],
"call_severity": "routine",
"coords": CALL_COORDS,
"is_thin_call": False,
"now": NOW,
"system_id": "sys-1",
"talkgroup_id": 999, # not in inc.talkgroup_ids
"talkgroup_name": "Tactical",
"tags": [],
"incident_type": "police",
"location": "Main St",
"location_coords": CALL_COORDS,
"reassignment": True, # suppress the unit-continuity path
"create_if_new": True,
}
def test_location_path_in_radius_but_no_unit_overlap_no_tight_proximity_does_not_link(caplog):
ctx = _loc_ctx(
incidents=[_inc("inc-loc", FAR_INC_COORDS, ["7-Adam"])],
call_units=["3-Boy"],
)
with caplog.at_level("INFO", logger="drb-c2-core"):
decision = _run_decision(ctx)
# Reaches, and is rejected by, the new guard (not an earlier path).
assert "location-path skipped" in caplog.text
assert decision["action"] != "link"
assert (decision.get("corr_debug") or {}).get("corr_path") != "location"
def test_location_path_links_on_unit_overlap_with_distinct_fit_signal():
ctx = _loc_ctx(
incidents=[_inc("inc-loc", FAR_INC_COORDS, ["5-Adam"])],
call_units=["5-Adam"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "location"
# NOT "unit_overlap" — that value belongs to the fast path's histogram bucket.
assert decision["corr_debug"]["corr_fit_signal"] == "location_unit_overlap"
def test_location_path_links_on_tight_proximity_without_unit_overlap():
ctx = _loc_ctx(
incidents=[_inc("inc-loc", NEAR_INC_COORDS, ["7-Adam"])],
call_units=["3-Boy"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "location"
assert decision["corr_debug"]["corr_fit_signal"] == "location_proximity"
def test_location_path_picks_nearest_in_radius_candidate():
# `recent` order puts the farther tight-proximity incident first; the guard
# must still select the nearest one.
ctx = _loc_ctx(
incidents=[
_inc("inc-mid", MID_INC_COORDS, ["3-Boy"]), # ~0.28 km, tight-fail
_inc("inc-near", NEAR_INC_COORDS, ["3-Boy"]), # ~0.13 km, tight-pass
],
call_units=["3-Boy"],
)
decision = _run_decision(ctx)
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-near"
assert decision["corr_debug"]["corr_path"] == "location"
+67
View File
@@ -0,0 +1,67 @@
"""
server-26#115 — the tiebreaker manufactured incidents because it was blind to
what would tell it two incidents are one.
Two low-risk supports for the reframed prompt:
1. `_extract_road_ids` collapses street-type synonyms, so "Mohegan Park Ave"
and "Mohegan Park Avenue" share a road id (they were splitting one
car-alarm incident into two).
2. `_inc_summary` now carries the incident title and talkgroup, the two
signals the model needs to recognise a same-channel continuation.
"""
from datetime import datetime, timezone
from app.internal.incident_correlator import (
_extract_road_ids, _location_mentions_road_overlap,
)
from app.internal.llm_correlator import _inc_summary, _prompt_incidents
NOW = datetime(2026, 9, 7, 8, 0, 0, tzinfo=timezone.utc)
def test_avenue_and_ave_are_the_same_road_id():
assert _extract_road_ids("Mohegan Park Avenue") == _extract_road_ids("Mohegan Park Ave")
assert _extract_road_ids("191 Broadway Street") == _extract_road_ids("191 Broadway St")
assert _extract_road_ids("North State Road") == _extract_road_ids("North State Rd")
def test_road_overlap_matches_across_the_synonym():
assert _location_mentions_road_overlap("multiple car alarms Mohegan Park Avenue",
["patrol to Mohegan Park Ave"]) is True
# still discriminates genuinely different streets
assert _location_mentions_road_overlap("Oak Avenue", ["Elm Avenue"]) is False
def test_inc_summary_carries_title_and_talkgroup():
s = _inc_summary({
"incident_id": "abc123",
"type": "police",
"talkgroup_ids": [9560],
"title": "Nuisance Alarm at Mohegan Park Ave",
"location": "Mohegan Park Ave",
"units": ["Headquarters"],
"tags": ["car-alarm"],
"updated_at": NOW.isoformat(),
}, NOW)
assert "title:'Nuisance Alarm at Mohegan Park Ave'" in s
assert "tg:[9560]" in s
assert "id:abc123" in s
def test_inc_summary_omits_missing_optional_fields():
s = _inc_summary({"incident_id": "x", "updated_at": NOW.isoformat()}, NOW)
assert "title:" not in s and "tg:" not in s and "loc:" not in s
assert s.startswith("id:x")
def test_prompt_incidents_is_most_recently_active_first_and_capped():
recent = [
{"incident_id": f"i{n}", "updated_at": f"2026-09-07T0{n}:00:00+00:00"}
for n in range(1, 8)
]
ordered = _prompt_incidents(recent)
assert [i["incident_id"] for i in ordered] == ["i7", "i6", "i5", "i4", "i3", "i2", "i1"]
assert len(_prompt_incidents(recent * 5)) == 20
# falls back to started_at when updated_at is absent, and never raises
assert _prompt_incidents([{"incident_id": "a", "started_at": NOW.isoformat()},
{"incident_id": "b"}])[0]["incident_id"] == "a"
+188
View File
@@ -18,6 +18,7 @@ from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
from app.internal.incident_correlator import (
_run_decision, _update_incident, _normalize_unit, _matching_units,
_max_severity, maybe_resolve_parent,
)
NOW = datetime(2026, 8, 16, 21, 0, 0, tzinfo=timezone.utc)
@@ -247,3 +248,190 @@ def test_normalised_units_link_a_call_that_exact_match_would_orphan():
call_units=["K-9A2"], is_thin_call=False, call_severity="routine",
))
assert decision["action"] == "link"
# ---------------------------------------------------------------------------
# Issue #17 — severity re-evaluation as calls attach (monotonic ladder)
#
# Decision: severity only ever rises, never falls, as more calls link (see
# _max_severity's docstring in incident_correlator.py for the full argument).
# An incident briefly assessed "major" genuinely was major at that moment;
# resolution (status/resolved_at), not a later calmer-sounding call, is what
# retires it. These tests lock in both halves of that: escalation raises the
# stored severity, and a later lower-severity call does not undo it.
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("current,new,expected", [
("routine", "major", "major"), # escalation — the motivating case
("routine", "minor", "minor"),
("minor", "moderate", "moderate"),
("major", "routine", "major"), # calmer call does NOT downgrade
("major", "minor", "major"),
("moderate", "moderate", "moderate"), # tie
(None, "moderate", "moderate"), # incident with no prior severity
("major", None, "major"),
("major", "bogus", "major"), # malformed value ranks as routine
("bogus", "minor", "minor"),
])
def test_max_severity_is_monotonic(current, new, expected):
assert _max_severity(current, new) == expected
@pytest.mark.asyncio
async def test_escalating_call_raises_stored_incident_severity():
"""The #17 motivating case: an incident opened routine, a later call is a
working structure fire — the incident's severity must reflect it."""
inc = _incident(2.0)
inc["severity"] = "routine"
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
inc, "call-2", 9048, "sys-1", [], None, None, [], [], None, NOW,
call_severity="major",
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["severity"] == "major"
@pytest.mark.asyncio
async def test_calmer_followup_call_does_not_downgrade_severity():
inc = _incident(2.0)
inc["severity"] = "major"
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
inc, "call-2", 9048, "sys-1", [], None, None, [], [], None, NOW,
call_severity="routine",
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["severity"] == "major"
# ---------------------------------------------------------------------------
# Issue #18 — every resolution site stamps resolved_at
#
# updated_at is not a substitute (thin/ack calls deliberately don't move it,
# unrelated field updates do) and existing rows are left null, not backfilled
# — null means "resolved before this field existed", not "never resolved".
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_signal_resolve_stamps_resolved_at():
"""All tracked units clear -> _update_incident's own auto-resolve path."""
inc = _incident(2.0)
inc["units_active"] = ["6-Adam"]
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
# standalone incident — maybe_resolve_parent's own doc_get short-circuits on None
mock_fstore.doc_get = AsyncMock(return_value=None)
await _update_incident(
inc, "call-2", 9048, "sys-1", [], None, None, [], [], None, NOW,
cleared_units=["6-Adam"],
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["status"] == "resolved"
assert updates["resolved_at"] == NOW.isoformat()
# ---------------------------------------------------------------------------
# Issue #16 — unit-continuity path must populate corr_matched_units
#
# fast/single and fast/disambig only set corr_matched_units when
# fit_signal == "unit_overlap"; unit-continuity has no such gate because a
# match there is unit-driven by construction (call_unit_set intersects the
# incident's units is literally how unit_candidates gets built) — so it must
# always populate the field, not conditionally.
# ---------------------------------------------------------------------------
def test_unit_continuity_link_reports_matched_units():
"""
Reproduces the server-26#16 production example: call units=["Post 1-2"]
should match an incident with units=["5-4", "9-0-8", "1-2"] via the
normalizer collapsing "Post 1-2" and "1-2" to the same key, on a
DIFFERENT talkgroup than the incident (so the fast/talkgroup path can't
fire first and this falls through to unit-continuity).
"""
inc = _incident(10.0) # idle 10min, within unit_continuity_max_idle_minutes (20)
inc["talkgroup_ids"] = ["1234"]
inc["units"] = ["5-4", "9-0-8", "1-2"]
decision = _run_decision(_ctx(
talkgroup_id=9999, # not in inc["talkgroup_ids"] — fast path can't match
all_active=[inc], recent=[],
call_units=["Post 1-2"], is_thin_call=False, call_severity="routine",
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "unit-continuity"
assert decision["corr_debug"]["corr_matched_units"] == ["Post 1-2"]
# ---------------------------------------------------------------------------
# server-26#24 — updated_at must never precede started_at
#
# The re-correlation sweep anchors `now` to the linking call's own
# started_at, which can be earlier than the incident's own started_at. Left
# unclamped that produces updated_at < started_at on the incident doc, which
# is what caused corr_incident_idle_min: -4.1 in production (commit 33a247d
# fixed the gates reading that negative value, not this write).
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_updated_at_never_precedes_started_at():
inc = _incident(5) # started_at == updated_at == NOW - 5min
inc["started_at"] = NOW.isoformat() # incident "started" at NOW
back_dated_now = NOW - timedelta(minutes=30) # a much older orphan call links in
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
inc, "call-1", 9048, "sys-1", [], None, None, ["6-Adam"], [], None,
back_dated_now,
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["updated_at"] == NOW.isoformat(), (
"updated_at must be floored at started_at, not the back-dated `now`"
)
@pytest.mark.asyncio
async def test_updated_at_uses_now_when_now_is_later_than_started_at():
"""The normal case (now is not back-dated before started_at) is unaffected."""
inc = _incident(5)
later_now = NOW + timedelta(minutes=1)
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
inc, "call-1", 9048, "sys-1", [], None, None, ["6-Adam"], [], None,
later_now,
)
updates = mock_fstore.doc_set.await_args.args[2]
assert updates["updated_at"] == later_now.isoformat()
@pytest.mark.asyncio
async def test_master_auto_resolve_stamps_resolved_at():
"""maybe_resolve_parent closes a master once every child has resolved."""
child_a = {"incident_id": "child-a", "parent_incident_id": "master-1"}
master = {
"incident_id": "master-1",
"status": "active",
"child_incident_ids": ["child-a", "child-b"],
}
child_b_resolved = {"incident_id": "child-b", "status": "resolved"}
async def fake_doc_get(collection, doc_id):
return {
"child-a": child_a,
"master-1": master,
"child-b": child_b_resolved,
}.get(doc_id)
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(side_effect=fake_doc_get)
mock_fstore.doc_set = AsyncMock()
await maybe_resolve_parent("child-a")
mock_fstore.doc_set.assert_awaited_once()
args = mock_fstore.doc_set.await_args.args
assert args[0] == "incidents"
assert args[1] == "master-1"
assert args[2]["status"] == "resolved"
assert "resolved_at" in args[2] and args[2]["resolved_at"]
@@ -0,0 +1,551 @@
"""
Over-merge guards — server-26#22.
All of this comes from the 2026-08-20 production dump (CORRELATION_REVIEW_0820.md),
where 4 of 6 sampled incidents were junk chains and the worst, `f5190670`, was
68 calls over 4h09m carrying 44 units, 12 tags and at least 13 genuinely distinct
events. That is a work shift filed as one incident.
Three defects combined to produce it, and each has cases below:
1. `is_thin_call` was `not units and not vehicles and not coords`, so a real
dispatch with tags and a street address counted as thin whenever no unit ID
parsed and the geocode failed. Thin calls are the one class that links with
NO `_call_fits_incident` check, so those dispatches were force-merged.
2. The thin path was bounded only on dispatch channels. Everywhere else it
used the full 90-minute fast-path window, any number of candidates, no fit.
3. Nothing capped an incident's total size. Every fit test in the correlator
is pairwise, so each individual link can be defensible while the chain they
accumulate is not — no pairwise rule can see the shape.
`_run_decision` and `_is_thin_call` are pure, so none of this needs Firestore.
"""
import pytest
from datetime import datetime, timedelta, timezone
from app.config import settings
import app.internal.incident_correlator as correlator_mod
from app.internal.incident_correlator import (
_run_decision, _is_thin_call, _idle_gate_minutes,
_incident_at_capacity, _incident_span_minutes, _call_fits_incident,
)
NOW = datetime(2026, 8, 20, 7, 0, 0, tzinfo=timezone.utc)
# TG 383 from the dump: "Ch 1 (Patched with 155.310)", a shared dispatch
# backbone carrying the whole department. Kept as two distinct fixture names
# for readability even though the channel's name no longer affects behavior
# (server-26#134).
DISPATCH_TG = "Ch 1 (Patched with 155.310)"
TACTICAL_TG = "Fireground 2"
def _ctx(**overrides) -> dict:
base = {
"call_id": "call-1",
"all_active": [],
"recent": [],
"call_doc": {},
"call_embedding": None,
"call_units": [],
"call_vehicles": [],
"call_cleared": [],
"call_severity": "routine",
"coords": None,
"is_thin_call": True,
"now": NOW,
"system_id": "sys-1",
"talkgroup_id": 383,
"talkgroup_name": DISPATCH_TG,
"tags": [],
"incident_type": None,
"location": None,
"location_coords": None,
"reassignment": False,
"create_if_new": True,
}
base.update(overrides)
return base
def _incident(idle_minutes: float = 0.2, **overrides) -> dict:
updated = NOW - timedelta(minutes=idle_minutes)
inc = {
"incident_id": "inc-1",
"system_ids": ["sys-1"],
"talkgroup_ids": ["383"],
"updated_at": updated.isoformat(),
"started_at": updated.isoformat(),
"status": "active",
"call_ids": ["seed-call"],
}
inc.update(overrides)
return inc
# ---------------------------------------------------------------------------
# 1. What counts as thin — the misclassification that drove the chains
# ---------------------------------------------------------------------------
def test_content_free_acknowledgement_is_thin():
""""10-4." — no unit, no vehicle, no coords, no tags, no place, routine."""
assert _is_thin_call([], [], None, [], None, "routine", False) is True
@pytest.mark.parametrize("field,value", [
("tags", ["welfare-check"]),
("location", "55 Hyman Hills Road"),
("call_severity", "minor"),
("call_severity", "moderate"),
("call_severity", "major"),
])
def test_extracted_content_makes_a_call_substantive(field, value):
"""
The 07:08 call in `f5190670`: "All units head over to the powerhouse, 55
Hyman Hills Road … she's 87 years old" — a brand new job that was called
thin purely because no unit ID parsed and the geocode failed. It attached
with no fit check and then overwrote the four-hour chain's title and pin.
"""
kwargs = {"tags": [], "location": None, "call_severity": "routine"}
kwargs[field] = value
assert _is_thin_call([], [], None, kwargs["tags"], kwargs["location"],
kwargs["call_severity"], False) is False
def test_reassignment_is_never_thin():
"""
upload.py blanks `units` when dispatch pulls a unit onto a NEW job, to stop
unit-overlap chaining. That made the call thin and routed it to the only
path with no fit check — the guard produced the merge it existed to prevent.
"""
assert _is_thin_call([], [], None, [], None, "routine", True) is False
@pytest.mark.parametrize("units,vehicles,coords", [
(["6-Adam"], [], None),
([], ["black Toyota Camry"], None),
([], [], {"lat": 41.08, "lng": -73.81}),
])
def test_original_thinness_signals_still_apply(units, vehicles, coords):
assert _is_thin_call(units, vehicles, coords, [], None, "routine", False) is False
def test_blank_location_string_does_not_make_a_call_substantive():
assert _is_thin_call([], [], None, [], " ", "routine", False) is True
# ---------------------------------------------------------------------------
# 2. A thin call needs a tight window; a real one needs a fit signal
# ---------------------------------------------------------------------------
def test_thin_call_with_no_overlap_does_not_attach_on_a_dispatch_channel():
inc = _incident(idle_minutes=40)
assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan"
def test_thin_call_with_no_overlap_does_not_attach_on_a_tactical_named_channel():
"""A channel's name no longer changes anything (server-26#134) — same
assertion as the dispatch-named case above, different fixture name."""
inc = _incident(idle_minutes=40)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
def test_tactical_named_channel_uses_the_dispatch_window_now():
"""server-26#134: 14 min was inside the old 15-min tactical window; now
every channel uses the 5-min window regardless of name."""
inc = _incident(idle_minutes=14)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
def test_tactical_named_channel_still_attaches_inside_the_dispatch_window():
inc = _incident(idle_minutes=settings.tg_dispatch_thin_idle_minutes - 1)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_path"] == "fast/thin"
def test_tactical_thin_call_is_ambiguous_with_two_candidates():
a = _incident(idle_minutes=3.0, incident_id="inc-a")
b = _incident(idle_minutes=4.0, incident_id="inc-b")
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], talkgroup_name=TACTICAL_TG,
))
assert decision["action"] == "orphan"
# ---------------------------------------------------------------------------
# server-26#158: srcaddr identity beats recency guesswork for thin calls
# ---------------------------------------------------------------------------
def test_thin_call_srcaddr_match_resolves_tier2_ambiguity():
"""
Same fixture as test_tactical_thin_call_is_ambiguous_with_two_candidates —
two candidates, tier-2 window, no unit ID parsed (transcript_too_short
skipped GPT). Without srcaddr this orphans. With it, the radio that sent
the call already touched inc-b, so that's the thread — not a guess.
"""
a = _incident(idle_minutes=3.0, incident_id="inc-a", srcaddrs=["9001"])
b = _incident(idle_minutes=4.0, incident_id="inc-b", srcaddrs=["9002"])
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], talkgroup_name=TACTICAL_TG,
call_srcaddr="9002",
))
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-b"
assert decision["corr_debug"]["corr_fit_signal"] == "thin_srcaddr_match"
def test_thin_call_srcaddr_match_overrides_recency_in_tier1():
"""
Both candidates are inside the 30s conversational window, where recency
alone would pick inc-a (more recently updated) even though the radio that
sent this call has only ever touched inc-b — the exact busy-channel,
two-concurrent-incidents misattach server-26#158 was filed for.
"""
a = _incident(idle_minutes=0.1, incident_id="inc-a", srcaddrs=["9001"])
b = _incident(idle_minutes=0.2, incident_id="inc-b", srcaddrs=["9002"])
decision = _run_decision(_ctx(
all_active=[a, b], recent=[a, b], call_srcaddr="9002",
))
assert decision["action"] == "link"
assert decision["matched_incident"]["incident_id"] == "inc-b"
def test_thin_call_with_no_srcaddr_match_falls_back_to_recency():
"""A radio ID that matches nothing on this talkgroup behaves exactly as
before — no regression for the ordinary case."""
a = _incident(idle_minutes=0.1, incident_id="inc-a", srcaddrs=["9001"])
decision = _run_decision(_ctx(
all_active=[a], recent=[a], call_srcaddr="unrelated-radio",
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_fit_signal"] == "thin_recency"
def test_call_with_unit_overlap_does_attach():
"""
Positive control: real evidence still links. Carrying units also means the
call is not thin, so it reaches _call_fits_incident and passes on
unit_overlap rather than being force-attached.
"""
inc = _incident(idle_minutes=3.0, units=["6-Adam", "K-9A2"])
assert _is_thin_call(["6-Adam"], [], None, [], None, "routine", False) is False
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], call_units=["6-Adam"], is_thin_call=False,
))
assert decision["action"] == "link"
assert decision["corr_debug"]["corr_fit_signal"] == "unit_overlap"
def test_substantive_call_with_no_signal_opens_its_own_incident():
"""
A tagged dispatch on a shared backbone with no unit/vehicle/geocode match
is a separate job, not a follow-up. Under the old thinness test this exact
call took fast/thin and merged.
"""
inc = _incident(idle_minutes=2.0)
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
tags=["welfare-check"], location="55 Hyman Hills Road",
))
assert decision["action"] == "new"
assert decision["incident_type"] == "other"
# ---------------------------------------------------------------------------
# 3. Negative idle — the sweep anchors `now` to the call's own started_at
# ---------------------------------------------------------------------------
def test_idle_gate_uses_distance_not_sign():
"""
Observed on `9d376ffe`: corr_incident_idle_min -4.1, because the sweep
evaluated a 02:45 call against an incident updated at 02:50. Every
`idle <= window` gate reads True for a negative number, so the gates
stopped bounding anything for precisely the calls the sweep re-examines.
"""
future = _incident(idle_minutes=-45)
assert _idle_gate_minutes(future, NOW) == pytest.approx(45.0)
def test_back_dated_thin_call_does_not_sail_through_the_recency_gate():
future = _incident(idle_minutes=-45)
assert _run_decision(_ctx(all_active=[future], recent=[future]))["action"] == "orphan"
def test_back_dated_call_does_not_bypass_the_content_divergence_veto(monkeypatch):
"""
Same `9d376ffe` failure mode, exercised directly against
`_call_fits_incident`: unit overlap plus a back-dated call (incident
updated 45 minutes AFTER the call's own `started_at`, which the sweep
passes as `now`) used to make the signed idle -45, so `idle_min >= 15`
read False and the content-divergence veto never ran — unit overlap
alone forced the merge regardless of what the call was actually about.
With the gate fixed to compare distance, idle_min is 45 (>= 15), the
veto runs, and a divergent embedding (patched below so the assertion
doesn't depend on numpy being installed in this environment) fails it.
"""
monkeypatch.setattr(correlator_mod, "_cosine_similarity", lambda a, b: 0.0)
inc = _incident(idle_minutes=-45, units=["6-Adam"])
inc["embedding"] = [1.0, 0.0]
fits, signal = _call_fits_incident(
inc, call_units=["6-Adam"], call_vehicles=[], call_coords=None,
proximity_km=settings.location_proximity_km,
call_embedding=[0.0, 1.0], now=NOW,
)
assert (fits, signal) == (False, "content_divergence")
# ---------------------------------------------------------------------------
# 4. Hard caps — path-independent, because pairwise fit tests can't see shape
# ---------------------------------------------------------------------------
def _long_running(minutes: float) -> dict:
started = NOW - timedelta(minutes=minutes)
return _incident(idle_minutes=0.2, started_at=started.isoformat())
def test_duration_cap_forces_a_new_incident():
"""
`f5190670` ran 4h09m. The one real incident in the dump ran 63 minutes.
The unit here overlaps the incident's own roster, so without the cap this
links on unit_overlap — the cap is the only thing separating them.
"""
inc = _long_running(settings.incident_max_duration_minutes + 30)
inc["units"] = ["6-Adam"]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
call_units=["6-Adam"], tags=["welfare-check"],
))
assert decision["action"] == "new"
def test_duration_cap_blocks_the_thin_path_too():
"""The cap is checked before any path runs, so 'no fit test' is no escape."""
inc = _long_running(settings.incident_max_duration_minutes + 30)
assert _run_decision(_ctx(all_active=[inc], recent=[inc]))["action"] == "orphan"
def test_incident_just_under_the_duration_cap_still_accepts_calls():
inc = _long_running(settings.incident_max_duration_minutes - 10)
inc["units"] = ["6-Adam"]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False, call_units=["6-Adam"],
))
assert decision["action"] == "link"
def test_call_count_cap_forces_a_new_incident():
inc = _incident(idle_minutes=0.2, units=["6-Adam"])
inc["call_ids"] = [f"c{i}" for i in range(settings.incident_max_calls)]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False,
call_units=["6-Adam"], tags=["vehicle-accident"],
))
assert decision["action"] == "new"
def test_incident_one_call_under_the_count_cap_still_accepts_calls():
inc = _incident(idle_minutes=0.2, units=["6-Adam"])
inc["call_ids"] = [f"c{i}" for i in range(settings.incident_max_calls - 1)]
decision = _run_decision(_ctx(
all_active=[inc], recent=[inc], is_thin_call=False, call_units=["6-Adam"],
))
assert decision["action"] == "link"
@pytest.mark.parametrize("inc,expect", [
(_incident(), None),
(_long_running(9999), "duration"),
])
def test_capacity_reason_names_the_cap_that_fired(inc, expect):
reason = _incident_at_capacity(inc, NOW)
assert (reason is None) if expect is None else reason.startswith(expect)
def test_span_survives_a_back_dated_reference_time():
"""
The sweep passes the call's own started_at as `now`, which can precede the
incident's last update — the span must still reflect what the incident has
actually accumulated, not go negative and defeat the cap.
"""
started = NOW - timedelta(hours=5)
inc = {"started_at": started.isoformat(), "updated_at": NOW.isoformat()}
past = NOW - timedelta(hours=4)
assert _incident_span_minutes(inc, past) == pytest.approx(300.0, abs=0.1)
def test_unparseable_started_at_is_never_capped_on_duration():
assert _incident_span_minutes({"started_at": "not-a-date"}, NOW) == 0.0
# ---------------------------------------------------------------------------
# 5. Regression: the `f5190670` shape must not reassemble
# ---------------------------------------------------------------------------
# The 13 distinct events visible in `f5190670`, at their real offsets from the
# 03:01 opener. Each arrived exactly as reproduced here: tags and often a place
# name, but no parsed unit and no successful geocode — which is what made the
# old thinness test classify them as chatter.
_F5190670_EVENTS = [
(0, ["vehicle-accident", "telephone-pole-strike"], "Airport Road traffic circle"),
(2, ["uber-passenger", "phone-pinging"], "traffic circle near New King Street"),
(13, ["sign-down"], None),
(26, ["burglary-alarm"], "34 Carlton Drive"),
(67, ["inspection"], "2 Filno River Road"),
(108, ["disabled-vehicle"], "Yonkers Ave"),
(119, ["altercation"], "137 East Main Street"),
(131, ["inspection"], "80 North Grasslands Road"),
(156, ["foot-patrol"], "Tanzania Road"),
(211, [], None), # unit roll call — pure noise
(222, ["vehicle-off-roadway"], None),
(240, ["premises-check"], "Hillcrest Drive"),
(247, ["welfare-check"], "55 Hyman Hills Road"),
]
# The department roster heard on that channel. `f5190670` accumulated 44 units,
# partly from the nine phonetic-alphabet roll calls it absorbed, and once an
# incident holds most of the roster essentially every later call overlaps it —
# mechanism B in the review, unit-overlap positive feedback. Reproduced here so
# the chain has a real engine driving it, not just the thin path.
_ROSTER = ["6-Adam", "7-Baker", "11-Victor", "45-Charlie", "K-9A2", "22-47"]
_START = datetime(2026, 5, 24, 3, 1, 0, tzinfo=timezone.utc)
def _overnight_traffic():
"""
The real shape of that channel: a transmission roughly every two minutes for
4h07m — 13 dispatched jobs, routine unit traffic drawn from one roster, and
acknowledgements in between. Yields (offset_min, units, tags, location).
"""
events = {off: (tags, loc) for off, tags, loc in _F5190670_EVENTS}
# The dispatches, at their real offsets, exactly as they arrived: tags and
# usually a place name, but no parsed unit and no successful geocode.
traffic = [(off, [], tags, loc) for off, (tags, loc) in events.items()]
# Mechanism B, the engine that kept the real chain alive for four hours: one
# job that legitimately opens with units, then keeps producing unit traffic
# all shift. Each follow-up genuinely overlaps on unit, so each link is
# individually defensible and each one refreshes updated_at — which keeps
# the incident permanently inside every recency gate. No pairwise fit test
# can refuse these; only a cap can stop the accumulation.
traffic.append((1, [_ROSTER[0]], ["prisoner-transport"], "Medical Center"))
traffic += [(m, [_ROSTER[0]], [], None) for m in range(5, 249, 4)]
# Everything else on the channel — one transmission every two minutes.
for n, minute in enumerate(range(0, 249, 2)):
if minute in events:
continue
if n % 3 == 0:
traffic.append((minute, [_ROSTER[1 + n % (len(_ROSTER) - 1)]], [], None))
else:
traffic.append((minute, [], [], None)) # "10-4"
return traffic
def _simulate(traffic):
"""
Replay traffic through the real decision engine, applying the same incident
mutations the commit layer would: a link appends the call, merges units, and
(unless thin) refreshes updated_at; "new" opens a doc. Returns
(incidents, placement) where placement maps call_id → incident_id.
"""
incidents: list[dict] = []
placement: dict[str, str] = {}
for i, (offset, units, tags, location) in enumerate(sorted(traffic)):
now = _START + timedelta(minutes=offset)
call_id = f"call-{i}"
thin = _is_thin_call(units, [], None, tags, location, "routine", False)
active = [inc for inc in incidents if inc["status"] == "active"]
decision = _run_decision(_ctx(
call_id=call_id, now=now, all_active=active, recent=active,
tags=tags, location=location, call_units=units, is_thin_call=thin,
))
if decision["action"] == "link":
inc = decision["matched_incident"]
inc["call_ids"].append(call_id)
inc["units"] = list(dict.fromkeys(inc["units"] + units))
inc["_last_call_at"] = now
if decision["corr_debug"].get("corr_path") != "fast/thin":
inc["updated_at"] = now.isoformat()
placement[call_id] = inc["incident_id"]
elif decision["action"] == "new":
incidents.append({
"incident_id": f"inc-{i}", "system_ids": ["sys-1"],
"talkgroup_ids": ["383"], "status": "active",
"started_at": now.isoformat(), "updated_at": now.isoformat(),
"call_ids": [call_id], "units": list(units),
"_started": now, "_last_call_at": now, "_offset": offset,
})
placement[call_id] = incidents[-1]["incident_id"]
return incidents, placement
def _live_span_minutes(inc: dict) -> float:
"""Minutes from an incident's first call to the last one it actually took."""
return (inc["_last_call_at"] - inc["_started"]).total_seconds() / 60
def test_f5190670_does_not_become_one_incident():
"""
The headline regression: a full overnight shift on one patched dispatch
backbone must not end up as a single incident. The real one was 68 calls,
4h09m, 44 units, 12 tags and at least 13 distinct events.
"""
traffic = _overnight_traffic()
incidents, placement = _simulate(traffic)
assert len(incidents) >= 12, f"the shift merged into {len(incidents)} incident(s)"
# Without the caps this same traffic produces a 125-call incident spanning
# 244 minutes; with the old thinness test on top of that, 153 calls over
# 247 minutes in 3 incidents — the `f5190670` shape, reproduced.
biggest = max(len(inc["call_ids"]) for inc in incidents)
assert biggest <= settings.incident_max_calls, (
f"one incident holds {biggest} calls, past the "
f"{settings.incident_max_calls}-call cap"
)
longest = max(_live_span_minutes(inc) for inc in incidents)
assert longest <= settings.incident_max_duration_minutes, (
f"an incident took calls across {longest:.0f}min, past the "
f"{settings.incident_max_duration_minutes}min cap"
)
def test_each_dispatched_job_gets_its_own_incident():
"""
The 13 events are unrelated jobs — a pole strike, a burglar alarm, two
inspections, an altercation, a welfare check. On a dispatch backbone with no
unit or geocode tying them together, none of them may join another's
incident. Under the old thinness test every one of these was "thin" and
force-attached to whatever was most recent.
"""
traffic = _overnight_traffic()
incidents, placement = _simulate(traffic)
dispatch_offsets = {off for off, _, tags, _ in traffic if tags}
dispatch_incidents = {
placement[f"call-{i}"]
for i, (off, _, tags, _) in enumerate(sorted(traffic))
if tags and f"call-{i}" in placement
}
assert len(dispatch_incidents) == len(dispatch_offsets), (
f"{len(dispatch_offsets)} jobs landed in {len(dispatch_incidents)} incident(s)"
)
def test_a_long_run_of_pure_chatter_never_builds_an_incident():
"""
Acknowledgements alone carry no content, so they cannot open an incident and
— with nothing recent to reply to — must not accrete into one either.
"""
incidents, _ = _simulate([(m, [], [], None) for m in range(0, 240, 6)])
assert incidents == []
@@ -0,0 +1,201 @@
"""
server-26#<pending> — pattern B clearance: a unit accepting a NEW dispatch
("dispatch: are you able to clear and take a run at X / unit: 10-4") carries
no self-reported clearance language intelligence.py's cleared_units
extraction looks for (that only catches pattern A, "Unit 7, 10-8"). Before
this fix, reassignment=True only ever suppressed the unit from re-linking to
their prior incident (upload.py's corr_units=[] on reassignment) — nothing
ever released them from it, so it sat "active" until the 90-minute idle
sweep timed it out instead of being marked cleared by a real event.
`_release_reassigned_units` closes that gap: when a scene is a reassignment,
scan the OTHER active incidents for unit overlap and release the unit there,
using the same units_active/units_cleared merge (`_apply_unit_clearance`)
that explicit 10-8 extraction already used via `_update_incident`.
"""
from datetime import datetime, timedelta, timezone
from unittest.mock import patch
import pytest
from app.internal.incident_correlator import (
_apply_unit_clearance, _release_reassigned_units,
)
NOW = datetime(2026, 9, 20, 12, 0, 0, tzinfo=timezone.utc)
def _incident(incident_id="inc-1", units_active=None, units_cleared=None,
system_ids=("sys-1",), **overrides):
inc = {
"incident_id": incident_id,
"system_ids": list(system_ids),
"units_active": list(units_active or []),
"units_cleared": list(units_cleared or []),
"status": "active",
"updated_at": (NOW - timedelta(minutes=5)).isoformat(),
}
inc.update(overrides)
return inc
def _ctx(call_units, all_active, system_id="sys-1", now=NOW):
return {"call_units": call_units, "all_active": all_active, "system_id": system_id, "now": now}
# ---------------------------------------------------------------------------
# _apply_unit_clearance — pure merge logic
# ---------------------------------------------------------------------------
def test_clearance_moves_unit_from_active_to_cleared():
inc = _incident(units_active=["6-3"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["6-3"])
assert active == []
assert cleared == ["6-3"]
assert resolved is True
def test_clearance_leaves_other_active_units_alone():
inc = _incident(units_active=["6-3", "6-7"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["6-3"])
assert active == ["6-7"]
assert cleared == ["6-3"]
assert resolved is False # 6-7 still active
def test_clearing_a_unit_not_tracked_as_active_is_a_noop_for_active_list():
inc = _incident(units_active=["6-7"], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, ["ghost-unit"])
assert active == ["6-7"]
# A unit never active on this incident is not recorded as cleared here
# either (server-26#170 replay review): its 10-8 says nothing about this
# incident, and recording it let the all-clear gate pass on a stray clear.
assert cleared == []
assert resolved is False
def test_no_units_ever_tracked_does_not_auto_resolve():
# An incident that never had a unit signal at all — clearing nothing
# must not manufacture a resolve.
inc = _incident(units_active=[], units_cleared=[])
active, cleared, resolved = _apply_unit_clearance(inc, [])
assert resolved is False
# ---------------------------------------------------------------------------
# _release_reassigned_units — reassignment releases the unit from its
# PRIOR incident, scoped correctly, without touching that incident's calls
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_reassignment_clears_unit_from_prior_incident():
prior = _incident(incident_id="inc-prior", units_active=["6-3", "6-7"])
ctx = _ctx(call_units=["6-3"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id="inc-new")
assert len(doc_sets) == 1
collection, doc_id, data = doc_sets[0]
assert collection == "incidents" and doc_id == "inc-prior"
assert data["units_active"] == ["6-7"]
assert data["units_cleared"] == ["6-3"]
assert "status" not in data # 6-7 still active — not auto-resolved
@pytest.mark.asyncio
async def test_reassignment_auto_resolves_when_last_unit_clears():
prior = _incident(incident_id="inc-prior", units_active=["6-3"])
ctx = _ctx(call_units=["6-3"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
async def fake_doc_get(collection, doc_id):
return None # no parent — maybe_resolve_parent exits immediately
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
mock_fstore.doc_get = fake_doc_get
await _release_reassigned_units(ctx, exclude_incident_id=None)
collection, doc_id, data = doc_sets[0]
assert data["status"] == "resolved"
assert "resolved_at" in data
@pytest.mark.asyncio
async def test_reassignment_never_touches_the_calls_own_incident():
# The call's own decision (link/new) already handled its own incident —
# excluding it here prevents double-writing or self-clearing on it.
same = _incident(incident_id="inc-new", units_active=["6-3"])
ctx = _ctx(call_units=["6-3"], all_active=[same])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id="inc-new")
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_does_not_cross_systems():
other_system = _incident(incident_id="inc-other-sys", units_active=["6-3"], system_ids=("sys-2",))
ctx = _ctx(call_units=["6-3"], all_active=[other_system], system_id="sys-1")
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_with_no_call_units_is_a_noop():
prior = _incident(incident_id="inc-prior", units_active=["6-3"])
ctx = _ctx(call_units=[], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert doc_sets == []
@pytest.mark.asyncio
async def test_reassignment_matches_units_by_normalized_key():
# "5-David" vs "5David" — same unit, different transcription — must
# still match via the existing _normalize_unit key, not exact string eq.
prior = _incident(incident_id="inc-prior", units_active=["5-David"])
ctx = _ctx(call_units=["5 David"], all_active=[prior])
doc_sets = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
async def fake_doc_get(collection, doc_id):
return None # no parent — maybe_resolve_parent exits immediately
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
mock_fstore.doc_get = fake_doc_get
await _release_reassigned_units(ctx, exclude_incident_id=None)
assert len(doc_sets) == 1
assert doc_sets[0][2]["units_cleared"] == ["5-David"]
+66
View File
@@ -0,0 +1,66 @@
"""
End-to-end CORS wiring for the one browser-facing REST surface.
The frontend's Archive page calls GET /calls/search with Authorization +
Content-Type headers, which forces the browser to send a CORS preflight
first. Before #110 that OPTIONS got a bare 405 with no Access-Control-*
headers and the fetch failed with "TypeError: Failed to fetch". These
tests drive the real app through TestClient so a regression in the
middleware wiring (not just the helper) is caught.
TestClient is NOT used as a context manager on purpose: that would run the
lifespan (mqtt_handler.connect(), the sweeper loops, dynsec bootstrap),
none of which is needed here -- CORSMiddleware answers a preflight before
routing or dependencies run.
"""
from fastapi.testclient import TestClient
from app.config import settings
from app.main import app
client = TestClient(app)
ALLOWED_ORIGIN = "https://drb.cusano.net"
DISALLOWED_ORIGIN = "https://evil.example.com"
def test_default_allowed_origin_matches_the_deployed_frontend():
# The frontend is served on the bare domain (infra Caddyfile.j2), so the
# default must allow exactly that origin without any env override.
assert ALLOWED_ORIGIN in settings.cors_origins
def test_preflight_for_calls_search_is_allowed():
resp = client.options(
"/calls/search",
headers={
"Origin": ALLOWED_ORIGIN,
"Access-Control-Request-Method": "GET",
"Access-Control-Request-Headers": "authorization,content-type",
},
)
assert resp.status_code == 200
assert resp.headers.get("access-control-allow-origin") == ALLOWED_ORIGIN
allow_methods = resp.headers.get("access-control-allow-methods", "").upper()
assert "GET" in allow_methods
# Bearer auth, not cookies -- credentials must never be advertised.
assert "access-control-allow-credentials" not in resp.headers
def test_preflight_from_disallowed_origin_gets_no_allow_origin():
resp = client.options(
"/calls/search",
headers={
"Origin": DISALLOWED_ORIGIN,
"Access-Control-Request-Method": "GET",
},
)
assert resp.headers.get("access-control-allow-origin") is None
def test_simple_get_from_allowed_origin_is_annotated():
# Even a non-preflight GET must carry Access-Control-Allow-Origin or the
# browser hides the response body from the page.
resp = client.get("/health", headers={"Origin": ALLOWED_ORIGIN})
assert resp.status_code == 200
assert resp.headers.get("access-control-allow-origin") == ALLOWED_ORIGIN
+55
View File
@@ -0,0 +1,55 @@
"""
CORS must never end up as "any origin, WITH credentials".
Starlette does not reject `allow_origins=["*"]` combined with
`allow_credentials=True`. It reflects the caller's Origin back in
Access-Control-Allow-Origin and still sends
Access-Control-Allow-Credentials: true, so the effective policy is the
opposite of what a wildcard usually means. main.py never enables
credentials at all (auth is a Bearer header, not a cookie), which makes
that pair unrepresentable; these tests hold it to that.
The policy lives in a pure function so it can be exercised directly --
reloading app.main to vary settings drags every router back through import
and is not worth the fragility.
"""
from starlette.middleware.cors import CORSMiddleware
from app.config import settings
from app.main import app, cors_allows_credentials
def test_wildcard_alone_disables_credentials():
assert cors_allows_credentials(["*"]) is False
def test_wildcard_among_real_origins_still_disables_credentials():
# A list that merely CONTAINS "*" is as permissive as ["*"] alone --
# Starlette treats any wildcard entry as allow-all.
assert cors_allows_credentials(["https://app.example.com", "*"]) is False
def test_credentials_never_enabled_even_for_named_origins():
# Auth here is a Bearer header, not a cookie, so credentialed CORS is
# never needed. The predicate is hard-off regardless of the origin list.
assert cors_allows_credentials(["https://app.example.com"]) is False
assert cors_allows_credentials([]) is False
def test_the_app_actually_mounted_that_policy():
"""Guards the wiring, not just the helper: a future edit to main.py that
hardcodes allow_credentials=True again fails here."""
opts = next(
(mw.kwargs for mw in app.user_middleware if mw.cls is CORSMiddleware), None
)
assert opts is not None, "CORSMiddleware is not mounted at all"
assert opts["allow_credentials"] is False
assert opts["allow_credentials"] is cors_allows_credentials(settings.cors_origins)
def test_health_exposes_a_build_stamp():
"""CI compares this against the commit it just deployed; a deploy that
leaves the previous container running is otherwise invisible."""
from app.main import _GIT_SHA
assert isinstance(_GIT_SHA, str) and _GIT_SHA
+145
View File
@@ -0,0 +1,145 @@
"""
server-26#163 — the STT eval harness: word_error_rate() and the three routes
that back the /admin "STT Eval" tab.
Load-bearing property, checked directly: eval annotation must never touch
`transcript`/`transcript_corrected`, re-run extraction, or unlink incidents —
that's PATCH /{call_id}/transcript's job, a production correction with real
side effects. This is pure measurement and must stay pure.
"""
from unittest.mock import AsyncMock, patch
import pytest
from fastapi.testclient import TestClient
from app.internal.wer import word_error_rate
from app.main import app
from app.internal.auth import require_admin_token, require_service_or_firebase_token
from app.routers import calls
client = TestClient(app)
ADMIN = {"role": "admin", "org_id": "org-A"}
def _override(decoded: dict):
# calls.router carries its own router-level require_service_or_firebase_token
# (app/main.py) ON TOP OF each admin route's own require_admin_token — both
# have to be overridden or the router-level one 401s before the route's own
# dependency is ever evaluated.
app.dependency_overrides[require_admin_token] = lambda: decoded
app.dependency_overrides[require_service_or_firebase_token] = lambda: decoded
def teardown_function():
app.dependency_overrides.pop(require_admin_token, None)
app.dependency_overrides.pop(require_service_or_firebase_token, None)
# ── word_error_rate ─────────────────────────────────────────────────────────
def test_identical_transcripts_are_zero_wer():
assert word_error_rate("K on the 600, I'm on Jackson Avenue.",
"K on the 600, I'm on Jackson Avenue.") == 0.0
def test_case_and_punctuation_are_ignored():
assert word_error_rate("Home Street and Forest Ave!", "home street and forest ave") == 0.0
def test_one_substitution_out_of_three_words():
assert word_error_rate("the cat sat", "the cat sit") == pytest.approx(1 / 3)
def test_empty_reference_is_undefined_not_zero():
"""A call nobody transcribed must not score as a perfect match."""
assert word_error_rate("", "anything") is None
assert word_error_rate(None, "anything") is None
def test_empty_hypothesis_against_real_reference_is_total_loss():
assert word_error_rate("home street and forest ave", "") == 1.0
def test_insertion_counts_against_the_hypothesis():
# reference 3 words, hypothesis adds 2 extra -> 2 insertions / 3 ref words
assert word_error_rate("show me clear", "show me clear right now") == pytest.approx(2 / 3)
# ── GET /calls/eval-queue ───────────────────────────────────────────────────
def _call(call_id, transcript="a real transcript here", corrected=None, eval_transcript=None, org_id="org-A"):
return {
"call_id": call_id, "org_id": org_id, "started_at": "2026-09-21T00:00:00+00:00",
"transcript": transcript, "transcript_corrected": corrected, "eval_transcript": eval_transcript,
}
def test_eval_queue_skips_already_evaluated_and_transcript_less_calls():
rows = [
_call("c1", eval_transcript="already done"),
_call("c2", transcript=None),
_call("c3"),
]
_override(ADMIN)
with patch.object(calls.fstore, "collection_where", AsyncMock(return_value=rows)):
resp = client.get("/calls/eval-queue")
assert resp.status_code == 200
body = resp.json()
assert [c["call_id"] for c in body["calls"]] == ["c3"]
assert body["matched"] == 1
def test_eval_queue_requires_an_org_scope():
_override({"role": "admin"}) # platform admin, no org claim
resp = client.get("/calls/eval-queue")
assert resp.status_code == 403
# ── GET /calls/eval-stats ───────────────────────────────────────────────────
def test_eval_stats_averages_wer_across_evaluated_calls_only():
rows = [
_call("c1", transcript="the cat sat", corrected="the cat sat", eval_transcript="the cat sat"), # 0.0 / 0.0
_call("c2", transcript="the cat sit", corrected="the cat sat", eval_transcript="the cat sat"), # raw 1/3, corrected 0.0
_call("c3", eval_transcript=None), # excluded entirely
]
_override(ADMIN)
with patch.object(calls.fstore, "collection_list", AsyncMock(return_value=rows)):
resp = client.get("/calls/eval-stats")
assert resp.status_code == 200
body = resp.json()
assert body["eval_count"] == 2
assert body["raw_wer"] == pytest.approx((0.0 + 1 / 3) / 2, abs=1e-4)
assert body["corrected_wer"] == 0.0
def test_eval_stats_with_nothing_evaluated_yet_reports_none_not_zero():
_override(ADMIN)
with patch.object(calls.fstore, "collection_list", AsyncMock(return_value=[_call("c1")])):
resp = client.get("/calls/eval-stats")
body = resp.json()
assert body == {"eval_count": 0, "raw_wer": None, "corrected_wer": None}
# ── PUT /{call_id}/eval-transcript ──────────────────────────────────────────
def test_put_eval_transcript_writes_only_eval_fields():
_override(ADMIN)
existing = _call("c1", transcript="raw text", corrected="corrected text")
with patch.object(calls.fstore, "doc_get", AsyncMock(return_value=existing)), \
patch.object(calls.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.put("/calls/c1/eval-transcript", json={"text": "the verified ground truth"})
assert resp.status_code == 200
(collection, doc_id, doc), _ = mock_set.await_args
assert collection == "calls" and doc_id == "c1"
assert doc["eval_transcript"] == "the verified ground truth"
assert doc["eval_transcript_at"]
assert "transcript" not in doc and "transcript_corrected" not in doc
def test_put_eval_transcript_404s_on_missing_call():
_override(ADMIN)
with patch.object(calls.fstore, "doc_get", AsyncMock(return_value=None)):
resp = client.put("/calls/nope/eval-transcript", json={"text": "x"})
assert resp.status_code == 404
@@ -0,0 +1,185 @@
"""
server-26#159: a citywide/patched feed can be received far from its own
coverage area — "New York City - NYPD Citywide 2 Patch" was ~56km from the
receiving node, well past geocode_max_km (40km). Real, correctly-geocoded
addresses on that talkgroup were rejected by intelligence._geocode_location's
node-distance sanity check every time, so location_coords never populated for
the whole system: location_proximity correlation was permanently dead there,
and the same real event reported at two nearby addresses two minutes apart
became two separate incidents instead of one.
`trust_named_region` fixes this narrowly: the node-distance check is a proxy
for "is this plausible" that only makes sense when the node's own position is
the best guess we have at the area. It must not apply when the query already
names a different region on its own terms (operator-set area_context, or a
municipality parsed straight from the talkgroup's own name) — and it must
never touch the anchor path, whose own radius is always authoritative.
"""
from unittest.mock import patch
import pytest
import app.internal.intelligence as intel
from app.config import settings
def _maps_result(lat: float, lng: float, location_type: str = "ROOFTOP"):
payload = {
"status": "OK",
"results": [{
"geometry": {
"location": {"lat": lat, "lng": lng},
"location_type": location_type,
},
}],
}
class _Resp:
def raise_for_status(self): pass
def json(self): return payload
class _Client:
async def __aenter__(self): return self
async def __aexit__(self, *a): return False
async def get(self, *a, **k): return _Resp()
return patch("httpx.AsyncClient", lambda *a, **k: _Client())
@pytest.fixture(autouse=True)
def _api_key():
# intelligence.py imports settings locally per-function (`from app.config
# import settings`), which binds the same cached singleton — patching the
# module-level object here reaches it, but `intel.settings` itself does
# not exist as an attribute.
with patch.object(settings, "google_maps_api_key", "test-key"):
yield
# Node at (0, 0); result at (1, 0) is ~111km away — well past the 40km default.
NODE_LAT, NODE_LON = 0.0, 0.0
FAR_LAT, FAR_LNG = 1.0, 0.0
@pytest.mark.asyncio
async def test_named_region_geocode_accepted_beyond_node_distance():
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords == {"lat": FAR_LAT, "lng": FAR_LNG}
@pytest.mark.asyncio
async def test_local_geocode_still_rejected_beyond_node_distance_without_named_region():
"""Regression guard: a bare street name with no named region still uses
the node as its only plausibility check, exactly as before this fix."""
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"Main Street",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=False,
)
assert coords is None
@pytest.mark.asyncio
async def test_anchor_path_ignores_trust_named_region():
"""The anchor's own radius is always authoritative — trust_named_region
is only a statement about the node fallback, never a way to widen an
anchor that was itself deliberately sized to discriminate."""
anchor = {"lat": NODE_LAT, "lng": NODE_LON, "radius_km": 10.0}
with _maps_result(FAR_LAT, FAR_LNG):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
anchor=anchor, trust_named_region=True,
)
assert coords is None
@pytest.mark.asyncio
async def test_named_region_geocode_within_node_distance_is_unaffected():
"""A close result is accepted the same way regardless of the flag."""
near_lat, near_lng = 0.05, 0.0 # ~5.5km from the node
with _maps_result(near_lat, near_lng):
coords = await intel._geocode_location(
"Main Street, Ossining, New York",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords == {"lat": near_lat, "lng": near_lng}
@pytest.mark.asyncio
async def test_imprecise_result_still_rejected_regardless_of_trust():
"""trust_named_region relaxes the distance check only — the location_type
precision filter (server-26#37) still applies unconditionally."""
with _maps_result(FAR_LAT, FAR_LNG, location_type="APPROXIMATE"):
coords = await intel._geocode_location(
"1108 Jackson Avenue, New York City - NYPD Citywide 2 Patch",
node_lat=NODE_LAT, node_lon=NODE_LON,
trust_named_region=True,
)
assert coords is None
# ---------------------------------------------------------------------------
# _location_query_parts — pure query assembly, no HTTP involved
# ---------------------------------------------------------------------------
def test_operator_configured_area_wins_and_is_named_region():
parts, named = intel._location_query_parts(
"High Street", {"municipality": "Yorktown", "state": "New York"},
"Tac 1", node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Yorktown", "New York"]
assert named is True
def test_local_talkgroup_name_gets_node_state_but_not_node_county():
"""
"Ossining PD" is genuinely local to the node, so appending the node's own
state is correct. Its COUNTY is dropped even here — server-26#159's fix
applies uniformly once a municipality is derived, since there is no way
to tell "local" and "distant-but-node-adjacent" apart from the string
alone, and the county was never necessary for a bare municipality name
that already disambiguates via the state.
"""
parts, named = intel._location_query_parts(
"High Street", {}, "Ossining PD",
node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Ossining", "New York"]
assert named is True
def test_citywide_patched_feed_does_not_get_the_nodes_county_grafted_on():
"""
server-26#159's actual production case: the talkgroup names its own
(distant) region, so the node's county (Westchester, ~56km away) must not
be appended — it would make the query self-contradictory ("...New York
City..., Westchester, New York") and risks degrading the geocode result's
precision independently of the distance check this issue also fixes.
"""
parts, named = intel._location_query_parts(
"1108 Jackson Avenue", {}, "New York City - NYPD Citywide 2 Patch",
node_state="New York", node_county="Westchester",
)
assert "Westchester" not in parts
assert parts == ["1108 Jackson Avenue", "New York City - NYPD Citywide 2 Patch", "New York"]
assert named is True
def test_uninformative_talkgroup_name_falls_back_to_node_county_and_state():
"""A tactical channel or bare code gives _municipality_from_tg nothing —
the only remaining evidence really is where the node sits, so the
original node-county-and-state fallback is preserved for this case."""
parts, named = intel._location_query_parts(
"High Street", {}, "Tac 1",
node_state="New York", node_county="Westchester",
)
assert parts == ["High Street", "Westchester", "New York"]
assert named is False
+482
View File
@@ -0,0 +1,482 @@
"""
An incident must not lie about what it is or where it is — server-26#23 / #26.
Both defects come from the 2026-08-20 production dump
(CORRELATION_REVIEW_0820.md) and both are the same shape: a field of the
incident header re-derived from whichever call linked most recently.
* `location` (the label) and `location_coords` (the map pin) were two
independent last-write-wins fields. A call could move one and not the
other, so they drifted: 5 of 6 incidents in the dump were pinned somewhere
other than the place they were labelled. `b9b4f392` said "100 South
Mosher" and pinned `Westmed`.
* `location` was never validated, so "Flames from 49" put `location: "49"`
on `9d376ffe` and the summarizer wrote "reported at location 49".
* `title` was re-derived from every classified call, so `b9b4f392` — opened
on a suspect search at 80 Grasslands Road — was named after its third
call, and `f5190670` after the thirteenth of its thirteen events.
The rules under test:
1. label and pin are ONE value, written together on every path;
2. a pin is only ever kept next to the label it was geocoded from;
3. a string with no word in it is not a place;
4. the title names the founding event and only ever escalates.
"""
import pytest
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
from app.internal.incident_correlator import (
_build_context, _create_incident, _update_incident,
_resolve_location_pair, _verified_pin, clean_location, location_is_unit,
)
NOW = datetime(2026, 8, 20, 7, 25, 0, tzinfo=timezone.utc)
# TG 383 from the dump: "Ch 1 (Patched with 155.310)" — a shared dispatch
# backbone, which is where every one of these chains happened.
DISPATCH_TG = "Ch 1 (Patched with 155.310)"
GRASSLANDS = {"lat": 41.0891, "lng": -73.8010}
WESTMED = {"lat": 41.0348, "lng": -73.7629}
# ---------------------------------------------------------------------------
# Harness — incidents are stored with merge=True, so folding each write back
# into the dict is exactly what Firestore does between calls.
# ---------------------------------------------------------------------------
async def _create(**call) -> dict:
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _create_incident(
call.get("call_id", "call-0"), "org-1",
call.get("incident_type", "police"), 383, DISPATCH_TG, "sys-1",
call.get("tags", []), call.get("location"), call.get("coords"),
call.get("units", []), [], None,
call.get("severity", "routine"), call.get("now", NOW),
)
return dict(mock_fstore.doc_set.await_args.args[2])
async def _link(inc: dict, **call) -> dict:
"""Run one call through _update_incident, fold the write back, return it."""
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_set = AsyncMock()
await _update_incident(
inc, call.get("call_id", "call-n"), 383, "sys-1",
call.get("tags", []), call.get("location"), call.get("coords"),
call.get("units", []), [], None, call.get("now", NOW),
talkgroup_name=DISPATCH_TG,
incident_type=call.get("incident_type"),
call_severity=call.get("severity", "routine"),
)
updates = dict(mock_fstore.doc_set.await_args.args[2])
inc.update(updates)
return updates
def _assert_pin_matches_label(doc: dict):
"""The invariant: a pin exists only alongside the label it was geocoded from."""
if doc.get("location_coords") is not None:
assert doc.get("location"), "pin with no label"
assert doc.get("location_coords_source") == doc["location"], (
f"pin sourced from {doc.get('location_coords_source')!r} "
f"but incident is labelled {doc['location']!r}"
)
# ---------------------------------------------------------------------------
# server-26#23 — the label and the pin are one value
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_label_and_pin_stay_consistent_across_a_chain_of_calls():
"""
Replays `b9b4f392` exactly: a suspect search at 80 Grasslands Road, then an
EMS transport to Westmed, then a brand-new open-911 dispatch at 100 South
Mosher. Production ended up labelled "100 South Mosher" and pinned at
Westmed — the label from one call, the pin from another.
"""
inc = await _create(
tags=["suspect-search"], location="80 Grasslands Road", coords=GRASSLANDS,
severity="moderate", incident_type="police",
)
_assert_pin_matches_label(inc)
await _link( # 07:41 — "one female to Westmed"
inc, call_id="call-ems", tags=["ems-transport"], location="Westmed",
coords=WESTMED, incident_type="ems", now=NOW + timedelta(minutes=16),
)
_assert_pin_matches_label(inc)
await _link( # 07:55 — "open 911 line", a different job entirely
inc, call_id="call-911", tags=["open-911"], location="100 South Mosher",
coords=None, incident_type="police", now=NOW + timedelta(minutes=30),
)
_assert_pin_matches_label(inc)
assert inc["location"] == "80 Grasslands Road"
assert inc["location_coords"] == GRASSLANDS
# Every place anyone named is still recorded — that is what the map path
# is drawn from; it just isn't the incident's own location.
assert inc["location_mentions"] == [
"80 Grasslands Road", "Westmed", "100 South Mosher",
]
@pytest.mark.asyncio
async def test_a_later_call_never_moves_the_pin_without_the_label():
"""The direct mechanism: coords updating on their own."""
inc = await _create(tags=["suspect-search"], location="80 Grasslands Road",
coords=None, incident_type="police")
assert inc["location_coords"] is None
updates = await _link(inc, tags=["ems-transport"], location="Westmed",
coords=WESTMED, incident_type="ems")
assert updates["location"] == "80 Grasslands Road"
assert updates["location_coords"] is None
_assert_pin_matches_label(inc)
@pytest.mark.asyncio
async def test_a_pin_can_still_be_filled_in_for_the_same_place():
"""
The one permitted change. Geocoding is not deterministic in practice — it
needs the node's position, an API quota and a response — so the same
address can fail on one call and resolve on the next. Filling in a pin the
incident never had is not a move; matching is deliberately by exact label,
so it can never quietly re-point at a different street.
"""
inc = await _create(tags=["welfare-check"], location="55 Hyman Hills Road",
coords=None, incident_type="police")
assert inc["location_coords"] is None
await _link(inc, tags=["welfare-check"], location="55 Hyman Hills Road",
coords=GRASSLANDS, incident_type="police")
assert inc["location"] == "55 Hyman Hills Road"
assert inc["location_coords"] == GRASSLANDS
_assert_pin_matches_label(inc)
def test_a_pin_that_cannot_be_tied_to_the_label_is_not_shown():
"""
Every incident written before this change carries a pin with no record of
where it came from — and the dump says 5 of 6 of those are wrong. An
unverifiable pin is dropped, not displayed: a missing pin reads as missing
data, a wrong one reads as fact.
"""
legacy = {"location": "100 South Mosher", "location_coords": WESTMED}
assert _verified_pin(legacy) is None
resolved = _resolve_location_pair(legacy, None, None)
assert resolved["location"] == "100 South Mosher"
assert resolved["location_coords"] is None
assert resolved["location_coords_source"] is None
tagged = {"location": "Westmed", "location_coords": WESTMED,
"location_coords_source": "westmed"}
assert _verified_pin(tagged) == WESTMED
# ---------------------------------------------------------------------------
# server-26#23 — "49" is not a place
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("junk", [
"49", # 9d376ffe, from "Fire received. Flames from 49."
"10-24", # a ten-code
"5-5-2", # a unit designator
" ",
"",
None,
"1",
])
def test_bare_numbers_are_rejected_as_locations(junk):
assert clean_location(junk) is None
@pytest.mark.parametrize("place", [
"80 Grasslands Road",
"Westmed",
"Rt 9",
"226 East Main Street, apartment number 1",
])
def test_real_place_names_survive(place):
assert clean_location(place) == place
@pytest.mark.asyncio
async def test_a_bare_number_never_reaches_the_correlator():
"""
Rejected at the context boundary, so it is not a location in the fit tests,
the thin-call test, the LLM prompt or the incident — and its coordinates go
with it, because they were geocoded from that very string.
"""
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(return_value={})
mock_fstore.collection_list = AsyncMock(return_value=[])
ctx = await _build_context(
call_id="call-49", units=None, vehicles=None, cleared_units=None,
location_coords={"lat": 41.0, "lng": -73.8}, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="fire", location="49",
reassignment=False, create_if_new=True,
)
assert ctx["location"] is None
assert ctx["location_coords"] is None
@pytest.mark.asyncio
async def test_a_scene_with_no_location_does_not_inherit_the_call_docs_pin():
"""
server-26#87. One call can be split into several scenes, and only the
primary scene's geocode is written to the call doc. A non-primary scene
that passes no location of its own must not inherit that pin — doing so
fabricates location_proximity, the strongest accept signal, for a scene
that has none, and drives it into the primary scene's incident.
"""
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(
return_value={"location_coords": GRASSLANDS}
)
mock_fstore.collection_list = AsyncMock(return_value=[])
ctx = await _build_context(
call_id="call-scene-2", units=None, vehicles=None, cleared_units=None,
location_coords=None, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="police", location=None,
reassignment=False, create_if_new=True,
)
assert ctx["coords"] is None
assert ctx["is_thin_call"] is True
@pytest.mark.asyncio
async def test_a_scene_does_not_inherit_the_call_docs_embedding_or_severity():
"""
server-26#80 / #95. Same shape as the #87 coords leak above:
intelligence.py writes only the PRIMARY scene's embedding and severity to
calls/{id}. A non-primary scene being correlated must be judged on its own
embedding (or none) and its own severity — not the call doc's — or a scene
about a different event scores against the wrong incident on the embedding
path and can inherit a minor/moderate/major rung it never had, clearing the
creation gate on borrowed weight.
"""
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(
return_value={"embedding": [0.1] * 1536, "severity": "major"}
)
mock_fstore.collection_list = AsyncMock(return_value=[])
ctx = await _build_context(
call_id="call-scene-2", units=None, vehicles=None, cleared_units=None,
location_coords=None, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="police", location=None,
reassignment=False, create_if_new=True,
embedding=None, severity=None,
)
assert ctx["call_embedding"] is None
assert ctx["call_severity"] == "routine"
assert ctx["is_thin_call"] is True
@pytest.mark.asyncio
async def test_a_scene_is_judged_on_its_own_embedding_and_severity():
"""The other half of #80/#95: the scene's own values are what land in ctx."""
scene_vec = [0.9] * 1536
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(
return_value={"embedding": [0.1] * 1536, "severity": "routine"}
)
mock_fstore.collection_list = AsyncMock(return_value=[])
ctx = await _build_context(
call_id="call-scene-2", units=None, vehicles=None, cleared_units=None,
location_coords=None, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="police", location=None,
reassignment=False, create_if_new=True,
embedding=scene_vec, severity="major",
)
assert ctx["call_embedding"] == scene_vec
assert ctx["call_severity"] == "major"
@pytest.mark.asyncio
async def test_the_llm_tier_reads_the_scene_transcript_not_the_whole_call():
"""
server-26#102. intelligence.py writes only the primary scene's corrected
text to calls/{id}. _call_block (the LLM correlation prompt) must reason
over the SCENE being correlated, not a whole-call transcript that also
contains the other scenes. _build_context threads the scene's text in;
with no scene text it falls back to the call doc (sweep / single-scene).
"""
with patch("app.internal.incident_correlator.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(return_value={
"transcript": "scene one about a fire. scene two about a traffic stop.",
})
mock_fstore.collection_list = AsyncMock(return_value=[])
scene = await _build_context(
call_id="call-1", units=None, vehicles=None, cleared_units=None,
location_coords=None, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="police", location=None,
reassignment=False, create_if_new=True,
transcript="scene two about a traffic stop.",
)
fallback = await _build_context(
call_id="call-1", units=None, vehicles=None, cleared_units=None,
location_coords=None, reference_time=NOW,
system_id="sys-1", talkgroup_id=383, talkgroup_name=DISPATCH_TG,
tags=[], incident_type="police", location=None,
reassignment=False, create_if_new=True,
)
assert scene["scene_transcript"] == "scene two about a traffic stop."
assert fallback["scene_transcript"] == "scene one about a fire. scene two about a traffic stop."
@pytest.mark.asyncio
async def test_a_bare_number_never_becomes_an_incident_location_or_title():
inc = await _create(tags=["flames"], location="49", coords=None,
incident_type="fire")
assert inc["location"] is None
assert inc["location_coords"] is None
assert "49" not in inc["title"]
assert inc["location_mentions"] == []
# ---------------------------------------------------------------------------
# server-26#26 — the title names the founding event
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_an_unrelated_later_call_does_not_rename_the_incident():
"""`b9b4f392` again, from the title's side."""
inc = await _create(tags=["suspect-search"], location="80 Grasslands Road",
coords=GRASSLANDS, severity="moderate", incident_type="police")
assert inc["title"] == "Suspect Search at 80 Grasslands Road"
await _link(inc, tags=["ems-transport"], location="Westmed", coords=WESTMED,
incident_type="ems", severity="routine")
await _link(inc, tags=["open-911"], location="100 South Mosher",
incident_type="police", severity="moderate")
assert inc["title"] == "Suspect Search at 80 Grasslands Road"
assert inc["title_tag"] == "Suspect Search"
@pytest.mark.asyncio
async def test_routine_status_traffic_never_touches_the_title():
inc = await _create(tags=["welfare-check"], location="55 Hyman Hills Road",
incident_type="police")
updates = await _link(inc, tags=[], incident_type=None, units=["6-Adam"])
assert "title" not in updates
@pytest.mark.asyncio
async def test_a_worse_event_takes_the_title_over():
"""
The one case where the header must change: a check-condition that turns
into a structure fire is a structure fire. Monotonic like _max_severity —
a calmer later call can never take it back.
"""
inc = await _create(tags=["check-condition"], location="226 East Main Street",
severity="minor", incident_type="police")
assert inc["title"] == "Check Condition at 226 East Main Street"
await _link(inc, tags=["structure-fire"], incident_type="fire", severity="major")
assert inc["title"] == "Structure Fire at 226 East Main Street"
assert inc["severity"] == "major"
await _link(inc, tags=["ems-transport"], incident_type="ems", severity="routine")
assert inc["title"] == "Structure Fire at 226 East Main Street"
@pytest.mark.asyncio
async def test_a_placeholder_title_is_filled_in_not_overwritten():
"""
An incident that opened on a call with no content tags is named after its
type ("Police — <talkgroup>"). That is a placeholder, not an event name,
so the first classified call may name it — and only the first.
"""
inc = await _create(tags=[], location=None, incident_type="police")
assert inc["title"] == f"Police — {DISPATCH_TG}"
assert inc["title_tag"] is None
await _link(inc, tags=["vehicle-accident"], location="Airport Road",
incident_type="police", severity="minor")
assert inc["title"] == "Vehicle Accident at Airport Road"
await _link(inc, tags=["disabled-vehicle"], location="Yonkers Avenue",
incident_type="police", severity="minor")
assert inc["title"] == "Vehicle Accident at Airport Road"
@pytest.mark.asyncio
async def test_the_title_picks_up_an_address_learned_later():
"""
Same event, new information — not a rename. The founding call classified
the event but named no place; a later call on the same event does.
"""
inc = await _create(tags=["welfare-check"], location=None, incident_type="police")
assert inc["title"] == f"Welfare Check — {DISPATCH_TG}"
await _link(inc, tags=["welfare-check"], location="55 Hyman Hills Road",
incident_type="police")
assert inc["title"] == "Welfare Check at 55 Hyman Hills Road"
assert inc["location"] == "55 Hyman Hills Road"
@pytest.mark.asyncio
async def test_a_legacy_incidents_title_is_not_claimed_by_the_next_call():
"""
Incidents created before this change have no `title_tag`, so their founding
event is unrecoverable. Their existing title is treated as the founding
one rather than handed to whichever call links next.
"""
legacy = {
"incident_id": "b9b4f392",
"title": "Suspect Search at 80 Grasslands Road",
"location": "80 Grasslands Road",
"call_ids": ["call-0"],
"started_at": NOW.isoformat(),
"updated_at": NOW.isoformat(),
}
updates = await _link(legacy, tags=["open-911"], location="100 South Mosher",
incident_type="police", severity="routine")
assert "title" not in updates
assert updates["location"] == "80 Grasslands Road"
# ── server-26#52: a unit call-sign must never become a map pin ────────────────
#
# "Post 1-2" passed clean_location (it has a word in it), geocoded against the
# Ossining anchor and produced a confident pin in the right town for an event
# with no known location. It was in the same incident's `units` all along.
@pytest.mark.parametrize("location,units", [
("Post 1-2", ["1-2", "Lincoln", "Post 1-2"]), # the dump's actual incident
("post 1-2", ["Post 1-2"]), # case-blind
("Post 1-2.", ["Post 1-2"]), # punctuation-blind
("Engine 4", ["Engine 4", "Ladder 1"]),
])
def test_location_matching_a_unit_is_rejected(location, units):
assert location_is_unit(location, units) is True
@pytest.mark.parametrize("location,units", [
("Water Street", ["1-2", "Post 1-2"]), # a real place, same incident
("South High", []), # no units extracted
("Riverdale Station", ["Lincoln"]),
("", ["Post 1-2"]), # nothing to compare
(None, ["Post 1-2"]),
])
def test_real_places_survive_the_unit_check(location, units):
assert location_is_unit(location, units) is False
def test_unit_check_does_not_match_on_substrings():
"""`1-2` is a unit; "1-2 Main Street" is an address that contains it."""
assert location_is_unit("1-2 Main Street", ["1-2"]) is False
@@ -0,0 +1,52 @@
"""
server-26#81 — any signed-in viewer could trigger OpenAI summary spend.
``POST /incidents/{incident_id}/summarize`` was gated by
``require_service_or_firebase_token``, which accepts ANY authenticated
Firebase user (including role "viewer"), not just admins. Hitting the route
spends OpenAI credits via the background summarizer task. The call-side
equivalent (``PATCH /calls/{id}/transcript``) was already moved to
``require_admin_token``; the incident side was not moved with it.
Following the wiring-test convention in test_admin_feature_flags.py
(``test_features_routes_use_the_agent_dependency_and_others_do_not``): assert
against the route's actual dependant.dependencies rather than round-tripping
through TestClient, so this pins the credential wiring itself and would fail
immediately if someone reverts the dependency back to the weak one.
"""
from app.internal import auth
from app.routers import incidents
def _deps(path: str, method: str) -> set:
for r in incidents.router.routes:
if r.path == path and method in r.methods:
return {d.call for d in r.dependant.dependencies}
raise AssertionError(f"no route {method} {path}")
def test_summarize_incident_requires_admin_not_any_firebase_user():
deps = _deps("/incidents/{incident_id}/summarize", "POST")
assert auth.require_admin_token in deps
assert auth.require_service_or_firebase_token not in deps
def test_read_only_incident_routes_still_accept_any_signed_in_user():
"""Guards against an overcorrection: reads are not spend, they stay open
to any authenticated viewer."""
assert auth.require_service_or_firebase_token in _deps("/incidents", "GET")
assert auth.require_service_or_firebase_token in _deps("/incidents/{incident_id}", "GET")
def test_other_mutating_incident_routes_are_still_admin_only():
"""Unchanged by this fix, but pinned so a future edit can't quietly
loosen them while touching this file."""
for path, method in [
("/incidents/summarize", "POST"),
("/incidents", "POST"),
("/incidents/{incident_id}", "PUT"),
("/incidents/{incident_id}", "DELETE"),
("/incidents/{incident_id}/calls/{call_id}", "POST"),
("/incidents/{incident_id}/calls/{call_id}", "DELETE"),
]:
assert auth.require_admin_token in _deps(path, method), f"{method} {path}"
@@ -0,0 +1,54 @@
"""
server-26#<pending> — no per-system unit-ID format awareness existed anywhere
in the pipeline (vocabulary_learner's "known local terms" is a flat glossary,
not a structured format). Departments use incompatible unit ID conventions
(Yorktown: "5-David", sometimes spoken as bare "David"; County:
"SAM-1"/"airport-3"/"parks-4", a location word + number) and the extraction
prompt had no way to be told which one a given system uses. This pins the
prompt-block builder and the template wiring that carries it.
"""
from app.internal.intelligence import (
_PROMPT_TEMPLATE, _build_unit_format_block, _build_ten_codes_block,
_build_transcript_block,
)
def test_empty_hint_produces_no_block():
assert _build_unit_format_block(None) == ""
assert _build_unit_format_block("") == ""
def test_hint_is_labelled_and_fed_to_the_model_verbatim():
block = _build_unit_format_block(
"Yorktown: <district>-<phonetic name>, e.g. 5-David. Sometimes spoken as just the name alone."
)
assert "unit ID format" in block
assert "5-David" in block
def test_prompt_template_renders_with_all_blocks_including_empty_unit_format():
# Regression guard: a missing placeholder in .format() raises KeyError at
# request time, not import time — this is the cheapest way to catch that
# before it reaches a live call.
rendered = _PROMPT_TEMPLATE.format(
transcript_block=_build_transcript_block("1. Test.", None),
talkgroup_name="Test TG",
system_id="sys-1",
ten_codes_block=_build_ten_codes_block({}),
vocabulary_block="",
unit_format_block=_build_unit_format_block(""),
)
assert "Test TG" in rendered
assert "1. Test." in rendered
def test_prompt_template_renders_with_a_populated_unit_format_block():
rendered = _PROMPT_TEMPLATE.format(
transcript_block=_build_transcript_block("1. Test.", None),
talkgroup_name="Test TG",
system_id="sys-1",
ten_codes_block=_build_ten_codes_block({}),
vocabulary_block="",
unit_format_block=_build_unit_format_block("County: <location>-<number>, e.g. SAM-1, airport-3."),
)
assert "SAM-1" in rendered
+23 -10
View File
@@ -67,7 +67,9 @@ async def test_checkin_creates_new_node(handler):
)
mock_fstore.doc_set.assert_called_once()
_, _, doc, _ = mock_fstore.doc_set.call_args[0]
# doc_set(collection, doc_id, data, merge=False) — merge is passed as a
# kwarg in mqtt_handler.py, so only 3 positional args land in call_args[0].
_, _, doc = mock_fstore.doc_set.call_args[0]
assert doc["node_id"] == "new-node"
assert doc["name"] == "Pi Zero W"
assert doc["status"] == "unconfigured"
@@ -84,7 +86,7 @@ async def test_checkin_new_node_defaults_lat_lon(handler):
await handler._handle_checkin("new-node", {})
_, _, doc, _ = mock_fstore.doc_set.call_args[0]
_, _, doc = mock_fstore.doc_set.call_args[0]
assert doc["lat"] == 0.0
assert doc["lon"] == 0.0
@@ -200,13 +202,17 @@ async def test_call_start_creates_call_doc(handler):
}
with patch("app.internal.mqtt_handler.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(return_value=node)
# _on_call_start looks the node up via doc_get_cached (cached read,
# added to cut Firestore read volume — see doc_get_cached in
# app/internal/firestore.py), not the uncached doc_get.
mock_fstore.doc_get_cached = AsyncMock(return_value=node)
mock_fstore.doc_set = AsyncMock()
await handler._on_call_start("node-01", payload)
mock_fstore.doc_set.assert_called_once()
_, _, doc, _ = mock_fstore.doc_set.call_args[0]
# doc_set(collection, doc_id, data, merge=False) — merge is a kwarg here too.
_, _, doc = mock_fstore.doc_set.call_args[0]
assert doc["call_id"] == "call-abc123"
assert doc["node_id"] == "node-01"
assert doc["system_id"] == "sys-001"
@@ -233,12 +239,12 @@ async def test_call_start_uses_now_when_started_at_missing(handler):
payload = {"call_id": "call-xyz", "tgid": 99}
with patch("app.internal.mqtt_handler.fstore") as mock_fstore:
mock_fstore.doc_get = AsyncMock(return_value=node)
mock_fstore.doc_get_cached = AsyncMock(return_value=node)
mock_fstore.doc_set = AsyncMock()
await handler._on_call_start("node-01", payload)
_, _, doc, _ = mock_fstore.doc_set.call_args[0]
_, _, doc = mock_fstore.doc_set.call_args[0]
assert doc["started_at"] is not None
@@ -250,11 +256,17 @@ async def test_call_end_updates_status_and_times(handler):
}
with patch("app.internal.mqtt_handler.fstore") as mock_fstore:
mock_fstore.doc_update = AsyncMock()
# _on_call_end writes via doc_set(merge=True) now, not doc_update — see
# the "Fix Upload 404 warning" commit: doc_update raised "No document
# to update" when call_end raced ahead of call_start, so it was
# switched to a merging doc_set. It also reads the node via the
# cached doc_get_cached to stamp org_id.
mock_fstore.doc_get_cached = AsyncMock(return_value=None)
mock_fstore.doc_set = AsyncMock()
await handler._on_call_end("node-01", payload)
updates = mock_fstore.doc_update.call_args[0][2]
updates = mock_fstore.doc_set.call_args[0][2]
assert updates["status"] == "ended"
assert updates["ended_at"] is not None
@@ -268,11 +280,12 @@ async def test_call_end_sets_audio_url_when_present(handler):
}
with patch("app.internal.mqtt_handler.fstore") as mock_fstore:
mock_fstore.doc_update = AsyncMock()
mock_fstore.doc_get_cached = AsyncMock(return_value=None)
mock_fstore.doc_set = AsyncMock()
await handler._on_call_end("node-01", payload)
updates = mock_fstore.doc_update.call_args[0][2]
updates = mock_fstore.doc_set.call_args[0][2]
assert updates["audio_url"] == "https://storage.example.com/call.mp3"
+20 -4
View File
@@ -35,8 +35,16 @@ def _node_naive(node_id, status, age_seconds):
async def test_stale_online_node_marked_offline():
nodes = [_node("node-01", "online", age_seconds=120)]
# A stale node also triggers app.routers.tokens.release_token(node_id) —
# added by the PulseAudio/Discord-token work (commit 2a690ec). It's
# imported inline inside _sweep, so it must be patched at its source
# module rather than relying on the global asyncio.to_thread patch above,
# which is scoped to the node-query call and would otherwise feed
# release_token's own internal to_thread call the wrong shape of data
# (raw node dicts instead of Firestore doc snapshots with .id).
with patch("asyncio.to_thread", new=AsyncMock(return_value=nodes)), \
patch("app.internal.node_sweeper.fstore") as mock_fstore:
patch("app.internal.node_sweeper.fstore") as mock_fstore, \
patch("app.routers.tokens.release_token", new=AsyncMock()):
mock_fstore.doc_update = AsyncMock()
await _sweep()
@@ -50,7 +58,8 @@ async def test_stale_recording_node_marked_offline():
nodes = [_node("node-02", "recording", age_seconds=200)]
with patch("asyncio.to_thread", new=AsyncMock(return_value=nodes)), \
patch("app.internal.node_sweeper.fstore") as mock_fstore:
patch("app.internal.node_sweeper.fstore") as mock_fstore, \
patch("app.routers.tokens.release_token", new=AsyncMock()):
mock_fstore.doc_update = AsyncMock()
await _sweep()
@@ -106,7 +115,8 @@ async def test_tz_naive_last_seen_is_handled():
nodes = [_node_naive("node-06", "online", age_seconds=120)]
with patch("asyncio.to_thread", new=AsyncMock(return_value=nodes)), \
patch("app.internal.node_sweeper.fstore") as mock_fstore:
patch("app.internal.node_sweeper.fstore") as mock_fstore, \
patch("app.routers.tokens.release_token", new=AsyncMock()):
mock_fstore.doc_update = AsyncMock()
await _sweep()
@@ -141,10 +151,16 @@ async def test_only_stale_nodes_updated_in_batch():
]
with patch("asyncio.to_thread", new=AsyncMock(return_value=nodes)), \
patch("app.internal.node_sweeper.fstore") as mock_fstore:
patch("app.internal.node_sweeper.fstore") as mock_fstore, \
patch("app.routers.tokens.release_token", new=AsyncMock()) as mock_release:
mock_fstore.doc_update = AsyncMock()
await _sweep()
assert mock_fstore.doc_update.call_count == 2
updated_ids = {call.args[1] for call in mock_fstore.doc_update.call_args_list}
assert updated_ids == {"node-08", "node-11"}
# Both newly-offline nodes should have their Discord token freed.
assert mock_release.call_count == 2
released_ids = {call.args[0] for call in mock_release.call_args_list}
assert released_ids == {"node-08", "node-11"}
@@ -0,0 +1,197 @@
"""
server-26#96 — every scene of a multi-scene call writes corr_debug onto the
SAME call doc via incident_correlator._apply_and_log, last-scene-wins. The
fix additionally nests each scene's corr_debug/transcript/incident_id under
scenes.<scene_index> on the call doc, keyed so Firestore's
`set(merge=True)` (a recursive merge of nested map fields — this is the
behaviour these tests assume and pin) lands each scene in its own map entry
instead of colliding.
Firestore itself isn't available in this sandbox (see tests/conftest.py), so
`_fake_doc_set` below implements that documented recursive-merge semantics by
hand and is used as the fstore stand-in — these tests both exercise
_apply_and_log's write shape AND pin the merge behaviour it depends on.
"""
import pytest
from unittest.mock import patch
from app.internal import incident_correlator
def _merge(dst: dict, src: dict) -> None:
"""Firestore DocumentReference.set(data, merge=True) semantics: nested
map fields are merged recursively by key, not replaced wholesale."""
for k, v in src.items():
if isinstance(v, dict) and isinstance(dst.get(k), dict):
_merge(dst[k], v)
else:
dst[k] = v
@pytest.mark.asyncio
async def test_multiscene_call_lands_each_scene_distinctly_and_flat_fields_last_write_wins():
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision0 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
}
ctx0 = {"call_id": "call-1", "scene_index": 0, "scene_transcript": "scene zero text"}
decision1 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "slow", "corr_consensus": "tiebreak"},
}
ctx1 = {"call_id": "call-1", "scene_index": 1, "scene_transcript": "scene one text"}
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision0, ctx0)
await incident_correlator._apply_and_log(decision1, ctx1)
doc = docs[("calls", "call-1")]
# Flat top-level fields: unchanged behaviour, last scene's write wins —
# the safe backward-compatible default for any reader that doesn't yet
# know about `scenes`.
assert doc["corr_path"] == "slow"
assert doc["corr_consensus"] == "tiebreak"
# New `scenes` map: both scenes present, distinct, uncorrupted by the
# second write.
assert set(doc["scenes"].keys()) == {"0", "1"}
assert doc["scenes"]["0"]["corr_debug"]["corr_path"] == "new"
assert doc["scenes"]["0"]["corr_debug"]["corr_consensus"] == "agreed"
assert doc["scenes"]["0"]["transcript"] == "scene zero text"
assert doc["scenes"]["1"]["corr_debug"]["corr_path"] == "slow"
assert doc["scenes"]["1"]["corr_debug"]["corr_consensus"] == "tiebreak"
assert doc["scenes"]["1"]["transcript"] == "scene one text"
@pytest.mark.asyncio
async def test_scene_entry_records_which_incident_it_resolved_to():
"""summarizer.py (#114) needs this to pick the right scene per incident."""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
with patch.object(incident_correlator, "fstore") as mock_fstore, \
patch.object(incident_correlator, "_apply_decision", return_value="inc-42"):
mock_fstore.doc_set = fake_doc_set
decision = {
"action": "new", "matched_incident": None, "incident_type": "fire",
"corr_debug": {"corr_path": "new"},
}
ctx = {"call_id": "call-2", "scene_index": 0, "scene_transcript": "structure fire"}
incident_id = await incident_correlator._apply_and_log(decision, ctx)
assert incident_id == "inc-42"
assert docs[("calls", "call-2")]["scenes"]["0"]["incident_id"] == "inc-42"
@pytest.mark.asyncio
async def test_single_scene_call_still_gets_a_scenes_map_equivalent_to_flat_fields():
"""scene_index defaults to 0 for every caller with no scene concept, so a
plain single-scene call is one entry in `scenes` — equivalent to reading
the flat fields, not a behaviour change for that population."""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "fast/thin", "corr_consensus": "rules_only"},
}
ctx = {"call_id": "call-3", "scene_transcript": "10-4"} # no scene_index key at all
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision, ctx)
doc = docs[("calls", "call-3")]
assert doc["corr_path"] == "fast/thin"
assert doc["scenes"] == {
"0": {
"transcript": "10-4",
"incident_id": None,
"corr_debug": {"corr_path": "fast/thin", "corr_consensus": "rules_only"},
"incident_type": None,
"severity": None,
}
}
@pytest.mark.asyncio
async def test_scene_entry_captures_its_own_incident_type_not_a_sibling_scenes():
"""
server-26#139: _call_is_substanceless's "type" veto reads ctx["incident_type"]
at decision time, but that value was never persisted per-scene — only the
last-scene-wins flat field, which #138's dump analysis couldn't
distinguish from cross-scene contamination. Pins _apply_and_log's write
side: each scene's own scenes.<n> entry carries its own incident_type/
severity, distinct from any other scene on the same call. Does NOT cover
whether the ctx handed to _call_is_substanceless is the same object that
reaches here — that linkage is pinned by test_consensus_gate.py and
test_incident_identity.py, not this file.
"""
docs: dict[tuple, dict] = {}
async def fake_doc_set(collection, doc_id, data, merge=True):
docs.setdefault((collection, doc_id), {})
_merge(docs[(collection, doc_id)], data)
decision0 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "tiebreak", "corr_gate_veto": "type"},
}
ctx0 = {
"call_id": "call-5", "scene_index": 0, "scene_transcript": "10-4, clear",
"incident_type": "traffic-stop", "call_severity": "routine",
}
decision1 = {
"action": "orphan", "matched_incident": None, "incident_type": None,
"corr_debug": {"corr_path": "new", "corr_consensus": "agreed"},
}
ctx1 = {
"call_id": "call-5", "scene_index": 1, "scene_transcript": "roll call",
"incident_type": None, "call_severity": "moderate",
}
with patch.object(incident_correlator, "fstore") as mock_fstore:
mock_fstore.doc_set = fake_doc_set
await incident_correlator._apply_and_log(decision0, ctx0)
await incident_correlator._apply_and_log(decision1, ctx1)
doc = docs[("calls", "call-5")]
scenes = doc["scenes"]
assert scenes["0"]["incident_type"] == "traffic-stop"
assert scenes["0"]["severity"] == "routine"
assert scenes["1"]["incident_type"] is None
assert scenes["1"]["severity"] == "moderate"
# _apply_and_log only ever flat-merges corr_debug's own keys (:1460) — a
# future corr_debug["incident_type"] would silently clobber
# intelligence.py's flat field, so this is asserted, not just commented.
assert "incident_type" not in doc
@pytest.mark.asyncio
async def test_empty_corr_debug_writes_nothing_same_as_before():
"""Preserve the pre-#96 short-circuit: no corr_debug means no write at
all, flat or nested."""
with patch.object(incident_correlator, "fstore") as mock_fstore, \
patch.object(incident_correlator, "_apply_decision", return_value=None):
mock_fstore.doc_set = None # would raise TypeError if ever called
decision = {"action": "orphan", "matched_incident": None, "incident_type": None, "corr_debug": {}}
ctx = {"call_id": "call-4", "scene_index": 0, "scene_transcript": "x"}
result = await incident_correlator._apply_and_log(decision, ctx)
assert result is None
+192
View File
@@ -0,0 +1,192 @@
"""
Unit tests for Maps-based place verification (server-26#37).
The property that matters most is the one that looks like a no-op: WITHOUT AN
ANCHOR, NOTHING HAPPENS. A system whose area is too wide to discriminate stores
no anchor, and verification must then skip entirely rather than accept whatever
geocodes. A check that passes everything is worse than no check, because it
reads as verification in the logs and in the data.
After that: a candidate may only rewrite a transcript if it actually sounds like
what was heard. Places Text Search will return the nearest plausible business
for any garbage string, so the API answering at all is not evidence.
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import place_verifier as pv
ANCHOR_AREA = {
"municipality": "Ossining",
"county": "Westchester",
"state": "New York",
"local_knowledge": [{"term": "Snowden Avenue", "meaning": "residential street"}],
"center": {"lat": 41.16, "lng": -73.86},
"radius_km": 6.0,
"resolved_from": "ossining|westchester|new york",
}
SEGS = [{"start": 0.0, "end": 1.0, "text": "Shout out to Optum."},
{"start": 1.0, "end": 2.0, "text": "Copy that."}]
@pytest.fixture(autouse=True)
def _enabled():
with patch.object(pv.settings, "place_verification_enabled", True), \
patch.object(pv.settings, "google_maps_api_key", "test-key"):
yield
def _geocode(result):
return patch.object(pv, "_geocode_in_anchor", AsyncMock(return_value=result))
def _places(result):
return patch.object(pv, "_places_soundalike", AsyncMock(return_value=result))
# -- Phonetics -----------------------------------------------------------------
@pytest.mark.parametrize("heard, real", [
("Snowden Avenue", "Snowdon Ave"),
("5 acre", "5-baker"),
("why vac", "YVAC"),
("Croton Ave", "Croton Avenue"),
])
def test_real_mishearings_score_above_the_threshold(heard, real):
assert pv.sounds_like(heard, real) >= pv.settings.place_soundalike_min_ratio
@pytest.mark.parametrize("heard, unrelated", [
("Optum", "Ossining"),
("Cool Parts", "Croton Point"),
])
def test_unrelated_names_score_below_it(heard, unrelated):
assert pv.sounds_like(heard, unrelated) < pv.settings.place_soundalike_min_ratio
# -- The skip path -------------------------------------------------------------
@pytest.mark.asyncio
async def test_no_anchor_means_skip_not_accept():
"""A statewide system stores no anchor. Nothing may be checked or rewritten."""
with patch.object(pv, "_geocode_in_anchor") as geo:
out = await pv.verify("c1", "text here", SEGS, ["Optum"], {"state": "Colorado"}, {})
assert out == (None, None)
geo.assert_not_called()
@pytest.mark.asyncio
async def test_no_locations_means_no_requests():
with patch.object(pv, "_geocode_in_anchor") as geo:
assert await pv.verify("c1", "t", None, [], ANCHOR_AREA, {}) == (None, None)
geo.assert_not_called()
@pytest.mark.asyncio
async def test_disabled_by_setting():
with patch.object(pv.settings, "place_verification_enabled", False), \
patch.object(pv, "_geocode_in_anchor") as geo:
assert await pv.verify("c1", "t", None, ["Optum"], ANCHOR_AREA, {}) == (None, None)
geo.assert_not_called()
# -- The accept path -----------------------------------------------------------
@pytest.mark.asyncio
async def test_a_place_that_resolves_inside_the_anchor_is_left_alone():
with _geocode({"lat": 41.16, "lng": -73.86}), _places(None) as places:
out = await pv.verify("c1", "Units to Snowden Avenue.", None,
["Snowden Avenue"], ANCHOR_AREA, {})
assert out == (None, None)
places.assert_not_called() # a hit must not cost a second request
@pytest.mark.asyncio
async def test_the_query_carries_the_full_place():
seen = {}
async def capture(query, anchor):
seen["query"] = query
return {"lat": 41.16, "lng": -73.86}
with patch.object(pv, "_geocode_in_anchor", capture):
await pv.verify("c1", "t", None, ["High Street"], ANCHOR_AREA, {})
assert seen["query"] == "High Street, Ossining, Westchester, New York"
# -- The correction path -------------------------------------------------------
@pytest.mark.asyncio
async def test_known_term_is_preferred_and_costs_nothing():
"""
A sound-alike the operator already entered is both free and more trustworthy
than anything Maps guesses, so it must be tried before any request goes out.
"""
with _geocode(None), _places(None) as places, \
patch.object(pv.area_context, "add_pending", AsyncMock()) as add:
text, segs = await pv.verify(
"c1", "Units to Snowdon Ave.", None, ["Snowdon Ave"], ANCHOR_AREA, {}
)
assert text == "Units to Snowden Avenue."
places.assert_not_called()
add.assert_not_called() # already known — nothing to propose
@pytest.mark.asyncio
async def test_a_maps_soundalike_is_applied_and_proposed_to_the_talkgroup():
candidate = {"term": "Croton Point", "meaning": "Croton Point Ave, Croton NY", "score": 0.8}
with _geocode(None), _places(candidate), \
patch.object(pv.area_context, "add_pending", AsyncMock(return_value=1)) as add:
text, segs = await pv.verify(
"c1", "Respond to Cool Parts.", None, ["Cool Parts"], ANCHOR_AREA, {},
system_id="sys-1", talkgroup_id=9048,
)
assert text == "Respond to Croton Point."
args = add.await_args.args
assert args[0] == "sys-1" and args[1] == 9048
assert args[2][0]["term"] == "Croton Point"
assert args[2][0]["source_call_ids"] == ["c1"]
@pytest.mark.asyncio
async def test_nothing_plausible_leaves_the_transcript_untouched():
"""
An invented name with no real counterpart nearby stays as it is. Guessing
would put a fabricated location into the incident record, which is the
outcome this whole pass exists to avoid.
"""
with _geocode(None), _places(None):
assert await pv.verify("c1", "Shout out to Optum.", SEGS,
["Optum"], ANCHOR_AREA, {}) == (None, None)
@pytest.mark.asyncio
async def test_segments_are_corrected_alongside_the_joined_text():
"""Extraction reads numbered segments, so a joined-only fix reaches nothing."""
with _geocode(None), _places(None), \
patch.object(pv.area_context, "add_pending", AsyncMock()):
text, segs = await pv.verify(
"c1", "Shout out to Snowdon Ave. Copy that.",
[{"start": 0.0, "end": 1.0, "text": "Shout out to Snowdon Ave."},
{"start": 1.0, "end": 2.0, "text": "Copy that."}],
["Snowdon Ave"], ANCHOR_AREA, {},
)
assert segs is not None
assert segs[0]["text"] == "Shout out to Snowden Avenue."
assert segs[0]["start"] == 0.0, "timing survives untouched"
assert segs[1]["text"] == "Copy that."
@pytest.mark.asyncio
async def test_a_geocoder_failure_never_breaks_the_transcript():
with patch.object(pv, "_geocode_in_anchor", AsyncMock(side_effect=RuntimeError("boom"))):
assert await pv.verify("c1", "t here", None, ["Optum"], ANCHOR_AREA, {}) == (None, None)
@pytest.mark.asyncio
async def test_only_a_bounded_number_of_nouns_is_checked():
with patch.object(pv.settings, "place_verify_max_per_call", 2), \
patch.object(pv, "_geocode_in_anchor", AsyncMock(return_value={"lat": 41.16, "lng": -73.86})) as geo:
await pv.verify("c1", "t", None, ["a", "b", "c", "d"], ANCHOR_AREA, {})
assert geo.await_count == 2
@@ -0,0 +1,97 @@
"""
server-26#131 — the re-correlation sweep's orphan filter checked incident_id/
incident_ids/corr_path but had no way to tell "never processed" apart from
"real-time pipeline (routers/upload.py _run_intelligence_pipeline) is still
mid-flight". Racing the sweep against an in-flight real-time correlation could
land the same call on two different incidents — the exact duplicate-link bug
#131 found in 3 live dumps (~2% of linked calls). This pins the fix: a call
whose intelligence_started_at marker is recent is held back from the sweep
regardless of how orphaned it otherwise looks.
"""
from datetime import datetime, timezone, timedelta
from unittest.mock import patch
import pytest
from app.internal import recorrelation_sweep
def _iso(dt: datetime) -> str:
return dt.isoformat()
class TestPipelineLikelyStillRunning:
def test_recent_marker_is_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=1))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is True
def test_old_marker_is_not_still_running(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": _iso(now - timedelta(minutes=30))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_marker_exactly_at_the_threshold_is_not_held_back(self):
# age_minutes < MIN_MINUTES_SINCE_PIPELINE_START (strict), so exactly
# at the threshold is old enough to release — pins the boundary so it
# can't drift to <= by accident and silently double the hold time.
now = datetime.now(timezone.utc)
threshold = recorrelation_sweep.MIN_MINUTES_SINCE_PIPELINE_START
call = {"intelligence_started_at": _iso(now - timedelta(minutes=threshold))}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
def test_no_marker_at_all_is_not_held_back(self):
"""A pre-#131 call doc, or the marker write itself failed — absence
isn't evidence of an in-flight pipeline, so the sweep must still be
able to pick these up (that's its whole job)."""
now = datetime.now(timezone.utc)
assert recorrelation_sweep._pipeline_likely_still_running({}, now) is False
def test_unparseable_marker_is_not_held_back(self):
now = datetime.now(timezone.utc)
call = {"intelligence_started_at": "not-a-timestamp"}
assert recorrelation_sweep._pipeline_likely_still_running(call, now) is False
@pytest.mark.asyncio
async def test_sweep_pass_skips_a_call_whose_pipeline_just_started():
"""Integration-shaped: a call that looks orphaned by every OTHER filter
(no incident_ids, no corr_path, no skip_reason, under the attempt budget)
but has a fresh intelligence_started_at must not reach correlate_call —
that's the race #131 found."""
now = datetime.now(timezone.utc)
racing_call = {
"call_id": "call-racing",
"started_at": _iso(now - timedelta(minutes=2)),
"ended_at": _iso(now - timedelta(minutes=1)),
"intelligence_started_at": _iso(now - timedelta(seconds=30)),
}
genuinely_orphaned_call = {
"call_id": "call-genuine-orphan",
"started_at": _iso(now - timedelta(minutes=20)),
"ended_at": _iso(now - timedelta(minutes=19)),
"intelligence_started_at": _iso(now - timedelta(minutes=19)),
}
async def fake_collection_where(collection, clauses):
assert collection == "calls"
return [racing_call, genuinely_orphaned_call]
correlate_calls: list[str] = []
async def fake_correlate_call(**kwargs):
correlate_calls.append(kwargs["call_id"])
return None # no match — exercises the "not linked" branch too
doc_sets: list[tuple] = []
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_sets.append((collection, doc_id, data))
with patch.object(recorrelation_sweep, "fstore") as mock_fstore, \
patch("app.internal.incident_correlator.correlate_call", fake_correlate_call):
mock_fstore.collection_where = fake_collection_where
mock_fstore.doc_set = fake_doc_set
await recorrelation_sweep._run_sweep_pass()
assert correlate_calls == ["call-genuine-orphan"]
+414
View File
@@ -0,0 +1,414 @@
"""
Replay (app/internal/replay.py): re-running the pipeline over past calls in a
sandbox. The properties that matter, in order: a replay never writes a live
call or incident; it runs the live correlation code with the clock pinned to
each call's own time; and a call seeded ahead of its turn is invisible to the
orphan sweep until it is processed.
"""
import asyncio
import copy
from datetime import datetime, timedelta, timezone
from unittest.mock import patch
import pytest
from app.internal import clock, replay
from app.internal import firestore as fstore
from app.internal.feature_flags import force_flags, resolve_flags, unforce_flags
# ---------------------------------------------------------------------------
# An in-memory Firestore that honours the sandbox redirect, so the real
# correlator can run against it.
# ---------------------------------------------------------------------------
def _merge(dst: dict, src: dict) -> dict:
for k, v in src.items():
if isinstance(v, dict) and isinstance(dst.get(k), dict):
_merge(dst[k], v)
else:
dst[k] = copy.deepcopy(v)
return dst
def _cmp(a, op, b) -> bool:
if a is None:
return False
if isinstance(a, str) and isinstance(b, datetime):
a = datetime.fromisoformat(a)
return {"==": a == b, ">=": a >= b, "<=": a <= b, ">": a > b, "<": a < b}[op]
class FakeStore:
def __init__(self):
self.data: dict[str, dict[str, dict]] = {}
def coll(self, name: str) -> dict:
return self.data.setdefault(fstore._path(name), {})
async def doc_set(self, collection, doc_id, data, merge=True):
c = self.coll(collection)
if merge and doc_id in c:
_merge(c[doc_id], data)
else:
c[doc_id] = copy.deepcopy(data)
async def doc_update(self, collection, doc_id, data):
await self.doc_set(collection, doc_id, data)
async def doc_get(self, collection, doc_id):
d = self.coll(collection).get(doc_id)
return copy.deepcopy(d) if d is not None else None
async def doc_get_cached(self, collection, doc_id, ttl=300.0):
return await self.doc_get(collection, doc_id)
async def doc_delete(self, collection, doc_id):
self.coll(collection).pop(doc_id, None)
async def collection_list(self, collection, **filters):
return [copy.deepcopy(d) for d in self.coll(collection).values()
if all(d.get(k) == v for k, v in filters.items())]
async def collection_where(self, collection, conditions, order_by=None,
limit_to=None, start_after=None):
rows = [copy.deepcopy(d) for d in self.coll(collection).values()
if all(_cmp(d.get(f), op, v) for f, op, v in conditions)]
for field, direction in reversed(order_by or []):
rows.sort(key=lambda d: d.get(field), reverse=direction == "DESCENDING")
return rows[:limit_to] if limit_to else rows
@pytest.fixture
def store():
s = FakeStore()
names = ("doc_set", "doc_update", "doc_get", "doc_get_cached", "doc_delete",
"collection_list", "collection_where")
patches = [patch.object(fstore, n, getattr(s, n)) for n in names]
for p in patches:
p.start()
replay._active_run_id = None
replay._active_task = None
yield s
for p in patches:
p.stop()
# ---------------------------------------------------------------------------
# The context-scoped pieces
# ---------------------------------------------------------------------------
def test_sandbox_redirects_only_calls_and_incidents():
assert fstore._path("calls") == "calls"
tok = fstore.enter_sandbox("replay_runs/r1")
try:
assert fstore._path("calls") == "replay_runs/r1/calls"
assert fstore._path("incidents") == "replay_runs/r1/incidents"
assert fstore._path("systems") == "systems"
assert fstore._path("config") == "config"
finally:
fstore.exit_sandbox(tok)
assert fstore._path("incidents") == "incidents"
@pytest.mark.asyncio
async def test_sandbox_and_clock_do_not_leak_into_a_concurrent_task():
"""A replay runs beside live uploads in one event loop. The live task
must see the real collections and the real clock."""
pinned = datetime(2026, 9, 21, 12, 0, tzinfo=timezone.utc)
seen = {}
replay_entered = asyncio.Event()
live_checked = asyncio.Event()
async def replay_task():
fstore.enter_sandbox("replay_runs/r1")
clock.pin(pinned)
replay_entered.set()
await live_checked.wait()
seen["replay"] = (fstore._path("calls"), clock.now())
async def live_task():
await replay_entered.wait()
seen["live"] = (fstore._path("calls"), clock.now())
live_checked.set()
await asyncio.gather(replay_task(), live_task())
assert seen["replay"] == ("replay_runs/r1/calls", pinned)
assert seen["live"][0] == "calls"
assert seen["live"][1] != pinned
@pytest.mark.asyncio
async def test_forced_flags_override_global_switches():
tok = force_flags({"correlation_enabled": True, "stt_enabled": False})
try:
flags, flag = await resolve_flags("sys-1")
assert flag("correlation_enabled") is True
assert flag("stt_enabled") is False
assert flag("summaries_enabled") is False
finally:
unforce_flags(tok)
def test_sandbox_seed_strips_live_answers():
call = {
"call_id": "c1", "org_id": "o", "talkgroup_id": 5, "srcaddr": 123,
"status": "ended", "transcript": "engine 5 responding", "segments": [{"t": 1}],
"incident_ids": ["live-inc"], "incident_id": "live-inc", "units": ["E5"],
"corr_path": "fast/thin", "scenes": {"0": {}}, "skip_reason": None,
"chatter_classifier_verdict": "x", "eval_transcript": "y", "embedding": [0.1],
}
seed = replay._sandbox_seed(call, "transcripts")
assert seed["transcript"] == "engine 5 responding"
assert seed["srcaddr"] == 123
assert seed["status"] == "replay_pending"
for gone in ("incident_ids", "incident_id", "units", "corr_path", "scenes",
"chatter_classifier_verdict", "eval_transcript", "embedding"):
assert gone not in seed
assert "transcript" not in replay._sandbox_seed(call, "audio")
def test_compute_metrics_separates_timeout_from_real_clears():
incidents = [
{"call_ids": ["a"], "status": "resolved", "resolved_via": "idle_timeout"},
{"call_ids": ["b", "c"], "status": "resolved", "resolved_via": "units_cleared",
"units_cleared": ["E5"]},
{"call_ids": ["d", "e", "f"], "status": "active"},
]
calls = [
{"call_id": "a", "incident_ids": ["1"], "scenes": {"0": {"corr_debug": {
"corr_path": "new", "corr_consensus": "rules_only"}}}},
{"call_id": "z", "corr_path": "unlinked"},
]
m = replay.compute_metrics(incidents, calls)
assert m["incidents"] == 3
assert m["single_call_incidents"] == 1
assert m["resolved_via"] == {"idle_timeout": 1, "units_cleared": 1, "still_active": 1}
assert m["incidents_with_units_cleared"] == 1
assert m["calls_orphaned"] == 1
assert m["corr_path"] == {"new": 1, "unlinked": 1}
assert m["llm_decisions"] == 0
# ---------------------------------------------------------------------------
# A whole run, through the real correlator
# ---------------------------------------------------------------------------
T0 = datetime(2026, 9, 21, 14, 0, tzinfo=timezone.utc)
def _live_call(i: int, minute: int, transcript: str) -> dict:
return {
"call_id": f"call-{i}", "org_id": "org-1", "node_id": "node-1",
"system_id": "sys-1", "talkgroup_id": 100, "talkgroup_name": "Police Dispatch",
"started_at": T0 + timedelta(minutes=minute),
"ended_at": T0 + timedelta(minutes=minute, seconds=20),
"duration_s": 20, "status": "ended",
"transcript": transcript,
"incident_ids": ["LIVE-INCIDENT"], "corr_path": "fast/thin",
}
def _scene(transcript: str, units: list[str]) -> dict:
return {
"tags": ["mva"], "incident_type": "accident", "location": "Main Street",
"location_coords": None, "units": units, "vehicles": [], "cleared_units": [],
"reassignment": False, "embedding": None, "severity": "moderate",
"transcript": transcript, "resolved": False,
}
@pytest.mark.asyncio
async def test_run_writes_only_to_its_sandbox_and_pins_the_clock(store):
live = {
"call-1": _live_call(1, 0, "Car 12, MVA Main Street"),
"call-2": _live_call(2, 1, "Car 12 on scene Main Street"),
"call-3": _live_call(3, 300, "Car 40, alarm Oak Avenue"),
}
store.data["calls"] = copy.deepcopy(live)
store.data["incidents"] = {"LIVE-INCIDENT": {"incident_id": "LIVE-INCIDENT", "org_id": "org-1",
"status": "active", "call_ids": ["call-1"]}}
live_before = copy.deepcopy(store.data)
extracted = []
async def fake_extract(call_id, transcript, talkgroup_name, **kw):
extracted.append(call_id)
# Prefetch seeds calls ahead of the clock; they must not look "ended" yet.
return [_scene(transcript, ["Car 12"] if "12" in transcript else ["Car 40"])]
with patch("app.internal.intelligence.extract_scenes", fake_extract):
calls, truncated = await replay.select_calls(
"org-1", T0 - timedelta(hours=1), T0 + timedelta(hours=6))
assert [c["call_id"] for c in calls] == ["call-1", "call-2", "call-3"]
assert not truncated
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1),
date_to=T0 + timedelta(hours=6), mode="transcripts", system_ids=None,
source_run_id=None, label="t", actor="test",
)
await replay._active_task
# Live collections are exactly as they were.
assert store.data["calls"] == live_before["calls"]
assert store.data["incidents"] == live_before["incidents"]
run = next(iter(store.data["replay_runs"].values()))
assert run["status"] == "done", run["errors"]
root = f"replay_runs/{run['run_id']}"
sb_calls = store.data[f"{root}/calls"]
sb_incidents = store.data[f"{root}/incidents"]
assert sorted(extracted) == ["call-1", "call-2", "call-3"]
assert all(c["status"] == "ended" for c in sb_calls.values())
assert "LIVE-INCIDENT" not in sb_incidents
# Incident timestamps come from the replayed calls, not the wall clock.
for inc in sb_incidents.values():
started = datetime.fromisoformat(inc["started_at"])
assert T0 <= started <= T0 + timedelta(hours=6)
# The two Car 12 calls are one job; the Car 40 call five hours later is another.
groups = sorted(sorted(i["call_ids"]) for i in sb_incidents.values())
assert groups == [["call-1", "call-2"], ["call-3"]]
# Each aged out on the replayed clock the way it would have live — its
# severity's quiet timer after its last activity, not "now".
assert run["metrics"]["resolved_via"] == {"idle_timeout": 2}
first = next(i for i in sb_incidents.values() if "call-1" in i["call_ids"])
idle = datetime.fromisoformat(first["resolved_at"]) - datetime.fromisoformat(first["updated_at"])
from app.internal.summarizer import _auto_resolve_minutes
limit = timedelta(minutes=_auto_resolve_minutes(first))
assert limit < idle <= limit + timedelta(minutes=5)
assert run["metrics"]["calls"] == 3
assert set(store.data[f"{root}/scenes"]) == {"call-1", "call-2", "call-3"}
@pytest.mark.asyncio
async def test_reuse_mode_correlates_without_extracting(store):
store.data["calls"] = {"call-1": _live_call(1, 0, "Car 12, MVA Main Street")}
async def fake_extract(call_id, transcript, talkgroup_name, **kw):
# What the real extract_scenes also does: write call-level fields.
await fstore.doc_set("calls", call_id, {"units": ["Car 12"], "tags": ["mva"]})
return [_scene(transcript, ["Car 12"])]
with patch("app.internal.intelligence.extract_scenes", fake_extract):
first = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
await replay._active_task
async def must_not_extract(*a, **kw):
raise AssertionError("reuse mode re-ran extraction")
with patch("app.internal.intelligence.extract_scenes", must_not_extract):
second = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="reuse", system_ids=None, source_run_id=first["run_id"], label="", actor="t")
await replay._active_task
run = store.data["replay_runs"][second["run_id"]]
assert run["status"] == "done", run["errors"]
assert run["progress"]["errors"] == 0
assert run["metrics"]["calls_linked"] == 1
# Extraction's call-level output came across too — the orphan sweep reads
# units/tags/location off the call doc, not off the scenes.
sb_call = store.data[f"replay_runs/{second['run_id']}/calls"]["call-1"]
assert sb_call["units"] == ["Car 12"]
assert sb_call["tags"] == ["mva"]
@pytest.mark.asyncio
async def test_one_run_at_a_time(store):
store.data["calls"] = {"call-1": _live_call(1, 0, "x")}
gate = asyncio.Event()
async def slow_extract(*a, **kw):
await gate.wait()
return []
with patch("app.internal.intelligence.extract_scenes", slow_extract):
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
with pytest.raises(replay.ReplayBusy):
await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
gate.set()
await replay._active_task
def test_stored_input_rebuilds_from_corrector_segments():
"""Live extraction overwrites transcript_corrected with scene 0's text;
the corrector's own output survives in segments_corrected."""
call = {
"transcript": "raw whisper",
"transcript_corrected": "scene zero only",
"segments": [{"text": "raw a"}, {"text": "raw b"}],
"segments_corrected": [{"text": "fixed a"}, {"text": "fixed b"}],
}
text, segs = replay._stored_input(call)
assert text == "fixed a fixed b"
assert segs == call["segments_corrected"]
assert replay._stored_input({"transcript": "raw", "segments": []}) == ("raw", [])
assert replay._stored_input({"transcript": "hum", "transcript_not_speech": True}) == (None, [])
@pytest.mark.asyncio
async def test_replay_never_touches_live_ai_health_or_review_queue():
from app.internal import ai_health, area_context
before = ai_health.snapshot()
tok = fstore.enter_sandbox("replay_runs/r1")
try:
with patch.object(ai_health, "_post_webhook") as hook, \
patch.object(fstore, "doc_get") as get:
for _ in range(10):
await ai_health.report_degraded("correlation_cheap", "gemini", "m", "429", "wait")
await ai_health.report_healthy("transcription")
assert await area_context.add_pending("sys-1", 5, [{"term": "x"}]) == 0
hook.assert_not_called()
get.assert_not_called()
finally:
fstore.exit_sandbox(tok)
assert ai_health.snapshot() == before
@pytest.mark.asyncio
async def test_run_aborts_when_an_ai_account_is_dead(store):
"""An unfunded OpenAI account made the first smoke run a sandbox of 290
orphans that looked like a result. A permanently failing tier now stops
the run and names the cause."""
store.data["calls"] = {
f"call-{i}": _live_call(i, i, "Car 12 responding to an MVA on Main Street") for i in range(1, 30)
}
def broke(*a, **kw):
raise RuntimeError("Error code: 429 - You exceeded your current quota (insufficient_quota)")
with patch("app.internal.intelligence._sync_extract", broke), \
patch("app.internal.intelligence.classify_chatter", return_value=(False, None)):
run = await replay.start_run(
org_id="org-1", date_from=T0 - timedelta(hours=1), date_to=T0 + timedelta(hours=1),
mode="transcripts", system_ids=None, source_run_id=None, label="", actor="t")
await replay._active_task
run = store.data["replay_runs"][run["run_id"]]
assert run["status"] == "failed"
assert any("out of credit" in e for e in run["errors"])
assert run["progress"]["done"] < 29
@pytest.mark.asyncio
async def test_live_extraction_failure_reports_to_ai_health():
from app.internal import ai_health, intelligence
def broke(*a, **kw):
raise RuntimeError("insufficient_quota")
with patch.object(intelligence, "_sync_extract", broke), \
patch.object(ai_health, "report_degraded") as degraded, \
patch.object(fstore, "doc_set"), patch.object(fstore, "doc_get_cached", return_value=None):
scenes = await intelligence.extract_scenes("c1", "Car 12 responding to an MVA on Main Street")
assert scenes == []
assert degraded.call_args.args[0] == "extraction"
assert degraded.call_args.kwargs["permanent"] is True
@@ -0,0 +1,89 @@
"""
server-26#96/#114 review (PR #132): `PATCH /calls/{id}/transcript` clears
stale intelligence fields before re-extraction runs, but `doc_set(...,
merge=True)` can only add/overwrite keys in a nested map, never remove one.
A call corrected from 3 scenes down to 1 would keep `scenes.1`/`scenes.2`
with pre-correction transcripts and incident_ids forever -- corrupting the
per-scene tally #96 exists to make trustworthy, and re-feeding stale text
into #114's summarizer fix if a stale scene's incident_id still names a real
incident. The fix deletes the field with `fstore.DELETE_FIELD` instead of
merging over it with an empty map (which is a no-op).
"""
from unittest.mock import AsyncMock, patch
import pytest
from fastapi import BackgroundTasks
from app.internal import firestore as fstore
from app.routers.calls import TranscriptUpdate, patch_transcript
@pytest.mark.asyncio
async def test_transcript_correction_deletes_the_scenes_field_not_merges_over_it():
call = {
"call_id": "call-1",
"system_id": "sys-1",
"node_id": "node-1",
"transcript": "old raw text",
# Simulates a prior 3-scene call, per #96's schema.
"scenes": {
"0": {"transcript": "scene zero", "incident_id": "inc-a", "corr_debug": {}},
"1": {"transcript": "scene one", "incident_id": "inc-b", "corr_debug": {}},
},
}
doc_set_calls: list[tuple] = []
doc_update_calls: list[tuple] = []
async def fake_doc_get(collection, doc_id):
if collection == "calls" and doc_id == "call-1":
return call
return None
async def fake_doc_set(collection, doc_id, data, merge=True):
doc_set_calls.append((collection, doc_id, data))
async def fake_doc_update(collection, doc_id, data):
doc_update_calls.append((collection, doc_id, data))
fake_flags = (None, lambda name: name == "correlation_enabled")
with patch("app.routers.calls.fstore.doc_get", new=fake_doc_get), \
patch("app.routers.calls.fstore.doc_set", new=fake_doc_set), \
patch("app.routers.calls.fstore.doc_update", new=fake_doc_update), \
patch("app.internal.feature_flags.resolve_flags", new=AsyncMock(return_value=fake_flags)):
result = await patch_transcript(
call_id="call-1",
body=TranscriptUpdate(transcript="corrected text"),
background_tasks=BackgroundTasks(),
_={},
)
assert result == {"ok": True, "call_id": "call-1"}
# The stale scenes map must be DELETED, not merged over with {} (a no-op
# under Firestore's set(merge=True) semantics) and not left untouched by
# a doc_set call that never mentions it.
scenes_deletions = [
(coll, doc_id, data) for (coll, doc_id, data) in doc_update_calls
if coll == "calls" and doc_id == "call-1" and "scenes" in data
]
assert len(scenes_deletions) == 1, (
f"expected exactly one doc_update clearing 'scenes', got {doc_update_calls}"
)
assert scenes_deletions[0][2]["scenes"] is fstore.DELETE_FIELD
# And no doc_set call should paper over the same field with an empty map
# instead -- that would silently do nothing and leave stale scenes intact.
for (coll, doc_id, data) in doc_set_calls:
if coll == "calls" and doc_id == "call-1":
assert "scenes" not in data, (
"a doc_set (merge=True) write must never carry 'scenes' -- "
"merging {} over an existing map is a no-op, not a delete"
)
def test_delete_field_is_the_real_firestore_sentinel():
"""Catches an import-path typo turning this into a silent no-op sentinel."""
from firebase_admin import firestore as fs
assert fstore.DELETE_FIELD is fs.DELETE_FIELD
@@ -0,0 +1,40 @@
"""
server-26#102 — a scene is correlated on its OWN transcript, not the whole call.
_scene_transcript_text slices the segments a scene owns. It must never return
"" (an empty slice would let incident_correlator._build_context fall back to
the call doc's whole-call transcript, re-opening the leak in exactly the case
— bad indices — where it matters).
"""
from app.internal.intelligence import _scene_transcript_text
SEGS = [
{"text": "structure fire, 12 Main"},
{"text": "engine 4 responding"},
{"text": "traffic stop, plate ABC"},
{"text": "one occupant"},
]
WHOLE = "structure fire, 12 Main engine 4 responding traffic stop, plate ABC one occupant"
def test_scene_owns_a_subset_of_segments():
assert _scene_transcript_text(WHOLE, SEGS, [0, 1], None) == "structure fire, 12 Main engine 4 responding"
assert _scene_transcript_text(WHOLE, SEGS, [2, 3], None) == "traffic stop, plate ABC one occupant"
def test_corrected_text_wins_when_present():
assert _scene_transcript_text(WHOLE, SEGS, [0], "cleaned up text") == "cleaned up text"
def test_no_segment_indices_falls_back_to_whole_call():
# single-segment calls are never numbered by _build_transcript_block → null indices
assert _scene_transcript_text(WHOLE, SEGS, None, None) == WHOLE
assert _scene_transcript_text(WHOLE, None, [0, 1], None) == WHOLE
def test_out_of_range_or_nonint_indices_fall_back_never_empty():
assert _scene_transcript_text(WHOLE, SEGS, [9, 10], None) == WHOLE # all out of range
assert _scene_transcript_text(WHOLE, SEGS, ["1", "2"], None) == WHOLE # 1-based strings, rejected
assert _scene_transcript_text(WHOLE, SEGS, [-1], None) == WHOLE # negative
# partial validity: keep what's in range
assert _scene_transcript_text(WHOLE, SEGS, [3, 99], None) == "one occupant"
@@ -0,0 +1,124 @@
"""
server-26#114 — the incident summarizer used to read doc["transcript"] (the
WHOLE call, raw) for every linked call, so a multi-scene call contributed
text from scenes it wasn't part of into an incident's summary, and
transcript_corrected was never consulted at all.
Fix: _scene_text_for_incident reads the server-26#96 `scenes` map to find the
scene(s) that actually resolved into a given incident_id, and falls back to
transcript_corrected-or-transcript for a call doc with no `scenes` field
(predates #96).
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import summarizer
from app.internal.summarizer import _scene_text_for_incident
# ---------------------------------------------------------------------------
# _scene_text_for_incident — pure function, no Firestore
# ---------------------------------------------------------------------------
def test_picks_the_scene_that_linked_to_this_incident():
doc = {
"transcript": "whole raw transcript blend",
"transcript_corrected": "whole corrected transcript blend",
"scenes": {
"0": {"transcript": "scene zero text", "incident_id": "inc-A", "corr_debug": {}},
"1": {"transcript": "scene one text", "incident_id": "inc-B", "corr_debug": {}},
},
}
assert _scene_text_for_incident(doc, "inc-A") == "scene zero text"
assert _scene_text_for_incident(doc, "inc-B") == "scene one text"
def test_joins_multiple_scenes_linked_to_the_same_incident_in_scene_order():
doc = {
"scenes": {
"1": {"transcript": "second", "incident_id": "inc-A"},
"0": {"transcript": "first", "incident_id": "inc-A"},
},
}
assert _scene_text_for_incident(doc, "inc-A") == "first\nsecond"
def test_old_schema_doc_falls_back_to_transcript_corrected_over_transcript():
doc = {"transcript": "raw", "transcript_corrected": "corrected"}
assert _scene_text_for_incident(doc, "inc-A") == "corrected"
def test_old_schema_doc_with_only_raw_transcript_still_returns_it():
doc = {"transcript": "raw only"}
assert _scene_text_for_incident(doc, "inc-A") == "raw only"
def test_scenes_present_but_none_match_falls_back_defensively():
"""Should not happen for a call_id genuinely in this incident's call_ids,
but silently dropping the call's contribution would be worse than a
whole-call fallback."""
doc = {
"transcript": "raw",
"transcript_corrected": "corrected",
"scenes": {"0": {"transcript": "x", "incident_id": "inc-OTHER"}},
}
assert _scene_text_for_incident(doc, "inc-A") == "corrected"
# ---------------------------------------------------------------------------
# _summarize_incident — end to end with fstore/Gemini mocked
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_summarize_incident_uses_scene_specific_text_for_a_multiscene_call():
"""
call-1 is a 2-scene call: scene 0 linked into inc-OTHER, scene 1 linked
into inc-1 (the incident being summarized). Only scene 1's text may reach
the model.
"""
call_1 = {
"call_id": "call-1",
"transcript": "scene zero text scene one text", # the old, wrong, whole-call blend
"scenes": {
"0": {"transcript": "scene zero text", "incident_id": "inc-OTHER"},
"1": {"transcript": "scene one text", "incident_id": "inc-1"},
},
}
async def fake_doc_get(collection, doc_id):
assert collection == "calls"
return call_1 if doc_id == "call-1" else None
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value={"summaries_enabled": True})), \
patch.object(summarizer, "fstore") as fs, \
patch.object(summarizer, "_sync_summarize", return_value="a summary") as sync:
fs.doc_get = AsyncMock(side_effect=fake_doc_get)
fs.doc_set = AsyncMock()
await summarizer._summarize_incident({"incident_id": "inc-1", "call_ids": ["call-1"]})
sync.assert_called_once()
_inc_arg, transcripts_arg = sync.call_args.args
assert transcripts_arg == ["scene one text"]
assert "scene zero text scene one text" not in transcripts_arg
@pytest.mark.asyncio
async def test_summarize_incident_falls_back_for_old_schema_call_doc():
"""A call doc with no `scenes` field at all — summarizer must still work,
using transcript_corrected over raw transcript."""
call_1 = {"call_id": "call-1", "transcript": "raw", "transcript_corrected": "corrected"}
async def fake_doc_get(collection, doc_id):
return call_1 if doc_id == "call-1" else None
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value={"summaries_enabled": True})), \
patch.object(summarizer, "fstore") as fs, \
patch.object(summarizer, "_sync_summarize", return_value="a summary") as sync:
fs.doc_get = AsyncMock(side_effect=fake_doc_get)
fs.doc_set = AsyncMock()
await summarizer._summarize_incident({"incident_id": "inc-1", "call_ids": ["call-1"]})
_inc_arg, transcripts_arg = sync.call_args.args
assert transcripts_arg == ["corrected"]
+111
View File
@@ -0,0 +1,111 @@
"""
Unit tests for talkgroup name resolution.
From the 2026-08-23 correlation dump: 84 of 100 incidents were titled
"Ems — TGID 9048" rather than "Ems — Ossining Police Dispatch", because
/upload took `talkgroup_name` from a multipart form field the node only fills
when OP25 already had the name — and never fell back to the system config the
way mqtt_handler's call_start path did. server-26#34.
resolve() is the single implementation both paths now share. These tests pin
its preference order, since the whole bug was one caller skipping a step.
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import talkgroups
SYSTEM = {
"config": {
"talkgroups": [
{"id": 9048, "name": "Ossining - Police Dispatch"},
{"id": 9600, "name": "MTA PD Districts 6/7/11 - Police Dispatch"},
{"id": 9563, "name": ""}, # present but unnamed
{"id": "9211", "name": "Ardsley"}, # id stored as a string
]
}
}
def _system(doc=SYSTEM):
return patch.object(talkgroups.fstore, "doc_get_cached", AsyncMock(return_value=doc))
@pytest.mark.asyncio
async def test_hint_wins_over_everything():
"""OP25 knew the name — no Firestore read at all."""
with patch.object(talkgroups.fstore, "doc_get_cached", AsyncMock()) as m:
got = await talkgroups.resolve("sys-1", 9048, hint="Whatever OP25 Said")
assert got == "Whatever OP25 Said"
m.assert_not_awaited()
@pytest.mark.asyncio
async def test_call_doc_used_before_system_config():
"""The call document already carries the name written at call_start."""
with patch.object(talkgroups.fstore, "doc_get_cached", AsyncMock()) as m:
got = await talkgroups.resolve(
"sys-1", 9048, hint=None, call_doc={"talkgroup_name": "From Call Doc"}
)
assert got == "From Call Doc"
m.assert_not_awaited()
@pytest.mark.asyncio
async def test_falls_back_to_system_config():
"""The case that was broken: nothing upstream knew the name, C2 did."""
with _system():
got = await talkgroups.resolve("sys-1", 9048, hint=None, call_doc={})
assert got == "Ossining - Police Dispatch"
@pytest.mark.asyncio
async def test_empty_call_doc_name_does_not_block_the_lookup():
"""A falsy talkgroup_name on the call doc must not short-circuit."""
with _system():
got = await talkgroups.resolve(
"sys-1", 9600, hint=None, call_doc={"talkgroup_name": ""}
)
assert got == "MTA PD Districts 6/7/11 - Police Dispatch"
@pytest.mark.asyncio
async def test_string_talkgroup_ids_in_config_still_match():
with _system():
assert await talkgroups.resolve("sys-1", 9211) == "Ardsley"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"system_id, tgid",
[(None, 9048), ("sys-1", None), (None, None)],
)
async def test_missing_inputs_return_none_without_reading(system_id, tgid):
with patch.object(talkgroups.fstore, "doc_get_cached", AsyncMock()) as m:
assert await talkgroups.resolve(system_id, tgid) is None
m.assert_not_awaited()
@pytest.mark.asyncio
async def test_unknown_talkgroup_returns_none_so_caller_keeps_tgid_fallback():
with _system():
assert await talkgroups.resolve("sys-1", 1234) is None
@pytest.mark.asyncio
async def test_named_entry_with_empty_string_returns_none():
"""An entry that exists but has no name is not a name."""
with _system():
assert await talkgroups.resolve("sys-1", 9563) is None
@pytest.mark.asyncio
async def test_missing_system_document_returns_none():
with _system(doc=None):
assert await talkgroups.resolve("sys-1", 9048) is None
@pytest.mark.asyncio
async def test_unparseable_talkgroup_id_returns_none():
with _system():
assert await talkgroups.resolve("sys-1", "not-a-number") is None
+94
View File
@@ -0,0 +1,94 @@
"""
node-26#9 — second-SDR ADS-B telemetry ingestion.
Two things matter here: the endpoint requires node identity (a service/admin
token has no node_id to attribute the sighting to, so it must 400 rather than
silently write an orphan doc), and org_id gets stamped from the node's own
Firestore doc so firestore.rules' docInMyOrg() can gate the frontend's read —
the same defensive-stamp pattern upload.py already uses for `calls`.
"""
from unittest.mock import AsyncMock, patch
from fastapi.testclient import TestClient
from app.main import app
from app.internal.auth import require_node_service_or_firebase_token
from app.routers import telemetry
client = TestClient(app)
def _override(decoded: dict):
app.dependency_overrides[require_node_service_or_firebase_token] = lambda: decoded
def teardown_function():
app.dependency_overrides.pop(require_node_service_or_firebase_token, None)
def test_service_token_without_node_id_is_rejected():
_override({"service": True})
resp = client.post("/telemetry/adsb", json={"aircraft": []})
assert resp.status_code == 400
def test_node_upload_upserts_and_stamps_org_id():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value={"org_id": "org-A"})), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/adsb", json={
"aircraft": [{"icao": "A1B2C3", "callsign": "UAL123", "lat": 41.1, "lon": -73.8}],
})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 1}
mock_set.assert_awaited_once()
(collection, doc_id, doc), kwargs = mock_set.await_args
assert collection == "aircraft"
assert doc_id == "A1B2C3"
assert doc["node_id"] == "node-1"
assert doc["org_id"] == "org-A"
assert kwargs.get("merge") is True
def test_node_upload_skips_entries_missing_icao():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value=None)), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/adsb", json={"aircraft": [{"icao": ""}]})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 0}
mock_set.assert_not_awaited()
def test_ais_service_token_without_node_id_is_rejected():
_override({"service": True})
resp = client.post("/telemetry/ais", json={"vessels": []})
assert resp.status_code == 400
def test_ais_node_upload_upserts_and_stamps_org_id():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value={"org_id": "org-A"})), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/ais", json={
"vessels": [{"mmsi": "123456789", "name": "MV TEST", "lat": 41.0, "lon": -73.9}],
})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 1}
mock_set.assert_awaited_once()
(collection, doc_id, doc), kwargs = mock_set.await_args
assert collection == "vessels"
assert doc_id == "123456789"
assert doc["node_id"] == "node-1"
assert doc["org_id"] == "org-A"
assert kwargs.get("merge") is True
def test_ais_node_upload_skips_entries_missing_mmsi():
_override({"node": True, "node_id": "node-1"})
with patch.object(telemetry.fstore, "doc_get_cached", AsyncMock(return_value=None)), \
patch.object(telemetry.fstore, "doc_set", AsyncMock()) as mock_set:
resp = client.post("/telemetry/ais", json={"vessels": [{"mmsi": ""}]})
assert resp.status_code == 200
assert resp.json() == {"ok": True, "count": 0}
mock_set.assert_not_awaited()
@@ -0,0 +1,281 @@
"""
Unit tests for the transcript correction pass (server-26#36).
Two properties carry real risk and are pinned hardest here:
* SCOPE RESOLUTION — talkgroup reference data must rank ABOVE system data.
A system spanning several counties can have a talkgroup covering one
municipality, and burying that municipality's streets under a county-wide
list is the failure this whole feature exists to avoid.
* SEGMENT ALIGNMENT — scene extraction maps scenes to transmissions by index
(segment_indices), so a corrected array of the wrong length would silently
attribute the wrong audio to a scene. Anything but an exact 1:1 match must
be discarded whole.
"""
import pytest
from unittest.mock import AsyncMock, patch
from app.internal import transcript_correction as tc
SYSTEM = {
"vocabulary": ["Croton-Harmon", "Metro-North"],
"ten_codes": {"10-4": "acknowledged", "10-13": "officer needs assistance"},
"area_context": {
"county": "Westchester",
"state": "New York",
"local_knowledge": [
{"term": "Route 9", "meaning": "north-south state highway"},
{"term": "Saw Mill Parkway"},
],
},
"config": {
"talkgroups": [
{
"id": 9048,
"name": "Ossining - Police Dispatch",
"vocabulary": ["Snowden Avenue", "Croton-Harmon"],
"area_context": {
"municipality": "Ossining",
"local_knowledge": [{"term": "Sing Sing", "meaning": "state prison"}],
},
},
{"id": 9600, "name": "Harrison - Police/EMS Dispatch"},
{"id": 9563, "ten_codes": {"10-4": "on scene"}},
]
},
}
def _system(doc=SYSTEM):
return patch.object(tc.fstore, "doc_get_cached", AsyncMock(return_value=doc))
@pytest.fixture(autouse=True)
def _api_key():
"""
The dev venv has no GEMINI_API_KEY, and correct() returns early without one
— which would make every assertion below pass for the wrong reason.
"""
with patch.object(tc.settings, "gemini_api_key", "test-key"):
yield
# ── Scope resolution ────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_talkgroup_vocabulary_ranks_above_system():
with _system():
ctx = await tc.resolve_context("sys-1", 9048)
assert ctx["vocabulary"][0] == "Snowden Avenue", "talkgroup terms must come first"
assert "Metro-North" in ctx["vocabulary"], "system terms are still inherited"
@pytest.mark.asyncio
async def test_duplicate_terms_are_not_repeated():
"""Croton-Harmon is on both scopes; it should appear once, at talkgroup rank."""
with _system():
ctx = await tc.resolve_context("sys-1", 9048)
assert [t.lower() for t in ctx["vocabulary"]].count("croton-harmon") == 1
@pytest.mark.asyncio
async def test_talkgroup_area_precedes_system_area():
with _system():
ctx = await tc.resolve_context("sys-1", 9048)
joined = "\n".join(ctx["area_lines"])
assert joined.index("Ossining") < joined.index("Westchester")
@pytest.mark.asyncio
async def test_talkgroup_without_own_data_inherits_system():
with _system():
ctx = await tc.resolve_context("sys-1", 9600)
assert ctx["vocabulary"] == ["Croton-Harmon", "Metro-North"]
assert any("Westchester" in line for line in ctx["area_lines"])
assert ctx["area"].get("municipality") is None, "inherits, invents nothing"
@pytest.mark.asyncio
async def test_talkgroup_ten_code_overrides_system_meaning():
with _system():
ctx = await tc.resolve_context("sys-1", 9563)
assert ctx["ten_codes"]["10-4"] == "on scene"
assert ctx["ten_codes"]["10-13"] == "officer needs assistance"
@pytest.mark.asyncio
@pytest.mark.parametrize("system_id, tgid", [(None, 9048), ("sys-1", None)])
async def test_missing_scope_is_not_an_error(system_id, tgid):
with _system():
ctx = await tc.resolve_context(system_id, tgid)
assert isinstance(ctx["vocabulary"], list)
@pytest.mark.asyncio
async def test_unconfigured_system_yields_empty_context():
with _system(doc=None):
ctx = await tc.resolve_context("sys-1", 9048)
assert ctx == {
"vocabulary": [], "ten_codes": {}, "area_lines": [],
"area": {}, "system_area": {}, "tg_area": {},
}
# ── Correction behaviour ────────────────────────────────────────────────────
def _gemini(payload):
return patch.object(tc, "_sync_gemini", lambda model, prompt: payload)
SEGS = [{"start": 0.0, "end": 1.0, "text": "Headquarters, 11-9."},
{"start": 1.0, "end": 2.0, "text": "Shout out to Optum."},
{"start": 2.0, "end": 3.0, "text": "360 north, back to Rose."}]
@pytest.mark.asyncio
async def test_short_transcript_is_never_sent():
"""9 of 29 calls in the sample window were <=3 words. Nothing to correct."""
with patch.object(tc, "_sync_gemini") as m:
out = await tc.correct("c1", "10-4.", None, system_id="sys-1")
assert out == (None, None, False)
m.assert_not_called()
@pytest.mark.asyncio
async def test_disabled_by_setting():
with patch.object(tc.settings, "transcript_correction_enabled", False), \
patch.object(tc, "_sync_gemini") as m:
assert await tc.correct("c1", "a b c d e", None) == (None, None, False)
m.assert_not_called()
@pytest.mark.asyncio
async def test_segments_corrected_when_lengths_match():
payload = {"corrected": "Headquarters, 11-9. Show it out to Ossining. 360 north, back to Route 9.",
"segments": ["Headquarters, 11-9.", "Show it out to Ossining.", "360 north, back to Route 9."]}
with _system(), _gemini(payload):
text, segs, not_speech = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1", talkgroup_id=9048)
assert not_speech is False
assert segs is not None and len(segs) == 3
assert segs[1]["text"] == "Show it out to Ossining."
assert segs[1]["start"] == 1.0, "timing must survive correction untouched"
@pytest.mark.asyncio
async def test_wrong_segment_count_is_discarded_whole():
"""A short array would silently misattribute audio to the wrong scene."""
payload = {"corrected": "fine", "segments": ["only", "two"]}
with _system(), _gemini(payload):
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1")
assert segs is None
assert text == "fine", "the joined correction still stands"
@pytest.mark.asyncio
async def test_non_string_segment_entries_are_discarded():
payload = {"corrected": None, "segments": ["ok", 42, "ok"]}
with _system(), _gemini(payload):
_, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1")
assert segs is None
@pytest.mark.asyncio
async def test_unchanged_segments_report_no_correction():
payload = {"corrected": None, "segments": [s["text"] for s in SEGS]}
with _system(), _gemini(payload):
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1")
assert (text, segs) == (None, None)
@pytest.mark.asyncio
async def test_echoed_transcript_counts_as_no_change():
with _system(), _gemini({"corrected": " x y z w "}):
text, _, _ = await tc.correct("c1", "x y z w", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_not_speech_is_surfaced():
with _system(), _gemini({"corrected": None, "not_speech": True}):
_, _, not_speech = await tc.correct("c1", "10-11. 10-12. 10-13. 10-14.", None, system_id="sys-1")
assert not_speech is True
@pytest.mark.asyncio
async def test_model_failure_leaves_the_transcript_alone():
"""Correction is an improvement, never a dependency."""
def boom(model, prompt):
raise RuntimeError("gemini exploded")
with _system(), patch.object(tc, "_sync_gemini", boom):
assert await tc.correct("c1", "x y z w", SEGS, system_id="sys-1") == (None, None, False)
# ── Code-token guard (server-26#162) ────────────────────────────────────────
# Caught live: the same call came back with "10-7" rewritten to "10-13" in one
# place and "10-4" in another. A real code swapped for a different real code
# reads exactly as trustworthy as a correct one — worse than leaving the raw
# mishearing in place, since nothing downstream can tell it happened.
@pytest.mark.asyncio
async def test_changed_ten_code_is_discarded():
payload = {"corrected": "10-13, we're back in town."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "10-7, we're back in town.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_invented_code_token_is_discarded():
"""Nothing code-shaped in the original — the model added one from nothing."""
payload = {"corrected": "ShotSpotter, 10-4, group of 3 shooting outside."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "Seven, group of 3 shooting outside.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_legitimate_place_correction_with_unchanged_codes_still_applies():
"""The guard must not collateral-damage a correction that never touches
a code token — Home/Forest for Holmes/4th-and-Rowe is exactly the kind of
fix this pass exists to make."""
payload = {"corrected": "10-13 coming over on Home Street and Forest Ave, 4-2."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct(
"c1", "10-13 coming over on Holmes Street and 4th and Rowe, 4-2.",
None, system_id="sys-1",
)
assert text == "10-13 coming over on Home Street and Forest Ave, 4-2."
@pytest.mark.asyncio
async def test_segment_code_change_discards_segments_only():
"""A code change in one segment discards the whole segments array (same
all-or-nothing rule as a length mismatch), but the independently-checked
joined correction still stands if it kept its own codes intact. The
joined `text`/`corrected` pair here is deliberately code-free — this test
isolates the segment-level guard, not the joined-text one."""
payload = {
"corrected": "Show it out to Ossining, back to Route 9.",
"segments": ["Headquarters, 10-13.", "Show it out to Ossining.", "360 north, back to Route 9."],
}
with _system(), _gemini(payload):
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1", talkgroup_id=9048)
assert segs is None
assert text == "Show it out to Ossining, back to Route 9."
@pytest.mark.asyncio
async def test_reference_data_reaches_the_prompt():
seen = {}
def capture(model, prompt):
seen["prompt"] = prompt
return {"corrected": None}
with _system(), patch.object(tc, "_sync_gemini", capture):
await tc.correct("c1", "x y z w", None, system_id="sys-1",
talkgroup_id=9048, talkgroup_name="Ossining - Police Dispatch")
p = seen["prompt"]
assert "Snowden Avenue" in p and "Ossining - Police Dispatch" in p
assert "Sing Sing — state prison" in p, "a term without its meaning is half the information"
assert "Ossining, Westchester, New York" in p, "state must reach the prompt (server-26#36)"
assert "10-13=officer needs assistance" in p
@@ -0,0 +1,62 @@
"""
server-26#<pending> — a transcript_too_short call (<=5 words: "10-8", "show me
clear", a unit check-in) never reached correlation at all. upload.py's
no-scenes fallback (the path that lets a no-transcript call still thin-link
by talkgroup) explicitly excluded ANY skip_reason, so short-but-real follow-up
and clearance traffic was permanently unlinkable — not just unextracted by
GPT, but never even attempted against the fast/thin path that already exists
for exactly this kind of content-free signal. garbage_transcript (Whisper
hallucination) has no real content behind it and should stay excluded.
"""
from unittest.mock import AsyncMock, patch
import pytest
from app.routers import upload
ALL_ON = {
"stt_enabled": True,
"correlation_enabled": True,
"summaries_enabled": True,
"vocabulary_learning_enabled": True,
"transcript_correction_enabled": True,
}
async def _run_ingest(skip_reason):
with patch("app.internal.feature_flags.get_flags",
AsyncMock(return_value=ALL_ON)), \
patch("app.internal.firestore.doc_get_cached",
AsyncMock(return_value={"system_id": "sys-1", "ai_flags": {}})), \
patch.object(upload, "fstore") as fs, \
patch.object(upload, "_correlate_with_consensus", AsyncMock(return_value=None)) as corr, \
patch("app.internal.transcription.transcribe_call",
AsyncMock(return_value=("10-8", []))), \
patch("app.internal.intelligence.extract_scenes", AsyncMock(return_value=[])), \
patch("app.internal.alerter.check_and_dispatch", AsyncMock()):
fs.doc_get = AsyncMock(return_value={"skip_reason": skip_reason} if skip_reason else {})
fs.doc_set = AsyncMock()
await upload._run_intelligence_pipeline(
call_id="call-1", node_id="node-1", system_id="sys-1",
talkgroup_id=101, talkgroup_name="PD Dispatch",
gcs_uri="gs://bucket/call-1.mp3",
)
return corr
@pytest.mark.asyncio
async def test_transcript_too_short_now_attempts_correlation():
corr = await _run_ingest("transcript_too_short")
corr.assert_awaited_once()
@pytest.mark.asyncio
async def test_garbage_transcript_still_skips_correlation():
corr = await _run_ingest("garbage_transcript")
corr.assert_not_awaited()
@pytest.mark.asyncio
async def test_no_skip_reason_still_attempts_correlation():
corr = await _run_ingest(None)
corr.assert_awaited_once()
+193 -6
View File
@@ -2,9 +2,10 @@
import { useAuth } from "@/components/AuthProvider";
import { c2api } from "@/lib/c2api";
import { ReplayTab } from "@/components/admin/ReplayTab";
import { useEffect, useState, useRef, useCallback } from "react";
import { useRouter } from "next/navigation";
import type { UserRecord, AuditEntry, UserRole } from "@/lib/types";
import type { UserRecord, AuditEntry, UserRole, CallRecord } from "@/lib/types";
// ---------------------------------------------------------------------------
// Shared primitives
@@ -1047,33 +1048,217 @@ function StaleCallsTab() {
);
}
// ---------------------------------------------------------------------------
// STT eval (server-26#163) — the eval harness for real transcription
// accuracy. Separate from patchTranscript's "fix this call" flow: this never
// re-runs extraction or touches an incident, it only records what was
// actually said next to what Whisper heard, so eval-stats can report a real
// WER instead of a guess. Built to be worked in short sessions, a handful of
// calls at a time, over however many sittings it takes — not a one-shot form.
// ---------------------------------------------------------------------------
function fmtPct(x: number | null | undefined): string {
return x === null || x === undefined ? "—" : `${(x * 100).toFixed(1)}%`;
}
function EvalStatsBar({ stats }: { stats: { eval_count: number; raw_wer: number | null; corrected_wer: number | null } | null }) {
return (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 flex flex-wrap gap-x-8 gap-y-2">
<div>
<p className="text-xs text-gray-500 font-mono">Calls verified</p>
<p className="text-white text-lg font-mono">{stats?.eval_count ?? "—"}</p>
</div>
<div>
<p className="text-xs text-gray-500 font-mono">Raw WER (whisper-1)</p>
<p className="text-white text-lg font-mono">{fmtPct(stats?.raw_wer)}</p>
</div>
<div>
<p className="text-xs text-gray-500 font-mono">Corrected WER (shipped)</p>
<p className="text-white text-lg font-mono">{fmtPct(stats?.corrected_wer)}</p>
</div>
{stats && stats.eval_count > 0 && stats.eval_count < 20 && (
<p className="text-xs text-amber-400 font-mono self-end">
fewer than 20 calls — numbers will move a lot until this grows
</p>
)}
</div>
);
}
function SttEvalTab() {
const [stats, setStats] = useState<{ eval_count: number; raw_wer: number | null; corrected_wer: number | null } | null>(null);
const [queue, setQueue] = useState<CallRecord[]>([]);
const [cursor, setCursor] = useState<string | null>(null);
const [exhausted, setExhausted] = useState(false);
const [draft, setDraft] = useState("");
const [loadingBatch, setLoadingBatch] = useState(false);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const fetching = useRef(false);
const current = queue[0] ?? null;
const refreshStats = useCallback(() => {
c2api.getEvalStats().then(setStats).catch(() => { /* stats are a nice-to-have, not load-bearing */ });
}, []);
const loadBatch = useCallback(async () => {
if (fetching.current) return;
fetching.current = true;
setLoadingBatch(true);
setError(null);
try {
const res = await c2api.getEvalQueue(5, cursor);
setQueue((q) => [...q, ...res.calls]);
setCursor(res.next_cursor);
if (res.calls.length === 0 && !res.next_cursor) setExhausted(true);
} catch (e) {
setError(String(e));
} finally {
setLoadingBatch(false);
fetching.current = false;
}
}, [cursor]);
useEffect(() => { refreshStats(); }, [refreshStats]);
// Auto-refill: whenever the local queue runs dry and there's more to scan
// (or we haven't checked yet), pull another batch. Covers the sparse-window
// case too — a page with matches:0 but a next_cursor just means "keep
// scanning", not "done", so this fires again on its own.
useEffect(() => {
if (queue.length === 0 && !exhausted) loadBatch();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [queue.length, exhausted]);
useEffect(() => {
setDraft(current ? (current.transcript_corrected || current.transcript || "") : "");
}, [current]);
async function saveAndNext() {
if (!current) return;
setSaving(true);
setError(null);
try {
await c2api.putEvalTranscript(current.call_id, draft);
setQueue((q) => q.slice(1));
refreshStats();
} catch (e) {
setError(String(e));
} finally {
setSaving(false);
}
}
function skip() {
setQueue((q) => q.slice(1));
}
return (
<div className="space-y-4">
<p className="text-xs text-gray-500 font-mono">
Listen to the audio, correct the transcript below until it matches what was actually said, then save.
This never touches the call&apos;s real transcript or re-runs anything — it only records ground truth
for measuring the pipeline. Do as many or as few as you have time for; it picks up where you left off.
</p>
<EvalStatsBar stats={stats} />
{error && (
<div className="bg-red-950 border border-red-800 rounded-lg p-3">
<p className="text-red-400 text-sm font-mono">{error}</p>
</div>
)}
{current ? (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 space-y-3">
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs font-mono text-gray-400">
<span>{new Date(current.started_at).toLocaleString()}</span>
<span>{current.talkgroup_name || (current.talkgroup_id ? `TGID ${current.talkgroup_id}` : "unknown talkgroup")}</span>
</div>
{current.audio_url ? (
// eslint-disable-next-line jsx-a11y/media-has-caption
<audio controls src={current.audio_url} className="w-full h-9" />
) : (
<p className="text-xs text-gray-500 italic">No audio on this call — skip it.</p>
)}
<div>
<label className="text-xs text-gray-400 block mb-1">
Machine transcript (pre-filled) — correct it into what was actually said
</label>
<textarea
value={draft}
onChange={(e) => setDraft(e.target.value)}
rows={4}
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm font-mono focus:outline-none focus:border-indigo-500"
/>
</div>
<div className="flex gap-2">
<button
onClick={saveAndNext}
disabled={saving || !draft.trim()}
className="bg-indigo-600 hover:bg-indigo-500 disabled:opacity-50 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors"
>
{saving ? "Saving…" : "Save & next"}
</button>
<button
onClick={skip}
disabled={saving}
className="bg-gray-800 hover:bg-gray-700 disabled:opacity-50 border border-gray-700 text-white text-sm font-mono px-4 py-1.5 rounded-lg transition-colors"
>
Skip
</button>
</div>
</div>
) : (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4">
<p className="text-sm font-mono text-gray-400">
{loadingBatch ? "Loading calls…" : exhausted ? "Nothing left to verify right now — check back after more calls come in." : "Loading…"}
</p>
</div>
)}
</div>
);
}
// ---------------------------------------------------------------------------
// Main admin page
// ---------------------------------------------------------------------------
type AdminTab = "features" | "correlation" | "users" | "audit" | "calls";
type AdminTab = "features" | "correlation" | "replay" | "users" | "audit" | "calls" | "eval";
const TAB_LABELS: { key: AdminTab; label: string }[] = [
{ key: "features", label: "AI Features" },
{ key: "correlation", label: "Correlation Debug" },
{ key: "replay", label: "Replay" },
{ key: "calls", label: "Calls" },
{ key: "eval", label: "STT Eval" },
{ key: "users", label: "Users" },
{ key: "audit", label: "Audit Log" },
];
export default function AdminPage() {
const { user, isAdmin } = useAuth();
const { user, isAdmin, loading: authLoading } = useAuth();
const router = useRouter();
const [tab, setTab] = useState<AdminTab>("features");
// Wait for the claims to resolve before deciding. isAdmin is false for the
// first render of every cold load (typed URL, hard refresh, bookmark) while
// AuthProvider fetches the ID token, so a guard that ignores authLoading
// redirects the admin off their own page every time and only ever lets them
// in via an in-app link. Same shape as /nodes, /systems and /settings.
useEffect(() => {
if (!isAdmin) router.replace("/dashboard");
}, [isAdmin, router]);
if (!authLoading && !isAdmin) router.replace("/");
}, [authLoading, isAdmin, router]);
if (authLoading) return null;
if (!isAdmin) return null;
// Users/Audit tabs benefit from full width; everything else is narrow
const wide = tab === "users" || tab === "audit";
const wide = tab === "users" || tab === "audit" || tab === "replay";
return (
<div className={`space-y-6 ${wide ? "" : "max-w-2xl"}`}>
@@ -1095,7 +1280,9 @@ export default function AdminPage() {
{tab === "features" && <FeaturesTab />}
{tab === "correlation" && <CorrelationDebugTab />}
{tab === "replay" && <ReplayTab />}
{tab === "calls" && <StaleCallsTab />}
{tab === "eval" && <SttEvalTab />}
{tab === "users" && <UsersTab currentUid={user?.uid ?? ""} />}
{tab === "audit" && <AuditLogTab />}
</div>
+19 -4
View File
@@ -1,8 +1,9 @@
"use client";
import { useState } from "react";
import { useEffect, useState } from "react";
import { useAuth } from "@/components/AuthProvider";
import { useAlerts } from "@/lib/useAlerts";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { c2api } from "@/lib/c2api";
import type { AlertRule } from "@/lib/types";
@@ -31,8 +32,8 @@ function RulesTab({ isAdmin }: { isAdmin: boolean }) {
}
}
// Load on first render of this tab
if (!loaded) { load(); }
// Load once when this tab mounts (load() self-guards on `loaded`).
useEffect(() => { load(); }, []);
async function handleCreate(e: React.FormEvent) {
e.preventDefault();
@@ -185,7 +186,7 @@ function RulesTab({ isAdmin }: { isAdmin: boolean }) {
export default function AlertsPage() {
const { isAdmin } = useAuth();
const { alerts, loading } = useAlerts();
const { alerts, loading, error } = useAlerts();
const [tab, setTab] = useState<"events" | "rules">("events");
async function handleAcknowledge(id: string) {
@@ -225,9 +226,22 @@ export default function AlertsPage() {
{tab === "events" && (
loading ? (
<p className="text-gray-500 text-sm font-mono">Loading…</p>
) : error ? (
<p className="text-red-400 text-sm font-mono">
{/requires an index|PERMISSION_DENIED|insufficient permissions/i.test(error)
? "Couldn't load alerts — a database index or security rule isn't deployed on the server yet (server-26 #13 / #51)."
: `Couldn't load alerts: ${error}`}
</p>
) : alerts.length === 0 ? (
<p className="text-gray-600 text-sm font-mono">No alerts triggered yet.</p>
) : (
<div className="space-y-3">
{/* Gate A / A2 (server-26#46) — the Snippet column is transcript text,
and the keyword match that fired the alert was made against it. */}
<MachineOutputNotice
variant="inline"
detail="alerts match against automated transcripts and may fire on, or miss, the wrong words."
/>
<div className="bg-gray-900 border border-gray-800 rounded-xl overflow-hidden">
<table className="w-full text-left">
<thead>
@@ -280,6 +294,7 @@ export default function AlertsPage() {
</tbody>
</table>
</div>
</div>
)
)}
+377 -233
View File
@@ -1,261 +1,405 @@
"use client";
import { useState, useMemo } from "react";
import { useCalls } from "@/lib/useCalls";
import { useSystems } from "@/lib/useSystems";
import { CallRow } from "@/components/CallRow";
// Archive — the call-level view. Until now /calls was a ten-line stub that
// redirected to /incidents, so there was no way to look at a call anywhere in
// the app: the nav's "Archive" link led to the incident list, and a call that
// never correlated was invisible. That is the wrong way round when correlation
// quality is the thing under development — the orphans are the evidence.
//
// Readable by every org member — the Firestore rules already let any member
// read every call in their org. The manual attribution controls stay
// admin-only, matching the admin gate on the link/unlink routes.
import { useCallback, useEffect, useMemo, useState } from "react";
import { useRouter } from "next/navigation";
import { useAuth } from "@/components/AuthProvider";
import type { CallRecord } from "@/lib/types";
import { useSystems } from "@/lib/useSystems";
import { useIncidents } from "@/lib/useIncidents";
import { c2api } from "@/lib/c2api";
import type { CallRecord, IncidentRecord } from "@/lib/types";
import { PageHeader } from "@/components/ui/PageHeader";
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { Button } from "@/components/ui/Button";
import { EmptyState, ErrorBanner } from "@/components/ui/EmptyState";
import { SkeletonCard } from "@/components/ui/Skeleton";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { DateRange, dayStart, dayEnd } from "@/components/ui/DateRange";
const inputCls =
"bg-gray-800 border border-gray-700 rounded-lg px-3 py-1.5 text-sm text-white font-mono " +
"placeholder:text-gray-600 focus:outline-none focus:border-indigo-500 w-full";
type LinkFilter = "any" | "orphan" | "linked";
type TranscriptFilter = "any" | "yes" | "no";
function filterCalls(calls: CallRecord[], filters: Filters): CallRecord[] {
const q = filters.query.trim().toLowerCase();
const tgid = filters.tgid.trim();
const LINK_FILTERS: { key: LinkFilter; label: string }[] = [
{ key: "any", label: "All" },
{ key: "orphan", label: "Orphans" },
{ key: "linked", label: "Linked" },
];
return calls.filter((c) => {
// System filter
if (filters.systemId && c.system_id !== filters.systemId) return false;
const TRANSCRIPT_FILTERS: { key: TranscriptFilter; label: string }[] = [
{ key: "any", label: "Any" },
{ key: "yes", label: "Transcribed" },
{ key: "no", label: "No transcript" },
];
// TGID filter (exact match on the number)
if (tgid && String(c.talkgroup_id ?? "") !== tgid) return false;
const PAGE_SIZE = 50;
// Free-text: talkgroup name, node_id, transcript, tags
if (q) {
const hay = [
c.talkgroup_name ?? "",
c.node_id,
c.transcript ?? "",
c.transcript_corrected ?? "",
...(c.tags ?? []),
].join(" ").toLowerCase();
if (!hay.includes(q)) return false;
function fmtWhen(iso?: string | null): string {
if (!iso) return "—";
try {
const d = new Date(iso);
return `${d.toLocaleDateString([], { month: "short", day: "numeric" })} ${d.toLocaleTimeString([], { hour: "2-digit", minute: "2-digit", second: "2-digit" })}`;
} catch {
return String(iso);
}
}
function fmtDuration(call: CallRecord): string {
if (!call.ended_at) return "active";
const ms = new Date(call.ended_at).getTime() - new Date(call.started_at).getTime();
const s = Math.max(0, Math.round(ms / 1000));
return s < 60 ? `${s}s` : `${Math.floor(s / 60)}m${String(s % 60).padStart(2, "0")}`;
}
function callIncidentIds(call: CallRecord): string[] {
if (call.incident_ids?.length) return call.incident_ids;
return call.incident_id ? [call.incident_id] : [];
}
/** One archive row: metadata, transcript, audio, and the attribution control. */
function ArchiveRow({
call,
systemName,
incidents,
canEdit,
onChanged,
}: {
call: CallRecord;
systemName?: string;
incidents: IncidentRecord[];
canEdit: boolean;
onChanged: () => void;
}) {
const [open, setOpen] = useState(false);
const [audioUrl, setAudioUrl] = useState<string | null>(null);
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
const [attachTo, setAttachTo] = useState("");
const linkedIds = callIncidentIds(call);
const text = call.transcript_corrected || call.transcript || "";
// The stored document holds only the private gs:// object location; a
// playable link is minted per read by the API, so fetch it on expand.
useEffect(() => {
if (!open || audioUrl) return;
let cancelled = false;
c2api
.getCall(call.call_id)
.then((full) => { if (!cancelled) setAudioUrl(full.audio_url ?? null); })
.catch(() => { /* audio is optional — the row is still useful without it */ });
return () => { cancelled = true; };
}, [open, audioUrl, call.call_id]);
async function attach() {
if (!attachTo) return;
setBusy(true); setError(null);
try {
await c2api.linkCallToIncident(attachTo, call.call_id);
setAttachTo("");
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
return true;
});
}
interface Filters {
query: string;
tgid: string;
systemId: string;
dateFrom: string;
dateTo: string;
}
const DEFAULT_FILTERS: Filters = {
query: "",
tgid: "",
systemId: "",
dateFrom: "",
dateTo: "",
};
function isActive(f: Filters) {
return f.query || f.tgid || f.systemId || f.dateFrom || f.dateTo;
}
export default function CallsPage() {
const [limitCount, setLimitCount] = useState(100);
const [filters, setFilters] = useState<Filters>(DEFAULT_FILTERS);
const dateFrom = filters.dateFrom ? new Date(filters.dateFrom + "T00:00:00") : undefined;
const dateTo = filters.dateTo ? new Date(filters.dateTo + "T23:59:59") : undefined;
const { calls, loading } = useCalls(limitCount, dateFrom, dateTo);
const { systems } = useSystems();
const { isAdmin } = useAuth();
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
const [showFilters, setShowFilters] = useState(false);
function set<K extends keyof Filters>(key: K, value: string) {
setFilters((f) => ({ ...f, [key]: value }));
}
const active = calls.filter((c) => c.status === "active");
const ended = calls.filter((c) => c.status === "ended");
const filtered = useMemo(() => filterCalls(ended, filters), [ended, filters]);
async function detach(incidentId: string) {
setBusy(true); setError(null);
try {
await c2api.unlinkCallFromIncident(incidentId, call.call_id);
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
}
const activeFilters = isActive(filters);
return (
<Card padding="none" className="overflow-hidden">
<button
onClick={() => setOpen((v) => !v)}
className="w-full text-left px-4 py-3 hover:bg-raised/40 transition-colors"
>
<div className="flex flex-wrap items-center gap-x-3 gap-y-1">
<span className="text-ink font-mono text-xs shrink-0">{fmtWhen(call.started_at)}</span>
<span className="text-ink-2 text-sm font-medium truncate">
{call.talkgroup_name || (call.talkgroup_id ? `TGID ${call.talkgroup_id}` : "unknown talkgroup")}
</span>
<span className="text-ink-muted text-xs font-mono">{fmtDuration(call)}</span>
{linkedIds.length === 0 ? (
<Badge tone="warning">orphan</Badge>
) : (
<Badge tone="neutral">{linkedIds.length === 1 ? "linked" : `${linkedIds.length} incidents`}</Badge>
)}
{!text && <Badge tone="danger">no transcript</Badge>}
{systemName && <span className="text-ink-muted text-xs ml-auto shrink-0">{systemName}</span>}
</div>
{text && !open && (
<p className="text-ink-muted text-xs mt-1.5 truncate">{text}</p>
)}
</button>
{open && (
<div className="px-4 pb-4 space-y-3 border-t border-line pt-3">
{text ? (
<p className="text-ink-2 text-sm leading-relaxed">{text}</p>
) : (
<p className="text-ink-muted text-xs italic">
No transcript. Either STT was off when this call landed, or Whisper rejected it as
silence or degenerate output.
</p>
)}
{audioUrl && (
/* eslint-disable-next-line jsx-a11y/media-has-caption */
<audio controls src={audioUrl} className="w-full h-9" />
)}
<dl className="grid grid-cols-2 sm:grid-cols-4 gap-x-4 gap-y-1 text-xs">
<div><dt className="text-ink-muted inline">call </dt><dd className="text-ink-2 font-mono inline">{call.call_id.slice(0, 8)}</dd></div>
<div><dt className="text-ink-muted inline">node </dt><dd className="text-ink-2 font-mono inline">{call.node_id ?? "—"}</dd></div>
<div><dt className="text-ink-muted inline">tgid </dt><dd className="text-ink-2 font-mono inline">{call.talkgroup_id ?? "—"}</dd></div>
<div><dt className="text-ink-muted inline">path </dt><dd className="text-ink-2 font-mono inline">{call.corr_path ?? "—"}</dd></div>
</dl>
{/* Manual attribution */}
<div className="space-y-2">
{linkedIds.map((id) => {
const inc = incidents.find((i) => i.incident_id === id);
return (
<div key={id} className="flex items-center gap-2 text-xs">
<span className="text-ink-muted">attached to</span>
<span className="text-ink-2 truncate">{inc?.title ?? id.slice(0, 8)}</span>
{canEdit && <button
onClick={() => detach(id)}
disabled={busy}
className="text-sev-major hover:underline disabled:opacity-50 shrink-0"
>
detach
</button>}
</div>
);
})}
{canEdit && <div className="flex flex-wrap items-center gap-2">
<select
value={attachTo}
onChange={(e) => setAttachTo(e.target.value)}
className="bg-surface border border-line rounded-md text-xs text-ink px-2 py-1.5 max-w-xs"
>
<option value="">Attach to incident…</option>
{incidents
.filter((i) => !linkedIds.includes(i.incident_id))
.slice(0, 100)
.map((i) => (
<option key={i.incident_id} value={i.incident_id}>
{fmtWhen(i.started_at)} — {i.title}
</option>
))}
</select>
<Button size="sm" variant="secondary" onClick={attach} disabled={!attachTo || busy}>
{busy ? "Saving…" : "Attach"}
</Button>
</div>}
</div>
{error && <ErrorBanner message={error} />}
</div>
)}
</Card>
);
}
export default function ArchivePage() {
const { user, orgId, isAdmin, loading: authLoading } = useAuth();
const router = useRouter();
const canView = Boolean(user && (orgId || isAdmin));
const { systems } = useSystems();
const { incidents } = useIncidents(200);
const [calls, setCalls] = useState<CallRecord[]>([]);
const [cursor, setCursor] = useState<string | null>(null);
const [moreAvailable, setMoreAvailable] = useState(false);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [link, setLink] = useState<LinkFilter>("any");
const [transcript, setTranscript] = useState<TranscriptFilter>("any");
const [systemId, setSystemId] = useState("");
const [q, setQ] = useState("");
const [submittedQ, setSubmittedQ] = useState("");
const [dateFrom, setDateFrom] = useState("");
const [dateTo, setDateTo] = useState("");
useEffect(() => {
if (!authLoading && !canView) router.replace("/");
}, [authLoading, canView, router]);
const load = useCallback(
async (nextCursor: string | null, append: boolean) => {
setLoading(true);
setError(null);
try {
const res = await c2api.searchCalls({
limit: PAGE_SIZE,
cursor: nextCursor,
link,
transcript,
system_id: systemId || undefined,
q: submittedQ || undefined,
date_from: dayStart(dateFrom)?.toISOString(),
date_to: dayEnd(dateTo)?.toISOString(),
});
setCalls((prev) => (append ? [...prev, ...res.calls] : res.calls));
setCursor(res.next_cursor);
setMoreAvailable(Boolean(res.next_cursor));
} catch (e) {
setError(String(e));
} finally {
setLoading(false);
}
},
[link, transcript, systemId, submittedQ, dateFrom, dateTo],
);
// Reload from the top whenever a filter changes.
useEffect(() => {
if (authLoading || !canView) return;
load(null, false);
}, [authLoading, canView, load]);
const systemName = useMemo(() => {
const m = new Map(systems.map((s) => [s.system_id, s.name]));
return (id?: string | null) => (id ? m.get(id) : undefined);
}, [systems]);
// Every hook runs before this guard — see the note in app/nodes/page.tsx.
if (authLoading || !canView) return null;
const orphanCount = calls.filter((c) => callIncidentIds(c).length === 0).length;
const noTranscript = calls.filter((c) => !(c.transcript_corrected || c.transcript)).length;
return (
<div className="space-y-6">
<div className="flex items-center justify-between">
<h1 className="text-xl font-bold text-white font-mono">Calls</h1>
<div className="flex items-center gap-3">
<span className="text-xs text-gray-500 font-mono">{calls.length} loaded</span>
<button
onClick={() => setShowFilters((v) => !v)}
className={`text-xs font-mono px-3 py-1.5 rounded-lg border transition-colors ${
activeFilters
? "border-indigo-600 bg-indigo-950 text-indigo-300"
: "border-gray-700 bg-gray-900 text-gray-400 hover:text-gray-200"
}`}
>
{showFilters ? "Hide filters" : "Filter"}
{activeFilters && " •"}
</button>
<PageHeader
title="Archive"
description={isAdmin
? "Every call on the account, correlated or not. Attach an orphan to the incident it belongs to, or detach one the correlator got wrong."
: "Every call on the account, correlated or not."}
/>
<div className="flex flex-wrap items-center gap-3">
<div className="flex gap-1 bg-surface border border-line rounded-lg p-1">
{LINK_FILTERS.map(({ key, label }) => (
<button
key={key}
onClick={() => setLink(key)}
className={`text-sm px-3 py-1.5 rounded-md transition-colors ${
link === key ? "bg-raised text-ink" : "text-ink-muted hover:text-ink-2"
}`}
>
{label}
</button>
))}
</div>
<div className="flex gap-1 bg-surface border border-line rounded-lg p-1">
{TRANSCRIPT_FILTERS.map(({ key, label }) => (
<button
key={key}
onClick={() => setTranscript(key)}
className={`text-sm px-3 py-1.5 rounded-md transition-colors ${
transcript === key ? "bg-raised text-ink" : "text-ink-muted hover:text-ink-2"
}`}
>
{label}
</button>
))}
</div>
<select
value={systemId}
onChange={(e) => setSystemId(e.target.value)}
className="bg-surface border border-line rounded-lg text-sm text-ink px-3 py-2"
>
<option value="">All systems</option>
{systems.map((s) => (
<option key={s.system_id} value={s.system_id}>{s.name}</option>
))}
</select>
<DateRange from={dateFrom} to={dateTo} onChange={(f, t) => { setDateFrom(f); setDateTo(t); }} />
<form
onSubmit={(e) => { e.preventDefault(); setSubmittedQ(q.trim()); }}
className="flex items-center gap-2 ml-auto"
>
<input
value={q}
onChange={(e) => setQ(e.target.value)}
placeholder="Search transcripts…"
className="bg-surface border border-line rounded-lg text-sm text-ink px-3 py-2 w-56"
/>
<Button size="sm" variant="secondary" type="submit">Search</Button>
</form>
</div>
{/* Filter bar */}
{showFilters && (
<div className="bg-gray-900 border border-gray-800 rounded-xl p-4 space-y-3">
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-3">
{/* Text search */}
<div className="lg:col-span-2">
<label className="text-xs text-gray-500 block mb-1">Search (talkgroup, node, transcript, tags)</label>
<input
type="text"
value={filters.query}
onChange={(e) => set("query", e.target.value)}
placeholder="fire, Engine 5, dispatch…"
className={inputCls}
/>
</div>
{calls.length > 0 && (
<p className="text-ink-muted text-xs font-mono">
{calls.length} calls · {orphanCount} orphaned · {noTranscript} without a transcript
</p>
)}
{/* TGID */}
<div>
<label className="text-xs text-gray-500 block mb-1">Talkgroup ID</label>
<input
type="number"
value={filters.tgid}
onChange={(e) => set("tgid", e.target.value)}
placeholder="e.g. 9048"
className={inputCls}
/>
</div>
{/* Gate A / A2 (server-26#46) — every row expands to a transcript. */}
<MachineOutputNotice
detail="transcripts and the incident links derived from them are automated output and may contain errors, including misheard names, addresses and unit numbers. Check the recording before acting on them."
/>
{/* System */}
<div>
<label className="text-xs text-gray-500 block mb-1">System</label>
<select
value={filters.systemId}
onChange={(e) => set("systemId", e.target.value)}
className={inputCls}
>
<option value="">All systems</option>
{systems.map((s) => (
<option key={s.system_id} value={s.system_id}>{s.name}</option>
))}
</select>
</div>
{error && <ErrorBanner message={`Couldn't load calls: ${error}`} />}
{/* Date from */}
<div>
<label className="text-xs text-gray-500 block mb-1">From date</label>
<input
type="date"
value={filters.dateFrom}
onChange={(e) => set("dateFrom", e.target.value)}
className={inputCls}
/>
</div>
{/* Date to */}
<div>
<label className="text-xs text-gray-500 block mb-1">To date</label>
<input
type="date"
value={filters.dateTo}
onChange={(e) => set("dateTo", e.target.value)}
className={inputCls}
/>
</div>
</div>
{activeFilters && (
<div className="flex items-center justify-between pt-1">
<p className="text-xs text-gray-500 font-mono">
{filtered.length} of {ended.length} calls match
</p>
<button
onClick={() => setFilters(DEFAULT_FILTERS)}
className="text-xs text-gray-500 hover:text-gray-300 font-mono transition-colors"
>
Clear all
</button>
</div>
)}
{loading && calls.length === 0 ? (
<div className="space-y-2">
<SkeletonCard /><SkeletonCard /><SkeletonCard />
</div>
) : calls.length === 0 && !error ? (
<EmptyState
title="No calls match these filters"
description="The search scans a bounded window of the most recent calls — widen the filters or clear the search text."
/>
) : (
<div className="space-y-2">
{calls.map((call) => (
<ArchiveRow
key={call.call_id}
call={call}
systemName={systemName(call.system_id)}
incidents={incidents}
canEdit={isAdmin}
onChanged={() => load(null, false)}
/>
))}
</div>
)}
{/* Live calls — never filtered */}
{active.length > 0 && (
<section>
<h2 className="text-sm font-semibold text-orange-400 uppercase tracking-wider mb-3">
Live ({active.length})
</h2>
<div className="bg-gray-900 border border-gray-800 rounded-xl overflow-hidden">
<table className="w-full text-sm">
<thead>
<tr className="text-xs text-gray-500 uppercase tracking-wider border-b border-gray-800">
<th className="px-4 py-2 text-left">Time</th>
<th className="px-4 py-2 text-left">Talkgroup</th>
<th className="px-4 py-2 text-left">System</th>
<th className="px-4 py-2 text-left">Node</th>
<th className="px-4 py-2 text-left">Duration</th>
<th className="px-4 py-2 text-left">Audio</th>
<th className="px-4 py-2"></th>
</tr>
</thead>
<tbody>
{active.map((c) => (
<CallRow key={c.call_id} call={c} systemName={systemMap[c.system_id ?? ""]?.name} isAdmin={isAdmin} />
))}
</tbody>
</table>
</div>
</section>
{moreAvailable && (
<div className="flex justify-center">
<Button variant="secondary" onClick={() => load(cursor, true)} disabled={loading}>
{loading ? "Loading…" : "Load more"}
</Button>
</div>
)}
{/* History */}
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-wider mb-3">
History{activeFilters && <span className="ml-2 text-indigo-400">({filtered.length} filtered)</span>}
</h2>
{loading ? (
<p className="text-gray-600 text-sm font-mono">Loading…</p>
) : filtered.length === 0 ? (
<p className="text-gray-600 text-sm font-mono">
{activeFilters ? "No calls match the current filters." : "No calls recorded yet."}
</p>
) : (
<>
<div className="bg-gray-900 border border-gray-800 rounded-xl overflow-hidden">
<table className="w-full text-sm">
<thead>
<tr className="text-xs text-gray-500 uppercase tracking-wider border-b border-gray-800">
<th className="px-4 py-2 text-left">Time</th>
<th className="px-4 py-2 text-left">Talkgroup</th>
<th className="px-4 py-2 text-left">System</th>
<th className="px-4 py-2 text-left">Node</th>
<th className="px-4 py-2 text-left">Duration</th>
<th className="px-4 py-2 text-left">Audio</th>
</tr>
</thead>
<tbody>
{filtered.map((c) => (
<CallRow key={c.call_id} call={c} systemName={systemMap[c.system_id ?? ""]?.name} isAdmin={isAdmin} />
))}
</tbody>
</table>
</div>
{ended.length >= limitCount && (
<button
onClick={() => setLimitCount((n) => n + 100)}
className="mt-4 text-sm text-indigo-400 hover:text-indigo-300 font-mono transition-colors"
>
Load more
</button>
)}
</>
)}
</section>
</div>
);
}
-170
View File
@@ -1,170 +0,0 @@
"use client";
import Link from "next/link";
import { useRouter } from "next/navigation";
import { useNodes, useUnconfiguredNodes } from "@/lib/useNodes";
import { useCalls, useActiveCalls } from "@/lib/useCalls";
import { useSystems } from "@/lib/useSystems";
import { useActiveIncidents } from "@/lib/useIncidents";
import { NodeCard } from "@/components/NodeCard";
import { CallRow } from "@/components/CallRow";
import { NodeConfigModal } from "@/components/NodeConfigModal";
import { TypeBadge } from "@/components/IncidentBadges";
import { severityBadge, severityRank } from "@/lib/severity";
import { useState } from "react";
import type { NodeRecord, IncidentRecord } from "@/lib/types";
import { useAuth } from "@/components/AuthProvider";
import { PageHeader } from "@/components/ui/PageHeader";
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { Button } from "@/components/ui/Button";
import { EmptyState, ErrorBanner } from "@/components/ui/EmptyState";
function StatCard({ label, value, accent }: { label: string; value: string | number; accent?: string }) {
return (
<Card>
<p className="text-xs text-gray-500 uppercase tracking-wider mb-1">{label}</p>
<p className={`text-3xl font-bold font-mono ${accent ?? "text-white"}`}>{value}</p>
</Card>
);
}
function fmtTime(iso: string) {
try { return new Date(iso).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit" }); }
catch { return iso; }
}
function IncidentSummaryCard({ incident }: { incident: IncidentRecord }) {
const router = useRouter();
return (
<Card hover className="cursor-pointer" onClick={() => router.push(`/incidents/${incident.incident_id}`)}>
<div className="flex items-center gap-2 mb-2 flex-wrap">
<TypeBadge type={incident.type} />
{severityBadge(incident.severity)}
</div>
<p className="text-white text-sm font-semibold leading-snug line-clamp-2">{incident.title ?? "Untitled incident"}</p>
<p className="text-gray-500 text-xs font-mono mt-2">
{fmtTime(incident.started_at)} · {incident.call_ids.length} call{incident.call_ids.length !== 1 ? "s" : ""}
</p>
</Card>
);
}
export default function DashboardPage() {
const { nodes, error: nodesError } = useNodes();
const { nodes: pending } = useUnconfiguredNodes();
const { calls, error: callsError } = useCalls(20);
const activeCalls = useActiveCalls();
const { systems, error: systemsError } = useSystems();
const activeIncidents = useActiveIncidents();
const [configNode, setConfigNode] = useState<NodeRecord | null>(null);
const { isAdmin } = useAuth();
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
const onlineCount = nodes.filter((n) => n.status !== "offline").length;
const fsError = nodesError ?? callsError ?? systemsError;
// Worst-first: the incident that most needs a human's attention leads the panel.
const sortedIncidents = [...activeIncidents].sort(
(a, b) => severityRank(b.severity) - severityRank(a.severity) || b.started_at.localeCompare(a.started_at)
);
const notableIncidentCount = activeIncidents.filter((i) => severityRank(i.severity) >= 2).length;
return (
<div className="space-y-8">
<PageHeader
title="Dashboard"
badge={notableIncidentCount > 0 && <Badge tone="danger">{notableIncidentCount} moderate+ active</Badge>}
/>
{fsError && <ErrorBanner message={`Firestore error: ${fsError}`} />}
{/* Pending config banner */}
{pending.length > 0 && (
<div className="bg-indigo-600/10 border border-indigo-600/40 rounded-lg p-4 flex items-center justify-between gap-3 flex-wrap">
<p className="text-indigo-300 text-sm font-mono">
{pending.length} new node{pending.length > 1 ? "s" : ""} connected and need{pending.length === 1 ? "s" : ""} configuration.
</p>
<Button size="sm" onClick={() => setConfigNode(pending[0])}>Configure now</Button>
</div>
)}
{/* Stats */}
<div className="grid grid-cols-2 md:grid-cols-4 gap-4">
<StatCard label="Active Incidents" value={activeIncidents.length} accent={activeIncidents.length > 0 ? "text-orange-400" : undefined} />
<StatCard label="Nodes Online" value={onlineCount} accent="text-green-400" />
<StatCard label="Active Calls" value={activeCalls.length} accent={activeCalls.length > 0 ? "text-orange-400" : undefined} />
<StatCard label="Systems" value={systems.length} />
</div>
{/* Active incidents — the primary "what's happening" view */}
<section>
<div className="flex items-center justify-between mb-3">
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-wider">Active Incidents</h2>
<Link href="/incidents" className="text-xs text-indigo-400 hover:text-indigo-300 font-mono transition-colors">
View all →
</Link>
</div>
{sortedIncidents.length === 0 ? (
<EmptyState
title="No active incidents"
description="Incidents appear here automatically as calls correlate into events."
/>
) : (
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-4">
{sortedIncidents.slice(0, 6).map((inc) => (
<IncidentSummaryCard key={inc.incident_id} incident={inc} />
))}
</div>
)}
</section>
{/* Nodes */}
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-wider mb-3">Nodes</h2>
{nodes.length === 0 ? (
<EmptyState title="No nodes registered yet" description="Deploy a field SDR node and it will show up here automatically." />
) : (
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
{nodes.map((n) => (
<NodeCard key={n.node_id} node={n} system={systemMap[n.assigned_system_id ?? ""]} />
))}
</div>
)}
</section>
{/* Recent calls */}
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-wider mb-3">Recent Calls</h2>
{calls.length === 0 ? (
<EmptyState title="No calls recorded yet" />
) : (
<Card padding="none" className="overflow-hidden overflow-x-auto">
<table className="w-full text-sm">
<thead>
<tr className="text-xs text-gray-500 uppercase tracking-wider border-b border-gray-800">
<th className="px-4 py-2 text-left">Time</th>
<th className="px-4 py-2 text-left">Talkgroup</th>
<th className="px-4 py-2 text-left">System</th>
<th className="px-4 py-2 text-left">Node</th>
<th className="px-4 py-2 text-left">Duration</th>
<th className="px-4 py-2 text-left">Audio</th>
</tr>
</thead>
<tbody>
{calls.map((c) => (
<CallRow key={c.call_id} call={c} systemName={systemMap[c.system_id ?? ""]?.name} isAdmin={isAdmin} />
))}
</tbody>
</table>
</Card>
)}
</section>
{configNode && (
<NodeConfigModal node={configNode} systems={systems} onClose={() => setConfigNode(null)} />
)}
</div>
);
}
+29 -5
View File
@@ -1,10 +1,13 @@
"use client";
import { useState } from "react";
import type { ReactNode } from "react";
import { Badge } from "@/components/ui/Badge";
import { LinkButton } from "@/components/ui/Button";
import { UnbuiltMarker } from "@/components/ui/UnbuiltMarker";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
const FAQS: { q: string; a: string }[] = [
const FAQS: { q: string; a: ReactNode }[] = [
{
q: "What hardware do I need to run a node?",
a: "A node is a small field SDR device running our edge-node software — it needs an SDR dongle capable of receiving your local P25 or analog trunked system, and a network connection to reach your DRB account. Full setup instructions are provided once you add a node.",
@@ -15,7 +18,14 @@ const FAQS: { q: string; a: string }[] = [
},
{
q: "Does DRB do the transcription and AI work itself, or is that a separate cost?",
a: "Transcription and incident correlation are included in every paid plan and run automatically on every recorded call. The Community plan includes AI features on a limited call volume; Pro and Enterprise scale with your node count.",
a: (
<>
Transcription and incident correlation run automatically on every recorded call and are
included — they are not billed as an add-on.
{/* Gate A / A2 (server-26#46) — qualified on the same screen as the claim. */}
<MachineOutputNotice className="mt-3 not-italic" />
</>
),
},
{
q: "Can I listen to live radio traffic without opening the dashboard?",
@@ -30,8 +40,22 @@ const FAQS: { q: string; a: string }[] = [
a: "You'll see a plan-limit notice in Settings → Billing before anything is blocked. In this demo build there's no live enforcement wired up yet — see the Billing settings page for what's stubbed vs. real.",
},
{
// Gate A / A1 (server-26#46): plan-tiered retention windows are an unbuilt
// entitlement — there is no TTL and no deletion sweep anywhere in the
// product (server-26#44). The claim is marked unbuilt inline, on this
// screen, rather than quietly dropped.
q: "How long is call and incident history kept?",
a: "Retention depends on plan — 7 days on Community, 90 days on Pro, and a year or more on Enterprise (negotiable). Historical calls remain searchable and linked to their incidents for the full retention window.",
a: (
<>
<UnbuiltMarker>Retention limits — not yet available</UnbuiltMarker>
<p className="mt-2">
Today nothing is deleted automatically: calls, recordings and incidents stay searchable and
linked to their incidents for as long as your account is open. Per-plan retention windows and
automatic deletion are not built yet, so we make no commitment about how long anything is kept
or when it goes away. If you need data removed, ask us and we will remove it by hand.
</p>
</>
),
},
{
q: "Is DMR supported?",
@@ -66,7 +90,7 @@ export default function FaqPage() {
{FAQS.map((item, i) => {
const open = openIndex === i;
return (
<div key={item.q}>
<div key={i}>
<button
onClick={() => setOpenIndex(open ? null : i)}
className="w-full flex items-center justify-between gap-4 py-5 text-left focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-indigo-500 rounded-lg"
@@ -76,7 +100,7 @@ export default function FaqPage() {
<ChevronIcon open={open} />
</button>
{open && (
<p className="text-gray-400 text-sm leading-relaxed pb-5 pr-8 animate-fade-in">{item.a}</p>
<div className="text-gray-400 text-sm leading-relaxed pb-5 pr-8 animate-fade-in">{item.a}</div>
)}
</div>
);
+14 -1
View File
@@ -1,8 +1,16 @@
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { LinkButton } from "@/components/ui/Button";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
const SECTIONS = [
const SECTIONS: {
eyebrow: string;
title: string;
body: string;
points: string[];
/** Section describes AI pipeline output — render the Gate A / A2 qualifier. */
qualify?: boolean;
}[] = [
{
eyebrow: "Correlation",
title: "Calls become incidents",
@@ -13,6 +21,7 @@ const SECTIONS = [
"Distance, timing, shared units, and talkgroup signals all feed the match",
"Every call keeps its correlation debug trail for admins to audit",
],
qualify: true,
},
{
eyebrow: "AI pipeline",
@@ -24,6 +33,7 @@ const SECTIONS = [
"Scene & entity extraction feeds the correlator and the incident summary",
"AI-generated incident summaries, regenerable on demand",
],
qualify: true,
},
{
eyebrow: "Situational awareness",
@@ -89,6 +99,9 @@ export default function FeaturesPage() {
</li>
))}
</ul>
{/* Gate A / A2 (server-26#46) — the sections that describe the AI
pipeline carry the same qualifier the product surfaces do. */}
{s.qualify && <MachineOutputNotice className="mt-5" />}
</Card>
</div>
))}
+73 -1
View File
@@ -4,9 +4,67 @@
@import 'leaflet/dist/leaflet.css';
/* ── Design tokens ────────────────────────────────────────────────────────────
* Single source of truth for surface, ink and encoding colour. `:root` is the
* LIGHT theme; `.dark` on <html> (set by ThemeProvider, darkMode: ["class"])
* swaps the same names to their dark values. Tailwind reads these through
* theme.extend.colors, so components say `bg-surface` / `text-ink-muted`
* instead of `bg-gray-900` / `text-gray-400`.
*
* Colour encoding is validated for colour-blindness — see UI_REDESIGN.md §2.3.
* Severity is the ONLY hue channel. Incident type is shape. Node state is value.
* Do not add a hue here for a category; add a glyph.
*/
:root {
--page: #F4F6F9;
--surface: #FFFFFF;
--raised: #F7F9FB;
--line: rgba(11,17,27,.11);
--line-strong: rgba(11,17,27,.22);
--ink: #101620;
--ink-2: #46536A;
--ink-muted: #6B788C;
--accent: #2A78D6;
--sev-moderate: #B07800;
--sev-major: #C0281F;
--map-bg: #E8ECF1;
--map-block: #DFE4EB;
--map-road: #FFFFFF;
--map-water: #D3E2EE;
}
.dark {
--page: #0B0E13;
--surface: #141922;
--raised: #1B2230;
--line: rgba(255,255,255,.09);
--line-strong: rgba(255,255,255,.17);
--ink: #E8EBF0;
--ink-2: #A5B0C2;
--ink-muted: #77839A;
--accent: #3987E5;
--sev-moderate: #C98500;
--sev-major: #D03B3B;
--map-bg: #0E131B;
--map-block: #161C26;
--map-road: #232C3A;
--map-water: #12202E;
}
/* ── Base ─────────────────────────────────────────────────────────────────── */
html, body {
@apply bg-gray-950 text-gray-100 font-mono;
@apply bg-page text-ink;
font-family: var(--font-sans), system-ui, sans-serif;
}
/* Mono is for machine identifiers only — timestamps, talkgroups, unit
* callsigns, node ids, frequencies, incident ids, number columns. */
.font-mono, code, kbd, pre, samp {
font-family: var(--font-mono), ui-monospace, monospace;
}
/* ── Light mode overrides ─────────────────────────────────────────────────── */
@@ -105,6 +163,20 @@ html:not(.dark) .border-indigo-800 { border-color: #a5b4fc !important; }
animation: pulse-ring 1.8s ease-out infinite;
}
/* ── Leaflet stacking fix ─────────────────────────────────────────────────────
* Leaflet's internal panes (z-index 200–700) and its zoom / layers controls
* (z-index 1000) otherwise paint above the sticky app Nav (z-40) and any modal
* overlay — on Live this put the account dropdown *behind* the map. Pinning the
* map container to its own low stacking context keeps Leaflet's internal layer
* order intact while dropping the whole map (tiles + controls) below the app
* chrome. The map's own overlay UI (legend, incident rail, clock, fit-all) sits
* outside .leaflet-container, so it is unaffected and still renders on top.
*/
.leaflet-container {
position: relative;
z-index: 0;
}
/* ── Form inputs ─────────────────────────────────────────────────────────── */
html:not(.dark) input:not([type="submit"]):not([type="button"]):not([type="reset"]),
html:not(.dark) select,
+185 -198
View File
@@ -1,276 +1,263 @@
"use client";
import dynamic from "next/dynamic";
import { useMemo, useState } from "react";
import { useParams, useRouter } from "next/navigation";
import { useState } from "react";
import { useIncident } from "@/lib/useIncidents";
import { useCallsByIncident } from "@/lib/useCalls";
import { useSystems } from "@/lib/useSystems";
import { useAuth } from "@/components/AuthProvider";
import { CallRow } from "@/components/CallRow";
import { CallSpineEntry } from "@/components/CallSpineEntry";
import { c2api } from "@/lib/c2api";
import type { IncidentRecord } from "@/lib/types";
import { TypeBadge } from "@/components/IncidentBadges";
import { severityBadge } from "@/lib/severity";
import { TypeGlyph } from "@/components/marks/TypeGlyph";
import { SeverityMark } from "@/components/marks/SeverityMark";
import { isKnownSeverity } from "@/lib/severity";
import { Button } from "@/components/ui/Button";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import type { CallRecord } from "@/lib/types";
const MapView = dynamic(() => import("@/components/MapView"), { ssr: false });
function StatusBadge({ status }: { status: IncidentRecord["status"] }) {
return (
<span className={`text-xs px-2 py-0.5 rounded-full font-mono ${
status === "active" ? "bg-green-900 text-green-300" : "bg-gray-800 text-gray-400"
}`}>
{status}
</span>
);
const EARLIER_PAGE_SIZE = 8;
function haversineKm(a: { lat: number; lng: number }, b: { lat: number; lng: number }): number {
const R = 6371;
const dLat = ((b.lat - a.lat) * Math.PI) / 180;
const dLng = ((b.lng - a.lng) * Math.PI) / 180;
const s =
Math.sin(dLat / 2) ** 2 +
Math.cos((a.lat * Math.PI) / 180) * Math.cos((b.lat * Math.PI) / 180) * Math.sin(dLng / 2) ** 2;
return R * 2 * Math.atan2(Math.sqrt(s), Math.sqrt(1 - s));
}
type Tab = "summary" | "units" | "details";
function elapsedLabel(startedAt: string, active: boolean, updatedAt: string): string {
const start = new Date(startedAt).getTime();
const end = active ? Date.now() : new Date(updatedAt).getTime();
const mins = Math.max(0, Math.round((end - start) / 60000));
if (mins < 60) return `${mins}m`;
const hrs = Math.floor(mins / 60);
return `${hrs}h ${mins % 60}m`;
}
export default function IncidentDetailPage() {
const params = useParams();
const id = params.id as string;
const id = params.id as string;
const router = useRouter();
const { incident, loading } = useIncident(id);
const { incident, loading } = useIncident(id);
const { calls, loading: callsLoading } = useCallsByIncident(id);
const { systems } = useSystems();
const { isAdmin } = useAuth();
const { isAdmin } = useAuth();
const [tab, setTab] = useState<Tab>("summary");
const [summarizing, setSummarizing] = useState(false);
const [resolving, setResolving] = useState(false);
const [resolving, setResolving] = useState(false);
const [earlierShown, setEarlierShown] = useState(EARLIER_PAGE_SIZE);
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
// Same ordering/filtering MapView's IncidentPathLayer uses, so the stop
// number shown on a spine entry matches the number on its map marker.
const geocodedCalls = useMemo(
() =>
calls
.filter((c): c is CallRecord & { location_coords: { lat: number; lng: number } } => !!c.location_coords)
.slice()
.sort((a, b) => a.started_at.localeCompare(b.started_at)),
[calls]
);
const stopNumberByCallId = useMemo(() => {
const m = new Map<string, number>();
geocodedCalls.forEach((c, i) => m.set(c.call_id, i + 1));
return m;
}, [geocodedCalls]);
const pathLengthKm = useMemo(() => {
let total = 0;
for (let i = 1; i < geocodedCalls.length; i++) {
total += haversineKm(geocodedCalls[i - 1].location_coords!, geocodedCalls[i].location_coords!);
}
return total;
}, [geocodedCalls]);
const newestFirst = useMemo(
() => calls.slice().sort((a, b) => b.started_at.localeCompare(a.started_at)),
[calls]
);
async function handleResolve() {
setResolving(true);
try { await c2api.updateIncident(id, { status: "resolved" }); }
catch (e) { console.error(e); }
finally { setResolving(false); }
finally { setResolving(false); }
}
async function handleSummarize() {
setSummarizing(true);
try { await c2api.summarizeIncident(id); }
catch (e) { console.error(e); }
finally { setSummarizing(false); }
finally { setSummarizing(false); }
}
if (loading) return <p className="text-gray-500 text-sm font-mono p-6">Loading…</p>;
if (!incident) return <p className="text-gray-500 text-sm font-mono p-6">Incident not found.</p>;
if (loading) return <p className="text-ink-muted text-sm p-6">Loading…</p>;
if (!incident) return <p className="text-ink-muted text-sm p-6">Incident not found.</p>;
const displayTags = incident.tags.filter((t) => t !== "auto-generated");
const unitsActive = incident.units_active ?? incident.units ?? [];
const unitsCleared = incident.units_cleared ?? [];
const vehicles = incident.vehicles ?? [];
const active = incident.status === "active";
const visible = newestFirst.slice(0, earlierShown);
const remaining = newestFirst.length - visible.length;
return (
<div className="space-y-4">
{/* Back */}
<button
onClick={() => router.back()}
className="text-xs text-gray-500 hover:text-gray-300 font-mono transition-colors"
className="text-xs text-ink-muted hover:text-ink-2 transition-colors"
>
← Incidents
</button>
{/* Header */}
{/* Header — glyph, severity chip, status, title at 27px, elapsed/path/count */}
<div className="flex flex-col sm:flex-row sm:items-start sm:justify-between gap-3">
<div className="flex flex-col gap-1.5">
<div className="flex flex-col gap-1.5 min-w-0">
<div className="flex items-center gap-2 flex-wrap">
<TypeBadge type={incident.type} />
<StatusBadge status={incident.status} />
{severityBadge(incident.severity)}
<TypeGlyph type={incident.type} size={20} className="text-ink-2" />
{isKnownSeverity(incident.severity) && <SeverityMark severity={incident.severity} showLabel size="md" />}
<span className={`text-xs px-2 py-0.5 rounded-full ${active ? "bg-accent/15 text-accent" : "bg-raised text-ink-2"}`}>
{active ? "Active" : "Resolved"}
</span>
</div>
<h1 className="text-lg sm:text-xl font-bold text-white font-mono leading-snug">
<h1 className="text-[27px] font-semibold text-ink leading-tight">
{incident.title ?? "Incident"}
</h1>
<p className="text-xs text-ink-muted font-mono">
{elapsedLabel(incident.started_at, active, incident.updated_at)} elapsed
{pathLengthKm > 0 && <> · {pathLengthKm.toFixed(1)} km path</>}
{" · "}{incident.call_ids.length} call{incident.call_ids.length !== 1 ? "s" : ""}
</p>
</div>
{isAdmin && (
<div className="flex gap-2 shrink-0 flex-wrap">
<button
onClick={handleSummarize}
disabled={summarizing}
className="text-xs bg-indigo-700 hover:bg-indigo-600 disabled:opacity-50 text-white px-3 py-1.5 rounded-lg transition-colors"
>
<Button variant="secondary" size="sm" onClick={handleSummarize} disabled={summarizing}>
{summarizing ? "Generating…" : "Regenerate summary"}
</button>
{incident.status === "active" && (
<button
onClick={handleResolve}
disabled={resolving}
className="text-xs bg-gray-800 hover:bg-gray-700 disabled:opacity-50 text-gray-300 px-3 py-1.5 rounded-lg transition-colors"
>
{resolving ? "Resolving…" : "Resolve"}
</button>
</Button>
{active && (
<Button variant="secondary" size="sm" onClick={handleResolve} disabled={resolving}>
{resolving ? "Resolving…" : "Mark resolved"}
</Button>
)}
</div>
)}
</div>
{/* Tags */}
{displayTags.length > 0 && (
<div className="flex flex-wrap gap-1">
{displayTags.map((t) => (
<span key={t} className="text-xs bg-gray-800 text-gray-300 px-2 py-0.5 rounded-full">
{t}
</span>
<span key={t} className="text-xs bg-raised text-ink-2 px-2 py-0.5 rounded-full">{t}</span>
))}
</div>
)}
{/* Map */}
{incident.location_coords && (
<div style={{ height: "280px" }}>
<MapView nodes={[]} activeCalls={[]} incidents={[incident]} />
</div>
)}
{/* Two columns: 828 / 612 per UI_REDESIGN.md §5.2 */}
<div className="grid grid-cols-1 lg:grid-cols-5 gap-5">
{/* Left */}
<div className="lg:col-span-3 space-y-4">
{incident.location_coords && (
<div style={{ height: "352px" }} className="rounded-xl overflow-hidden border border-line">
<MapView nodes={[]} activeCalls={[]} incidents={[incident]} calls={calls} />
</div>
)}
{/* Two-panel body */}
<div className="grid grid-cols-1 lg:grid-cols-5 gap-4">
{/* Left: tabs — Summary / Units / Details */}
<div className="lg:col-span-2 bg-gray-900 border border-gray-800 rounded-xl overflow-hidden flex flex-col">
{/* Tab bar */}
<div className="flex border-b border-gray-800 shrink-0">
{(["summary", "units", "details"] as Tab[]).map((t) => (
<button
key={t}
onClick={() => setTab(t)}
className={`flex-1 px-4 py-2.5 text-xs font-mono capitalize transition-colors ${
tab === t
? "text-white border-b-2 border-indigo-500 bg-gray-800/40"
: "text-gray-500 hover:text-gray-300"
}`}
>
{t}
</button>
))}
{/* Summary — first, in prose. Not a tab. */}
<div className="space-y-2.5">
{incident.summary ? (
<p className="text-[16.5px] text-ink leading-[1.58]">{incident.summary}</p>
) : (
<p className="text-sm text-ink-muted italic">
No summary yet.{" "}
{isAdmin && (
<button onClick={handleSummarize} disabled={summarizing} className="text-accent not-italic hover:underline">
Generate now
</button>
)}
</p>
)}
{/* Gate A / A2 (server-26#46): the summary, the title, the location,
the units and the vehicles below are ALL pipeline output, so the
notice sits on this screen with them — not on a policy page. */}
<MachineOutputNotice />
</div>
{/* Tab content */}
<div className="p-4 flex-1 overflow-y-auto">
{tab === "summary" && (
incident.summary ? (
<p className="text-sm text-gray-300 leading-relaxed">{incident.summary}</p>
) : (
<p className="text-sm text-gray-600 font-mono italic">
No summary yet.{" "}
{isAdmin && (
<button
onClick={handleSummarize}
disabled={summarizing}
className="text-indigo-400 hover:text-indigo-300 not-italic transition-colors"
>
Generate now
</button>
)}
</p>
)
)}
{tab === "units" && (
<div className="space-y-4">
<div>
<p className="text-xs text-gray-500 uppercase tracking-wider font-mono mb-2">Units</p>
{incident.units?.length > 0 ? (
<div className="flex flex-wrap gap-1">
{incident.units.map((u) => (
<span key={u} className="text-xs bg-gray-800 text-gray-300 px-2 py-0.5 rounded font-mono">{u}</span>
))}
</div>
) : (
<p className="text-xs text-gray-600 font-mono italic">None extracted.</p>
)}
</div>
<div>
<p className="text-xs text-gray-500 uppercase tracking-wider font-mono mb-2">Vehicles</p>
{incident.vehicles?.length > 0 ? (
<div className="flex flex-wrap gap-1">
{incident.vehicles.map((v) => (
<span key={v} className="text-xs bg-gray-800 text-gray-300 px-2 py-0.5 rounded font-mono">{v}</span>
))}
</div>
) : (
<p className="text-xs text-gray-600 font-mono italic">None extracted.</p>
)}
</div>
</div>
)}
{tab === "details" && (
<div className="space-y-3 text-xs font-mono">
{incident.location && (
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Location</p>
<p className="text-gray-300">{incident.location}</p>
</div>
)}
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Started</p>
<p className="text-gray-300">{new Date(incident.started_at).toLocaleString()}</p>
</div>
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Last activity</p>
<p className="text-gray-300">{new Date(incident.updated_at).toLocaleString()}</p>
</div>
{incident.talkgroup_ids?.length > 0 && (
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Talkgroups</p>
<p className="text-gray-300">{incident.talkgroup_ids.join(", ")}</p>
</div>
)}
{incident.severity && (
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Severity</p>
<p className="text-gray-300 capitalize">{incident.severity}</p>
</div>
)}
<div>
<p className="text-gray-500 uppercase tracking-wider mb-1">Total calls</p>
<p className="text-gray-300">{incident.call_ids.length}</p>
</div>
</div>
)}
</div>
</div>
{/* Right: calls */}
<div className="lg:col-span-3">
<p className="text-xs text-gray-500 uppercase tracking-wider font-mono mb-2">
Calls ({calls.length})
</p>
{callsLoading ? (
<p className="text-gray-600 text-sm font-mono">Loading…</p>
) : calls.length === 0 ? (
<p className="text-gray-600 text-sm font-mono">No calls linked yet.</p>
) : (
<div className="bg-gray-900 border border-gray-800 rounded-xl overflow-hidden overflow-x-auto">
<table className="w-full text-sm">
<thead>
<tr className="text-xs text-gray-500 uppercase tracking-wider border-b border-gray-800">
<th className="px-4 py-2 text-left">Time</th>
<th className="px-4 py-2 text-left">Talkgroup</th>
<th className="px-4 py-2 text-left hidden sm:table-cell">System</th>
<th className="px-4 py-2 text-left hidden sm:table-cell">Node</th>
<th className="px-4 py-2 text-left">Duration</th>
<th className="px-4 py-2 text-left">Audio</th>
<th className="px-4 py-2"></th>
</tr>
</thead>
<tbody>
{calls.map((c) => (
<CallRow
key={c.call_id}
call={c}
systemName={systemMap[c.system_id ?? ""]?.name}
isAdmin={isAdmin}
/>
{/* On scene / Cleared */}
<div className="grid grid-cols-2 gap-4">
<div>
<p className="text-xs text-ink-muted uppercase tracking-wide mb-2">On scene</p>
{unitsActive.length > 0 ? (
<div className="flex flex-wrap gap-1">
{unitsActive.map((u) => (
<span key={u} className="text-xs bg-raised text-ink-2 px-2 py-0.5 rounded font-mono">{u}</span>
))}
</tbody>
</table>
</div>
) : (
<p className="text-xs text-ink-muted italic">None extracted.</p>
)}
</div>
<div>
<p className="text-xs text-ink-muted uppercase tracking-wide mb-2">Cleared</p>
{unitsCleared.length > 0 ? (
<div className="flex flex-wrap gap-1">
{unitsCleared.map((u) => (
<span key={u} className="text-xs bg-transparent border border-line text-ink-muted px-2 py-0.5 rounded font-mono line-through">{u}</span>
))}
</div>
) : (
<p className="text-xs text-ink-muted italic">None yet.</p>
)}
</div>
</div>
{vehicles.length > 0 && (
<div>
<p className="text-xs text-ink-muted uppercase tracking-wide mb-2">Vehicles</p>
<div className="flex flex-wrap gap-1">
{vehicles.map((v) => (
<span key={v} className="text-xs bg-raised text-ink-2 px-2 py-0.5 rounded font-mono">{v}</span>
))}
</div>
</div>
)}
</div>
{/* Right — the call spine */}
<div className="lg:col-span-2">
<p className="text-xs text-ink-muted uppercase tracking-wide mb-1">
Calls ({calls.length})
</p>
{/* Gate A / A2 — the spine renders transcripts. */}
{calls.length > 0 && <MachineOutputNotice variant="inline" className="mb-2" />}
{callsLoading ? (
<p className="text-ink-muted text-sm">Loading…</p>
) : calls.length === 0 ? (
<p className="text-ink-muted text-sm">No calls linked yet.</p>
) : (
<div>
{visible.map((c) => (
<CallSpineEntry
key={c.call_id}
call={c}
stopNumber={stopNumberByCallId.get(c.call_id)}
isAdmin={isAdmin}
/>
))}
{remaining > 0 && (
<button
onClick={() => setEarlierShown((n) => n + EARLIER_PAGE_SIZE)}
className="text-xs text-accent hover:underline mt-2"
>
{remaining} earlier call{remaining !== 1 ? "s" : ""}
</button>
)}
</div>
)}
</div>
</div>
</div>
);
+240 -154
View File
@@ -4,149 +4,141 @@ import { useMemo, useState } from "react";
import { useRouter } from "next/navigation";
import { useAuth } from "@/components/AuthProvider";
import { useIncidents } from "@/lib/useIncidents";
import { useActiveCalls } from "@/lib/useCalls";
import { c2api } from "@/lib/c2api";
import type { IncidentRecord } from "@/lib/types";
import { PageHeader } from "@/components/ui/PageHeader";
import { Card } from "@/components/ui/Card";
import { Button } from "@/components/ui/Button";
import { Badge } from "@/components/ui/Badge";
import { EmptyState } from "@/components/ui/EmptyState";
import { EmptyState, ErrorBanner } from "@/components/ui/EmptyState";
import { SkeletonCard } from "@/components/ui/Skeleton";
import { severityBadge, severityRank } from "@/lib/severity";
import { TypeBadge } from "@/components/IncidentBadges";
// Severity badge/ordering now lives in lib/severity.ts (shared with CallRow).
// `severityBadge()` already returns null for the legacy "unknown" value.
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { DateRange, dayStart, dayEnd } from "@/components/ui/DateRange";
import { isKnownSeverity, severityRank } from "@/lib/severity";
import { SeverityMark, SeveritySpine } from "@/components/marks/SeverityMark";
import { TypeGlyph } from "@/components/marks/TypeGlyph";
type SeverityFilter = "all" | "minor" | "moderate" | "major";
const SEVERITY_FILTERS: { key: SeverityFilter; label: string }[] = [
{ key: "all", label: "All" },
{ key: "minor", label: "Minor+" },
{ key: "all", label: "All" },
{ key: "minor", label: "Minor+" },
{ key: "moderate", label: "Moderate+" },
{ key: "major", label: "Major only" },
{ key: "major", label: "Major only" },
];
const FILTER_THRESHOLD: Record<SeverityFilter, number> = { all: -1, minor: 1, moderate: 2, major: 3 };
type SortMode = "recent" | "severity";
type StatusFilter = "any" | "active" | "resolved";
const INCIDENT_TYPES = ["fire", "police", "ems", "accident", "other"];
// Firestore holds the paging; text/type/status filtering runs over the loaded
// window, so "Load more" also widens what the search can find.
const PAGE_SIZE = 100;
function matchesSearch(inc: IncidentRecord, needle: string): boolean {
if (!needle) return true;
const hay = [
inc.title, inc.location, inc.summary, inc.type,
...(inc.units ?? []), ...(inc.vehicles ?? []), ...(inc.tags ?? []),
...(inc.location_mentions ?? []),
].filter(Boolean).join(" ").toLowerCase();
return hay.includes(needle);
}
// The Firestore client surfaces a missing composite index or an undeployed
// ruleset as a raw multi-line string with a console URL in it — not something
// to put in front of an operator. Collapse the known infra failures to a plain
// line; pass anything else straight through so a real bug still shows.
function friendlyIncidentsError(raw: string): string {
if (/requires an index|PERMISSION_DENIED|Missing or insufficient permissions|failed-precondition/i.test(raw)) {
return "Couldn't load incidents — the incidents database index isn't deployed on the server yet. This is a one-time backend deploy step (server-26 #13 / #51), not a problem with your data.";
}
return `Couldn't load incidents: ${raw}`;
}
function fmtTime(iso: string) {
try { return new Date(iso).toLocaleString(); } catch { return iso; }
try { return new Date(iso).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" }); } catch { return iso; }
}
// ---------------------------------------------------------------------------
// Rows / cards
// ---------------------------------------------------------------------------
function dayBucket(iso: string): string {
const d = new Date(iso);
const now = new Date();
const startOfDay = (x: Date) => new Date(x.getFullYear(), x.getMonth(), x.getDate()).getTime();
const diffDays = Math.round((startOfDay(now) - startOfDay(d)) / 86_400_000);
if (diffDays === 0) return "Today";
if (diffDays === 1) return "Yesterday";
return d.toLocaleDateString([], { weekday: "long", month: "short", day: "numeric" });
}
function IncidentRow({ incident, isAdmin, onResolve }: {
function timeAgo(iso: string): string {
const s = Math.floor((Date.now() - new Date(iso).getTime()) / 1000);
if (s < 60) return `${s}s ago`;
if (s < 3600) return `${Math.floor(s / 60)}m ago`;
if (s < 86400) return `${Math.floor(s / 3600)}h ago`;
return `${Math.floor(s / 86400)}d ago`;
}
// Same rail-card anatomy as Live (MapView's incident panel), at browse
// density: severity spine, type glyph, severity chip, ON AIR pill, title,
// location, units-on-scene chips, age + call count. UI_REDESIGN.md §5.1/§5.2
// — the point is that Live and Incidents read as the same object.
function IncidentBrowseRow({
incident,
isAdmin,
onAir,
onResolve,
}: {
incident: IncidentRecord;
isAdmin: boolean;
onAir: boolean;
onResolve: (id: string) => void;
}) {
const router = useRouter();
const sev = isKnownSeverity(incident.severity) ? incident.severity : "routine";
const units = incident.units_active ?? incident.units ?? [];
return (
<tr
className="border-b border-gray-800 last:border-0 hover:bg-gray-900/60 cursor-pointer transition-colors"
<div
className="flex gap-3 py-3 px-3 rounded-lg hover:bg-raised cursor-pointer transition-colors items-stretch"
onClick={() => router.push(`/incidents/${incident.incident_id}`)}
>
<td className="px-4 py-3"><TypeBadge type={incident.type} /></td>
<td className="px-4 py-3 text-white text-sm">{incident.title ?? "—"}</td>
<td className="px-4 py-3">
<Badge tone={incident.status === "active" ? "success" : "neutral"}>{incident.status}</Badge>
</td>
<td className="px-4 py-3">{severityBadge(incident.severity)}</td>
<td className="px-4 py-3 text-gray-400 text-xs font-mono">{incident.call_ids.length}</td>
<td className="px-4 py-3 text-gray-400 text-xs font-mono">{fmtTime(incident.started_at)}</td>
<td className="px-4 py-3 text-gray-400 text-xs font-mono">{fmtTime(incident.updated_at)}</td>
<td className="px-4 py-3">
{isAdmin && incident.status === "active" && (
<Button
size="sm" variant="secondary"
onClick={(e) => { e.stopPropagation(); onResolve(incident.incident_id); }}
>
Resolve
</Button>
)}
</td>
</tr>
);
}
function IncidentCards({ incidents, isAdmin, onResolve }: {
incidents: IncidentRecord[];
isAdmin: boolean;
onResolve: (id: string) => void;
}) {
const router = useRouter();
return (
<div className="space-y-2">
{incidents.map((inc) => (
<Card
key={inc.incident_id}
padding="sm"
hover
className="cursor-pointer active:bg-gray-800"
onClick={() => router.push(`/incidents/${inc.incident_id}`)}
<SeveritySpine severity={sev} />
<TypeGlyph type={incident.type} size={20} className="text-ink-2 mt-0.5 shrink-0" />
<div className="min-w-0 flex-1">
<div className="flex items-center gap-2 flex-wrap">
<SeverityMark severity={sev} showLabel />
{onAir && (
<span className="text-[10px] font-semibold px-1.5 py-0.5 rounded bg-sev-major/15 text-sev-major uppercase tracking-wide">
On air
</span>
)}
<Badge tone={incident.status === "active" ? "brand" : "neutral"}>{incident.status}</Badge>
</div>
<p className="text-ink text-sm font-semibold leading-snug mt-0.5 truncate">{incident.title ?? "Incident"}</p>
{incident.location && <p className="text-ink-muted text-xs mt-0.5 truncate">{incident.location}</p>}
<div className="flex items-center gap-2 flex-wrap mt-1">
{units.slice(0, 4).map((u) => (
<span key={u} className="text-[10px] font-mono px-1.5 py-0.5 rounded bg-raised text-ink-2">{u}</span>
))}
<span className="text-xs text-ink-muted font-mono ml-auto">
{fmtTime(incident.started_at)} · {timeAgo(incident.started_at)} · {incident.call_ids.length} call{incident.call_ids.length !== 1 ? "s" : ""}
</span>
</div>
</div>
{isAdmin && incident.status === "active" && (
<Button
size="sm" variant="secondary"
className="self-center shrink-0"
onClick={(e) => { e.stopPropagation(); onResolve(incident.incident_id); }}
>
<div className="flex items-center justify-between gap-2 mb-1.5">
<div className="flex items-center gap-2">
<TypeBadge type={inc.type} />
<Badge tone={inc.status === "active" ? "success" : "neutral"}>{inc.status}</Badge>
</div>
{isAdmin && inc.status === "active" && (
<Button size="sm" variant="secondary" onClick={(e) => { e.stopPropagation(); onResolve(inc.incident_id); }}>
Resolve
</Button>
)}
</div>
<p className="text-white text-sm font-semibold leading-snug">{inc.title ?? "—"}</p>
<div className="flex items-center gap-2 mt-1">
{severityBadge(inc.severity)}
<p className="text-gray-500 text-xs font-mono">
{fmtTime(inc.started_at)} · {inc.call_ids.length} call{inc.call_ids.length !== 1 ? "s" : ""}
</p>
</div>
</Card>
))}
Resolve
</Button>
)}
</div>
);
}
function IncidentTable({ incidents, isAdmin, onResolve }: {
incidents: IncidentRecord[];
isAdmin: boolean;
onResolve: (id: string) => void;
}) {
return (
<>
<div className="sm:hidden">
<IncidentCards incidents={incidents} isAdmin={isAdmin} onResolve={onResolve} />
</div>
<div className="hidden sm:block bg-gray-900 border border-gray-800 rounded-xl overflow-hidden overflow-x-auto">
<table className="w-full text-left">
<thead>
<tr className="border-b border-gray-800 text-xs text-gray-500 uppercase">
<th className="px-4 py-3">Type</th>
<th className="px-4 py-3">Title</th>
<th className="px-4 py-3">Status</th>
<th className="px-4 py-3">Severity</th>
<th className="px-4 py-3">Calls</th>
<th className="px-4 py-3">Started</th>
<th className="px-4 py-3">Updated</th>
<th className="px-4 py-3"></th>
</tr>
</thead>
<tbody>
{incidents.map((inc) => (
<IncidentRow key={inc.incident_id} incident={inc} isAdmin={isAdmin} onResolve={onResolve} />
))}
</tbody>
</table>
</div>
</>
);
}
function CreateModal({ onClose, onCreate }: { onClose: () => void; onCreate: (body: object) => Promise<void> }) {
const [title, setTitle] = useState("");
const [type, setType] = useState("other");
@@ -166,20 +158,20 @@ function CreateModal({ onClose, onCreate }: { onClose: () => void; onCreate: (bo
return (
<div className="fixed inset-0 bg-black/60 flex items-center justify-center z-50 p-4">
<form onSubmit={handleSubmit} className="bg-gray-900 border border-gray-700 rounded-xl p-6 w-full max-w-md space-y-4">
<h2 className="text-white font-bold">Create Incident</h2>
<form onSubmit={handleSubmit} className="bg-surface border border-line rounded-xl p-6 w-full max-w-md space-y-4">
<h2 className="text-ink font-semibold">Create Incident</h2>
<div>
<label className="text-xs text-gray-400 block mb-1">Title</label>
<label className="text-xs text-ink-muted block mb-1">Title</label>
<input
required value={title} onChange={(e) => setTitle(e.target.value)}
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm focus:outline-none focus:border-indigo-500"
className="w-full bg-raised border border-line rounded-lg px-3 py-2 text-ink text-sm focus:outline-none focus:border-accent"
/>
</div>
<div>
<label className="text-xs text-gray-400 block mb-1">Type</label>
<label className="text-xs text-ink-muted block mb-1">Type</label>
<select
value={type} onChange={(e) => setType(e.target.value)}
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm focus:outline-none"
className="w-full bg-raised border border-line rounded-lg px-3 py-2 text-ink text-sm focus:outline-none"
>
{["fire", "police", "ems", "accident", "other"].map((t) => (
<option key={t} value={t}>{t}</option>
@@ -187,10 +179,10 @@ function CreateModal({ onClose, onCreate }: { onClose: () => void; onCreate: (bo
</select>
</div>
<div>
<label className="text-xs text-gray-400 block mb-1">Summary (optional)</label>
<label className="text-xs text-ink-muted block mb-1">Summary (optional)</label>
<textarea
value={summary} onChange={(e) => setSummary(e.target.value)} rows={2}
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm focus:outline-none resize-none"
className="w-full bg-raised border border-line rounded-lg px-3 py-2 text-ink text-sm focus:outline-none resize-none"
/>
</div>
<div className="flex gap-3 justify-end">
@@ -202,29 +194,67 @@ function CreateModal({ onClose, onCreate }: { onClose: () => void; onCreate: (bo
);
}
// ---------------------------------------------------------------------------
// Page
// ---------------------------------------------------------------------------
export default function IncidentsPage() {
const { isAdmin } = useAuth();
const { incidents, loading } = useIncidents();
const { isAdmin } = useAuth();
const [pageLimit, setPageLimit] = useState(PAGE_SIZE);
const [dateFrom, setDateFrom] = useState("");
const [dateTo, setDateTo] = useState("");
const rangeFrom = useMemo(() => dayStart(dateFrom), [dateFrom]);
const rangeTo = useMemo(() => dayEnd(dateTo), [dateTo]);
const { incidents, loading, error, hasMore } = useIncidents(pageLimit, rangeFrom, rangeTo);
const activeCalls = useActiveCalls();
const [showCreate, setShowCreate] = useState(false);
const [severityFilter, setSeverityFilter] = useState<SeverityFilter>("all");
const [sortMode, setSortMode] = useState<SortMode>("recent");
const [statusFilter, setStatusFilter] = useState<StatusFilter>("any");
const [typeFilter, setTypeFilter] = useState("");
const [search, setSearch] = useState("");
const onAirIncidentIds = useMemo(() => {
const s = new Set<string>();
for (const c of activeCalls) {
for (const id of c.incident_ids?.length ? c.incident_ids : c.incident_id ? [c.incident_id] : []) s.add(id);
}
return s;
}, [activeCalls]);
const filtered = useMemo(() => {
const threshold = FILTER_THRESHOLD[severityFilter];
const list = incidents.filter((i) => severityRank(i.severity) >= threshold);
const needle = search.trim().toLowerCase();
const list = incidents.filter((i) =>
severityRank(i.severity) >= threshold &&
(statusFilter === "any" || i.status === statusFilter) &&
(!typeFilter || i.type === typeFilter) &&
matchesSearch(i, needle)
);
if (sortMode === "severity") {
return [...list].sort((a, b) => severityRank(b.severity) - severityRank(a.severity) || b.started_at.localeCompare(a.started_at));
}
return list; // useIncidents() already orders by started_at desc
}, [incidents, severityFilter, sortMode]);
}, [incidents, severityFilter, sortMode, statusFilter, typeFilter, search]);
const filtersActive = severityFilter !== "all" || statusFilter !== "any" || typeFilter !== "" || search.trim() !== "" || dateFrom !== "" || dateTo !== "";
function clearFilters() {
setSeverityFilter("all"); setStatusFilter("any"); setTypeFilter(""); setSearch("");
setDateFrom(""); setDateTo(""); setPageLimit(PAGE_SIZE);
}
const active = filtered.filter((i) => i.status === "active");
const resolved = filtered.filter((i) => i.status === "resolved");
const hiddenCount = incidents.length - filtered.length;
const activeCount = filtered.filter((i) => i.status === "active").length;
// Timeline grouping (Today / Yesterday / date) replaces the old
// active/resolved two-table split — status is now a chip on the row, not a
// section boundary, so an active and a resolved incident from the same
// evening read as what they are: the same kind of object.
const groups = useMemo(() => {
const byDay = new Map<string, IncidentRecord[]>();
for (const inc of filtered) {
const key = dayBucket(inc.started_at);
if (!byDay.has(key)) byDay.set(key, []);
byDay.get(key)!.push(inc);
}
return byDay;
}, [filtered]);
async function handleResolve(id: string) {
try { await c2api.updateIncident(id, { status: "resolved" }); }
@@ -235,31 +265,66 @@ export default function IncidentsPage() {
<div className="space-y-6">
<PageHeader
title="Incidents"
badge={active.length > 0 && <Badge tone="danger">{active.length} active</Badge>}
badge={activeCount > 0 && <Badge tone="danger">{activeCount} active</Badge>}
action={isAdmin && <Button onClick={() => setShowCreate(true)}>+ Create Incident</Button>}
/>
{/* Severity filter + sort — severity is a filter dimension, not decoration */}
{/* Gate A / A2 (server-26#46) — every row's title, location and unit
chips are pipeline output, so the notice rides with the list. */}
<MachineOutputNotice variant="inline" />
<div className="flex flex-wrap items-center justify-between gap-3">
<div className="flex flex-wrap gap-1 bg-gray-900 border border-gray-800 rounded-lg p-1 w-fit">
<div className="flex flex-wrap gap-1 bg-surface border border-line rounded-lg p-1 w-fit">
{SEVERITY_FILTERS.map(({ key, label }) => (
<button
key={key}
onClick={() => setSeverityFilter(key)}
className={`text-sm font-mono px-3.5 py-1.5 rounded-md transition-colors ${
severityFilter === key ? "bg-gray-800 text-white" : "text-gray-500 hover:text-gray-300"
className={`text-sm px-3.5 py-1.5 rounded-md transition-colors ${
severityFilter === key ? "bg-raised text-ink" : "text-ink-muted hover:text-ink-2"
}`}
>
{label}
</button>
))}
</div>
<label className="flex items-center gap-2 text-xs font-mono text-gray-500">
<input
type="search"
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder="Search title, location, units…"
className="bg-surface border border-line rounded-lg text-sm text-ink px-3 py-2 w-full sm:w-64 focus:outline-none focus:border-accent"
/>
</div>
<div className="flex flex-wrap items-center gap-3">
<select
value={statusFilter}
onChange={(e) => setStatusFilter(e.target.value as StatusFilter)}
className="bg-surface border border-line rounded-lg px-2 py-1.5 text-sm text-ink-2 focus:outline-none focus:border-accent"
>
<option value="any">Any status</option>
<option value="active">Active</option>
<option value="resolved">Resolved</option>
</select>
<select
value={typeFilter}
onChange={(e) => setTypeFilter(e.target.value)}
className="bg-surface border border-line rounded-lg px-2 py-1.5 text-sm text-ink-2 focus:outline-none focus:border-accent"
>
<option value="">All types</option>
{INCIDENT_TYPES.map((t) => <option key={t} value={t}>{t}</option>)}
</select>
<DateRange
from={dateFrom}
to={dateTo}
onChange={(f, t) => { setDateFrom(f); setDateTo(t); setPageLimit(PAGE_SIZE); }}
/>
<label className="flex items-center gap-2 text-xs text-ink-muted ml-auto">
Sort
<select
value={sortMode}
onChange={(e) => setSortMode(e.target.value as SortMode)}
className="bg-gray-900 border border-gray-800 rounded-lg px-2 py-1.5 text-gray-200 focus:outline-none focus:border-indigo-500"
className="bg-surface border border-line rounded-lg px-2 py-1.5 text-ink-2 focus:outline-none focus:border-accent"
>
<option value="recent">Most recent</option>
<option value="severity">Highest severity</option>
@@ -274,40 +339,61 @@ export default function IncidentsPage() {
) : (
<>
{hiddenCount > 0 && (
<p className="text-xs text-gray-600 font-mono">
{hiddenCount} incident{hiddenCount !== 1 ? "s" : ""} hidden by the severity filter.
<p className="text-xs text-ink-muted">
{hiddenCount} of {incidents.length} loaded incident{incidents.length !== 1 ? "s" : ""} hidden by filters
{hasMore && " — load more to search further back"}.
</p>
)}
{active.length > 0 && (
<section>
<h2 className="text-sm font-mono text-gray-400 uppercase tracking-wider mb-3">Active</h2>
<IncidentTable incidents={active} isAdmin={isAdmin} onResolve={handleResolve} />
{Array.from(groups.entries()).map(([day, incs]) => (
<section key={day}>
<h2 className="text-sm text-ink-muted font-medium mb-1">{day}</h2>
<div className="bg-surface border border-line rounded-xl divide-y divide-line">
{incs.map((inc) => (
<IncidentBrowseRow
key={inc.incident_id}
incident={inc}
isAdmin={isAdmin}
onAir={onAirIncidentIds.has(inc.incident_id)}
onResolve={handleResolve}
/>
))}
</div>
</section>
))}
{/* An empty list is only news when the query actually succeeded.
A failed Firestore query (missing composite index, denied rules)
also leaves `incidents` empty, and rendering "no incidents
recorded yet" over the top of it told the operator the radio was
quiet when the page had simply failed to load — server-26#13. */}
{filtered.length === 0 && error && (
<ErrorBanner message={friendlyIncidentsError(error)} />
)}
{resolved.length > 0 && (
<section>
<h2 className="text-sm font-mono text-gray-400 uppercase tracking-wider mb-3">Resolved</h2>
<IncidentTable incidents={resolved} isAdmin={isAdmin} onResolve={handleResolve} />
</section>
)}
{filtered.length === 0 && (
{filtered.length === 0 && !error && (
<EmptyState
title={incidents.length === 0 ? "No incidents recorded yet" : "No incidents match this filter"}
title={incidents.length === 0 && !filtersActive ? "No incidents recorded yet" : "No incidents match these filters"}
description={
incidents.length === 0
incidents.length === 0 && !filtersActive
? "Incidents appear automatically once calls start correlating."
: "Try a lower severity threshold."
: "Try clearing a filter, or load older incidents."
}
action={
incidents.length > 0 && severityFilter !== "all" ? (
<Button variant="secondary" size="sm" onClick={() => setSeverityFilter("all")}>Clear filter</Button>
filtersActive ? (
<Button variant="secondary" size="sm" onClick={clearFilters}>Clear filters</Button>
) : undefined
}
/>
)}
{hasMore && (
<div className="flex justify-center">
<Button variant="secondary" onClick={() => setPageLimit((n) => n + PAGE_SIZE)}>
Load more
</Button>
</div>
)}
</>
)}
+21 -2
View File
@@ -1,9 +1,26 @@
import type { Metadata } from "next";
import { IBM_Plex_Sans, IBM_Plex_Mono } from "next/font/google";
import { AuthProvider } from "@/components/AuthProvider";
import { ThemeProvider } from "@/components/ThemeProvider";
import { ChromeSwitcher } from "@/components/ChromeSwitcher";
import "./globals.css";
/* Sans for humans (headings, summaries, transcripts, buttons, nav);
* mono for machines (timestamps, talkgroups, unit ids). See UI_REDESIGN.md §2.1. */
const plexSans = IBM_Plex_Sans({
subsets: ["latin"],
weight: ["400", "500", "600"],
variable: "--font-sans",
display: "swap",
});
const plexMono = IBM_Plex_Mono({
subsets: ["latin"],
weight: ["400", "500"],
variable: "--font-mono",
display: "swap",
});
export const metadata: Metadata = {
title: "DRB — Public-Safety Radio Intelligence",
description: "Live incident awareness from field SDR nodes — transcribed, correlated, and mapped in real time.",
@@ -11,12 +28,14 @@ export const metadata: Metadata = {
export default function RootLayout({ children }: { children: React.ReactNode }) {
return (
<html lang="en">
// suppressHydrationWarning: the inline script below adds `.dark` to <html>
// before hydration, so the server/client className legitimately differs.
<html lang="en" suppressHydrationWarning className={`${plexSans.variable} ${plexMono.variable}`}>
<head>
{/* Prevent flash of wrong theme before React hydrates */}
<script dangerouslySetInnerHTML={{ __html: `(function(){try{var t=localStorage.getItem('drb-theme');if(t!=='light')document.documentElement.classList.add('dark');}catch(e){}})();` }} />
</head>
<body className="min-h-screen bg-gray-950">
<body className="min-h-screen bg-page text-ink">
<ThemeProvider>
<AuthProvider>
<ChromeSwitcher>{children}</ChromeSwitcher>
+43 -15
View File
@@ -1,48 +1,74 @@
"use client";
import { useState } from "react";
import { useEffect, useState } from "react";
import Link from "next/link";
import { signInWithEmailAndPassword, GoogleAuthProvider, signInWithPopup } from "firebase/auth";
import { auth } from "@/lib/firebase";
import { c2api } from "@/lib/c2api";
import { useRouter } from "next/navigation";
import { useAuth } from "@/components/AuthProvider";
import { describeAuthError } from "@/lib/authErrors";
export default function LoginPage() {
const [email, setEmail] = useState("");
const [password, setPassword] = useState("");
const [error, setError] = useState<string | null>(null);
const [loading, setLoading] = useState(false);
const [misconfigured, setMisconfigured] = useState(false);
const [submitting, setSubmitting] = useState(false);
const router = useRouter();
const { user, loading: authLoading, orgId } = useAuth();
// Do NOT navigate straight from the sign-in handlers below: signInWith*
// resolves before AuthProvider's onAuthStateChanged listener has fetched
// claims and set/cleared the drb_session cookie. Pushing to the home route
// immediately races that — for a no-org account the cookie never gets
// set, so middleware.ts bounces the very next request straight back to
// /login, which is the ping-pong this screen used to cause. Instead,
// react to AuthProvider's own settled state: this also covers a user who
// arrives here already signed in (e.g. redirected from a protected route
// by middleware while their Firebase session was still valid) — same
// destination logic, no separate code path, no bounce.
useEffect(() => {
if (authLoading) return;
if (!user) return;
router.replace(orgId ? "/" : "/onboarding");
}, [authLoading, user, orgId, router]);
async function handleSubmit(e: React.FormEvent) {
e.preventDefault();
setLoading(true);
setSubmitting(true);
setError(null);
setMisconfigured(false);
try {
await signInWithEmailAndPassword(auth, email, password);
c2api.recordSession().catch(() => {});
router.push("/dashboard");
} catch {
setError("Invalid email or password.");
} finally {
setLoading(false);
// Redirect happens via the effect above once claims are settled.
} catch (err) {
const info = describeAuthError(err, "Invalid email or password.");
setError(info.message);
setMisconfigured(info.misconfiguration);
setSubmitting(false);
}
}
async function handleGoogle() {
setLoading(true);
setSubmitting(true);
setError(null);
setMisconfigured(false);
try {
await signInWithPopup(auth, new GoogleAuthProvider());
c2api.recordSession().catch(() => {});
router.push("/dashboard");
} catch {
setError("Google sign-in failed. Try again.");
} finally {
setLoading(false);
// Redirect happens via the effect above once claims are settled.
} catch (err) {
const info = describeAuthError(err, "Google sign-in failed. Try again.");
setError(info.message);
setMisconfigured(info.misconfiguration);
setSubmitting(false);
}
}
const loading = submitting || !!user;
return (
<div className="max-w-sm mx-auto pt-16">
<Link href="/" className="flex items-center justify-center gap-2 mb-6 font-mono font-bold text-white">
@@ -75,7 +101,9 @@ export default function LoginPage() {
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm focus:outline-none focus:border-indigo-500"
/>
</div>
{error && <p className="text-red-400 text-xs">{error}</p>}
{error && (
<p className={`text-xs ${misconfigured ? "text-amber-400" : "text-red-400"}`}>{error}</p>
)}
<button
type="submit"
disabled={loading}
+5 -78
View File
@@ -1,80 +1,7 @@
"use client";
import { redirect } from "next/navigation";
import { useEffect, useState } from "react";
import dynamic from "next/dynamic";
import { useNodes } from "@/lib/useNodes";
import { useActiveCalls } from "@/lib/useCalls";
import { useActiveIncidents } from "@/lib/useIncidents";
const MapView = dynamic(() => import("@/components/MapView"), { ssr: false });
export default function MapPage() {
const { nodes, loading } = useNodes();
const activeCalls = useActiveCalls();
const incidents = useActiveIncidents();
const [kiosk, setKiosk] = useState(false);
const [lastUpdated, setLastUpdated] = useState<Date | null>(null);
// Track when data last refreshed
useEffect(() => {
if (!loading) setLastUpdated(new Date());
}, [nodes, activeCalls, incidents, loading]);
// Kiosk mode: full-viewport fixed overlay sits above the sticky nav (z-40 → z-50)
if (kiosk) {
return (
<div className="fixed inset-0 z-50 bg-gray-950">
<MapView
nodes={nodes}
activeCalls={activeCalls}
incidents={incidents}
lastUpdated={lastUpdated}
/>
<button
onClick={() => setKiosk(false)}
title="Exit fullscreen"
className="absolute bottom-[5.5rem] left-3 z-[1002] bg-gray-950/90 border border-gray-700 rounded px-3 py-1.5 text-xs font-mono text-gray-300 hover:text-white hover:border-gray-500 transition-colors flex items-center gap-1.5"
>
<svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round">
<path d="M8 3v3a2 2 0 0 1-2 2H3m18 0h-3a2 2 0 0 1-2-2V3m0 18v-3a2 2 0 0 1 2-2h3M3 16h3a2 2 0 0 1 2 2v3"/>
</svg>
Exit fullscreen
</button>
</div>
);
}
return (
<div className="space-y-4">
<div className="flex items-center justify-between">
<h1 className="text-xl font-bold text-white font-mono">Map</h1>
<button
onClick={() => setKiosk(true)}
title="Fullscreen / kiosk mode"
className="text-xs font-mono text-gray-500 hover:text-gray-300 transition-colors flex items-center gap-1.5"
>
<svg width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round">
<path d="M8 3H5a2 2 0 0 0-2 2v3m18 0V5a2 2 0 0 0-2-2h-3m0 18h3a2 2 0 0 0 2-2v-3M3 16v3a2 2 0 0 0 2 2h3"/>
</svg>
Fullscreen
</button>
</div>
{loading ? (
<div className="flex items-center justify-center h-[calc(100vh-10rem)] border border-gray-800 rounded-lg text-gray-600 font-mono text-sm">
Loading map…
</div>
) : (
<div className="w-full h-[calc(100vh-10rem)] border border-gray-800 rounded-lg overflow-hidden">
<MapView
nodes={nodes}
activeCalls={activeCalls}
incidents={incidents}
lastUpdated={lastUpdated}
/>
</div>
)}
</div>
);
// The map is no longer a destination you navigate to — it's the product,
// and the product is the landing page. See UI_REDESIGN.md §3.
export default function MapPageRedirect() {
redirect("/");
}
+104
View File
@@ -0,0 +1,104 @@
"use client";
// The nav's "Network" destination (components/Nav.tsx) — "my equipment".
// The redesign added the link but never the route, so it 404'd and the three
// screens behind it (/nodes, /systems, /tokens) had no entry point in the nav
// at all. This is the hub: it counts what's there, surfaces nodes that still
// need configuring, and hands off to the existing pages.
import { useEffect } from "react";
import Link from "next/link";
import { useRouter } from "next/navigation";
import { useAuth } from "@/components/AuthProvider";
import { useNodes } from "@/lib/useNodes";
import { useSystems } from "@/lib/useSystems";
import { PageHeader } from "@/components/ui/PageHeader";
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
function HubCard({
href,
title,
description,
count,
countLabel,
badge,
}: {
href: string;
title: string;
description: string;
count: number | null;
countLabel: string;
badge?: React.ReactNode;
}) {
return (
<Link href={href} className="block">
<Card hover className="h-full">
<div className="flex items-start justify-between gap-3">
<h2 className="text-ink font-semibold text-sm">{title}</h2>
{badge}
</div>
<p className="text-ink-muted text-xs mt-1.5 leading-snug">{description}</p>
<p className="text-ink text-2xl font-mono mt-4">
{count === null ? "—" : count}
<span className="text-ink-muted text-xs font-sans ml-2">{countLabel}</span>
</p>
</Card>
</Link>
);
}
export default function NetworkPage() {
const { isAdmin, isOperator, loading: authLoading } = useAuth();
const router = useRouter();
const { nodes, loading: nodesLoading } = useNodes();
const { systems, loading: systemsLoading } = useSystems();
useEffect(() => {
if (!authLoading && !isAdmin && !isOperator) router.replace("/");
}, [authLoading, isAdmin, isOperator, router]);
// Every hook runs before this guard — see the note in app/nodes/page.tsx.
if (authLoading || (!isAdmin && !isOperator)) return null;
const pending = nodes.filter((n) => !n.configured);
const online = nodes.filter((n) => n.status === "online" || n.status === "recording");
return (
<div className="space-y-6">
<PageHeader
title="Network"
description="The equipment on your account — field nodes, the radio systems they decode, and the Discord bot tokens they use."
/>
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 gap-4">
<HubCard
href="/nodes"
title="Nodes"
description="Field SDR nodes: status, location, and per-node configuration."
count={nodesLoading ? null : nodes.length}
countLabel={nodesLoading ? "" : `total · ${online.length} up`}
badge={
pending.length > 0 ? (
<Badge tone="warning">{pending.length} need setup</Badge>
) : undefined
}
/>
<HubCard
href="/systems"
title="Systems"
description="Radio system definitions — control channels, talkgroups, and per-system AI flags."
count={systemsLoading ? null : systems.length}
countLabel={systemsLoading ? "" : "configured"}
/>
<HubCard
href="/tokens"
title="Bot Tokens"
description="Discord bot tokens available for nodes to claim when relaying live audio."
count={null}
countLabel="manage"
/>
</div>
</div>
);
}
+8 -2
View File
@@ -9,6 +9,7 @@ import { useCalls } from "@/lib/useCalls";
import { StatusBadge } from "@/components/StatusBadge";
import { NodeConfigModal } from "@/components/NodeConfigModal";
import { CallRow } from "@/components/CallRow";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import { useAuth } from "@/components/AuthProvider";
import { c2api } from "@/lib/c2api";
import type { NodeRecord } from "@/lib/types";
@@ -59,7 +60,7 @@ function DiscordJoinModal({
<div className="fixed inset-0 bg-black/60 flex items-center justify-center z-50 p-4">
<form
onSubmit={handleSubmit}
className="bg-gray-900 border border-gray-700 rounded-xl p-6 space-y-4 font-mono w-full max-w-sm"
className="bg-gray-900 border border-gray-700 rounded-xl p-6 space-y-4 font-mono w-full max-w-sm max-h-[90vh] overflow-y-auto"
>
<h3 className="text-white font-semibold">Join Discord Voice</h3>
<div>
@@ -119,7 +120,10 @@ export default function NodeDetailPage() {
const [approving, setApproving] = useState(false);
const [deleting, setDeleting] = useState(false);
const { systems } = useSystems();
const { calls } = useCalls(20);
// TODO(server-26#109 item5): server-side node_id filter. A where("node_id","==",id)
// alongside the existing org_id equality + started_at orderBy needs a brand-new
// composite index, so for now pull a wider window and filter client-side.
const { calls } = useCalls(200);
const { isAdmin } = useAuth();
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
@@ -335,6 +339,8 @@ export default function NodeDetailPage() {
{/* Recent calls */}
<section>
<h2 className="text-sm font-semibold text-gray-400 uppercase tracking-wider mb-3">Recent Calls</h2>
{/* Gate A / A2 (server-26#46) — each row expands to a transcript. */}
{nodeCalls.length > 0 && <MachineOutputNotice variant="inline" className="mb-3" />}
{nodeCalls.length === 0 ? (
<p className="text-gray-600 text-sm font-mono">No calls recorded from this node.</p>
) : (
+8 -3
View File
@@ -16,12 +16,17 @@ export default function NodesPage() {
const { systems } = useSystems();
useEffect(() => {
if (!authLoading && !isAdmin && !isOperator) router.replace("/dashboard");
if (!authLoading && !isAdmin && !isOperator) router.replace("/");
}, [authLoading, isAdmin, isOperator, router]);
if (authLoading || (!isAdmin && !isOperator)) return null;
const [configNode, setConfigNode] = useState<NodeRecord | null>(null);
// Every hook must run before this guard. React tracks hooks by call order,
// so returning early on the first render and then reaching a useState on the
// next one is error #310 ("rendered more hooks than during the previous
// render") -- which crashed this whole page to a blank client-exception
// screen the moment auth resolved.
if (authLoading || (!isAdmin && !isOperator)) return null;
const systemMap = Object.fromEntries(systems.map((s) => [s.system_id, s]));
const pending = nodes.filter((n) => !n.configured);
@@ -37,7 +42,7 @@ export default function NodesPage() {
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
{pending.map((n) => (
<div key={n.node_id} onClick={() => setConfigNode(n)} className="cursor-pointer">
<NodeCard node={n} system={systemMap[n.assigned_system_id ?? ""]} />
<NodeCard node={n} system={systemMap[n.assigned_system_id ?? ""]} linkToDetail={false} />
</div>
))}
</div>
+2 -2
View File
@@ -29,7 +29,7 @@ export default function OnboardingPage() {
return;
}
if (orgId) {
router.replace("/dashboard");
router.replace("/");
}
}, [loading, user, orgId, router]);
@@ -43,7 +43,7 @@ export default function OnboardingPage() {
// Firebase custom claims only show up in a *freshly fetched* ID token —
// getIdTokenResult(true) inside refreshClaims forces that fetch, then
// AuthProvider's own state (orgId) updates and the effect above
// redirects to /dashboard.
// redirects to "/" (Live).
await refreshClaims();
} catch (err) {
setError(err instanceof Error ? err.message : "Could not set up your organization. Try again.");
+32 -24
View File
@@ -1,8 +1,11 @@
"use client";
import Link from "next/link";
import { LinkButton } from "@/components/ui/Button";
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { PLANS } from "@/lib/billing";
import { useAuth } from "@/components/AuthProvider";
import { LiveView } from "@/components/LiveView";
const CAPABILITIES = [
{
@@ -37,7 +40,7 @@ const STEPS = [
{ n: "03", title: "Your team watches", body: "Incidents show up on the live map and dashboard with an AI summary, units on scene, and every related recording." },
];
export default function MarketingHomePage() {
function MarketingHomePage() {
return (
<div>
{/* Hero */}
@@ -54,8 +57,8 @@ export default function MarketingHomePage() {
bot to relay the audio live to your team.
</p>
<div className="flex flex-wrap items-center gap-3 mt-8">
<LinkButton href="/login" size="lg">Get started</LinkButton>
<LinkButton href="/pricing" variant="secondary" size="lg">View pricing</LinkButton>
<LinkButton href="/waitlist" size="lg">Request access</LinkButton>
<LinkButton href="/login" variant="secondary" size="lg">Sign in</LinkButton>
</div>
</div>
</div>
@@ -96,31 +99,21 @@ export default function MarketingHomePage() {
</div>
</section>
{/* Pricing teaser */}
{/* Pricing — Gate A (minutes #42/#62): no price on a public surface. */}
<section className="border-t border-gray-800">
<div className="max-w-screen-xl mx-auto px-4 md:px-6 py-16 md:py-20">
<div className="flex flex-col md:flex-row md:items-end justify-between gap-4 mb-10">
<div className="flex flex-col md:flex-row md:items-end justify-between gap-4">
<div>
<h2 className="text-display-sm text-white">Plans for one node or a whole region</h2>
<p className="text-gray-400 mt-2">Start free. Upgrade when you add nodes or need longer retention.</p>
<h2 className="text-display-sm text-white">Pricing is in development</h2>
<p className="text-gray-400 mt-2 max-w-xl">
We would rather talk to you about what you need than post a number we would have to
walk back. Tell us about your coverage area and we will figure it out together.
</p>
</div>
<Link href="/pricing" className="text-indigo-400 hover:text-indigo-300 text-sm font-mono transition-colors shrink-0">
See full plan comparison →
More on pricing &rarr;
</Link>
</div>
<div className="grid grid-cols-1 sm:grid-cols-3 gap-5">
{PLANS.map((plan) => (
<Card key={plan.id} padding="lg" highlighted={plan.highlighted}>
{plan.highlighted && <Badge tone="brand" className="mb-3">Most popular</Badge>}
<h3 className="text-white font-semibold">{plan.name}</h3>
<p className="text-gray-500 text-xs mt-1">{plan.tagline}</p>
<p className="text-white text-2xl font-bold font-mono mt-4">
{plan.priceMonthlyUsd === null ? "Custom" : plan.priceMonthlyUsd === 0 ? "Free" : `$${plan.priceMonthlyUsd}`}
{plan.priceMonthlyUsd !== null && plan.priceMonthlyUsd > 0 && <span className="text-gray-500 text-sm font-normal">/mo</span>}
</p>
</Card>
))}
</div>
</div>
</section>
@@ -129,13 +122,28 @@ export default function MarketingHomePage() {
<div className="max-w-screen-xl mx-auto px-4 md:px-6 py-16 md:py-20 text-center">
<h2 className="text-display-sm text-white">Bring your first node online</h2>
<p className="text-gray-400 mt-3 max-w-xl mx-auto">
Sign in to create an account, add a node, and start seeing incidents within minutes of your first call.
Tell us about your coverage area. We will get you a node online and you will see incidents within minutes of your first call.
</p>
<div className="mt-8">
<LinkButton href="/login" size="lg">Get started</LinkButton>
<LinkButton href="/waitlist" size="lg">Request access</LinkButton>
</div>
</div>
</section>
</div>
);
}
/**
* "/" is marketing for a signed-out visitor and Live (the map) for anyone
* signed in — see UI_REDESIGN.md §3, "the map is the home screen". A user
* with no org_id yet still sees marketing (unchanged — that's the
* pre-redesign behaviour for an unprovisioned account, out of scope here;
* ChromeSwitcher's own no-claim guard only fires off marketing paths).
*/
export default function HomePage() {
const { user, loading, orgId } = useAuth();
if (loading) return null;
if (user && orgId) return <LiveView />;
return <MarketingHomePage />;
}
+25 -79
View File
@@ -1,99 +1,45 @@
"use client";
import { useState } from "react";
import Link from "next/link";
import { PLANS, type BillingInterval } from "@/lib/billing";
import { Card } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { LinkButton } from "@/components/ui/Button";
function CheckIcon() {
return (
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="3" strokeLinecap="round" strokeLinejoin="round" className="text-green-400 shrink-0 mt-0.5">
<polyline points="20 6 9 17 4 12" />
</svg>
);
}
/**
* Gate A (BUSINESS_MODEL.md, board minutes #42, enforced by minutes #62):
* no price may appear on a public surface until the pricing model is ratified
* and the entitlements behind it exist. The previous version of this page
* rendered the invented $0/$79/Custom catalog from lib/billing.ts with a
* below-the-fold disclaimer — a false price anchor with a footnote is worse
* than no price. Do not re-import PLANS here. Tracked: server-26#46.
*/
export default function PricingPage() {
const [interval, setInterval] = useState<BillingInterval>("monthly");
return (
<div className="max-w-screen-xl mx-auto px-4 md:px-6 py-16 md:py-20">
<div className="text-center max-w-2xl mx-auto">
<h1 className="text-display-sm md:text-display text-white">Simple, node-based pricing</h1>
<h1 className="text-display-sm md:text-display text-white">Pricing is in development</h1>
<p className="text-gray-400 mt-4">
Every plan includes the full incident pipeline — transcription, correlation, mapping, and the Discord relay.
Plans differ in how many nodes and seats you get, and how far back your history goes.
We are still working out what a fair price looks like for the people who actually use this.
Rather than post a number we would have to walk back, we would rather talk to you about what
you need and what it is worth.
</p>
</div>
{/* Interval toggle */}
<div className="flex items-center justify-center gap-1 mt-10 bg-gray-900 border border-gray-800 rounded-lg p-1 w-fit mx-auto">
{(["monthly", "annual"] as BillingInterval[]).map((i) => (
<button
key={i}
onClick={() => setInterval(i)}
className={`text-sm font-mono px-4 py-1.5 rounded-md transition-colors capitalize ${
interval === i ? "bg-gray-800 text-white" : "text-gray-500 hover:text-gray-300"
}`}
>
{i}
{i === "annual" && <span className="ml-1.5 text-green-400 text-xs">save ~17%</span>}
</button>
))}
</div>
{/* Plan cards */}
<div className="grid grid-cols-1 md:grid-cols-3 gap-6 mt-10 items-stretch">
{PLANS.map((plan) => {
const price = interval === "annual" ? plan.priceAnnualUsd : plan.priceMonthlyUsd;
const priceLabel =
price === null ? "Custom" : price === 0 ? "Free" : `$${interval === "annual" ? Math.round(price / 12) : price}`;
return (
<Card key={plan.id} padding="lg" highlighted={plan.highlighted} className="flex flex-col">
{plan.highlighted && <Badge tone="brand" className="mb-3 w-fit">Most popular</Badge>}
<h2 className="text-white text-lg font-bold">{plan.name}</h2>
<p className="text-gray-500 text-sm mt-1.5 leading-relaxed">{plan.tagline}</p>
<div className="mt-6">
<span className="text-white text-3xl font-bold font-mono">{priceLabel}</span>
{price !== null && price > 0 && <span className="text-gray-500 text-sm">/mo</span>}
{interval === "annual" && price !== null && price > 0 && (
<p className="text-gray-600 text-xs mt-1">billed ${plan.priceAnnualUsd}/year</p>
)}
</div>
<div className="mt-6">
<LinkButton href="/login" variant={plan.highlighted ? "primary" : "secondary"} fullWidth>
{plan.priceMonthlyUsd === null ? "Contact sales" : "Get started"}
</LinkButton>
</div>
<ul className="mt-6 space-y-2.5 flex-1">
{plan.features.map((f) => (
<li key={f} className="flex items-start gap-2 text-sm text-gray-300">
<CheckIcon />
{f}
</li>
))}
</ul>
</Card>
);
})}
</div>
<p className="text-center text-gray-600 text-xs font-mono mt-8">
Prices shown are sample figures for this demo build — nothing here is connected to a live payment processor.
</p>
<Card padding="lg" className="mt-12 max-w-2xl mx-auto">
<h2 className="text-white text-lg font-bold">What you get today</h2>
<p className="text-gray-400 text-sm mt-2 leading-relaxed">
The full incident pipeline — transcription, correlation into incidents, mapping, and the
Discord relay. Node counts, seats, and history length are set per account while we work out
the plan structure.
</p>
<div className="mt-6">
<LinkButton href="/waitlist" fullWidth>Request access</LinkButton>
</div>
</Card>
<div className="text-center mt-16">
<p className="text-gray-400">
Questions about a plan?{" "}
Questions?{" "}
<Link href="/faq" className="text-indigo-400 hover:text-indigo-300 transition-colors">Check the FAQ</Link>
{" "}or{" "}
<Link href="/login" className="text-indigo-400 hover:text-indigo-300 transition-colors">sign in to talk to us</Link>.
<Link href="/waitlist" className="text-indigo-400 hover:text-indigo-300 transition-colors">get in touch</Link>.
</p>
</div>
</div>
@@ -10,6 +10,7 @@ import { Card, CardHeader } from "@/components/ui/Card";
import { Badge } from "@/components/ui/Badge";
import { Button } from "@/components/ui/Button";
import { SkeletonCard } from "@/components/ui/Skeleton";
import { UnbuiltMarker } from "@/components/ui/UnbuiltMarker";
function fmtDate(iso: string) {
return new Date(iso).toLocaleDateString("en-US", { month: "short", day: "numeric", year: "numeric" });
@@ -68,6 +69,19 @@ function UsageBar({ label, used, limit }: { label: string; used: number; limit:
);
}
/**
* Gate A / A1 (server-26#46). The plan cards below render taglines that claim
* entitlements with no backend behind them. Those claims get marked unbuilt
* inline, on this screen, next to the plan that makes them. This is a labelling
* change only — it does not build any of these, and it must never grow into a
* price or a checkout path (Gate B still bars charging anyone).
*/
const UNBUILT_CLAIMS: Partial<Record<PlanId, string[]>> = {
free: ["Retention window"],
pro: ["Retention window"],
enterprise: ["Custom retention", "SSO / SAML", "Uptime SLA", "Data residency"],
};
const INVOICE_TONE: Record<Invoice["status"], "success" | "warning" | "neutral" | "danger"> = {
paid: "success",
open: "warning",
@@ -169,6 +183,13 @@ export default function BillingSettingsPage() {
>
<p className="text-white font-semibold text-sm">{p.name}</p>
<p className="text-gray-500 text-xs mt-1 flex-1">{p.tagline}</p>
{(UNBUILT_CLAIMS[p.id] ?? []).length > 0 && (
<div className="flex flex-wrap gap-1 mt-2">
{(UNBUILT_CLAIMS[p.id] ?? []).map((claim) => (
<UnbuiltMarker key={claim}>{claim} — not yet available</UnbuiltMarker>
))}
</div>
)}
<p className="text-white text-lg font-bold font-mono mt-3">
{p.priceMonthlyUsd === null ? "Custom" : p.priceMonthlyUsd === 0 ? "Free" : `$${p.priceMonthlyUsd}/mo`}
</p>
+1 -1
View File
@@ -27,7 +27,7 @@ export default function SettingsLayout({ children }: { children: React.ReactNode
const router = useRouter();
useEffect(() => {
if (!loading && !canAccess) router.replace("/dashboard");
if (!loading && !canAccess) router.replace("/");
}, [loading, canAccess, router]);
if (loading || !canAccess) return null;
+61 -4
View File
@@ -39,6 +39,30 @@ function EnrollmentTokensPanel() {
const [label, setLabel] = useState("");
const [minting, setMinting] = useState(false);
const [justMinted, setJustMinted] = useState<string | null>(null);
const [cmdCopied, setCmdCopied] = useState(false);
const [tokenCopied, setTokenCopied] = useState(false);
// The label the operator typed for the token that was just minted — used as
// the node id in the install command below. Captured on mint because `label`
// itself is cleared afterward.
const [mintedLabel, setMintedLabel] = useState<string | null>(null);
// The paste-ready one-shot install command for a fresh Pi. The node id comes
// from the label just entered (spaces → dashes; install.sh requires
// [A-Za-z0-9_-]); if that yields nothing it falls back to a node-XXX
// placeholder. The MQTT broker host is the documented mqtt.<domain> sibling
// of the api host (install.sh header) — a DNS assumption the operator checks.
const c2Url = (process.env.NEXT_PUBLIC_C2_URL ?? "https://api.example.net").replace(/\/$/, "");
const mqttBroker = (() => {
try { return `mqtt.${new URL(c2Url).hostname.replace(/^api\./, "")}`; }
catch { return "mqtt.example.net"; }
})();
const nodeIdForCmd =
(mintedLabel ?? "").trim().replace(/\s+/g, "-").replace(/[^A-Za-z0-9_-]/g, "") || "node-XXX";
const installCmd = justMinted
? `curl -fsSL https://git.vpn.cusano.net/logan/node-26/raw/tag/v1/install.sh \\
| sudo bash -s -- --token ${justMinted} --node-id ${nodeIdForCmd} \\
--c2-url ${c2Url} --mqtt-broker ${mqttBroker}`
: "";
const load = useCallback(() => {
c2api.listEnrollmentTokens()
@@ -57,6 +81,7 @@ function EnrollmentTokensPanel() {
try {
const result = await c2api.mintEnrollmentToken(label.trim());
setJustMinted(result.token);
setMintedLabel(label.trim());
setLabel("");
load();
} catch (err) {
@@ -87,11 +112,43 @@ function EnrollmentTokensPanel() {
<p className="text-xs text-indigo-200 font-mono mb-1">
New token — copy it now, it won&apos;t be shown again:
</p>
<p className="text-xs text-indigo-100 font-mono break-all bg-gray-900 rounded px-2 py-1.5">{justMinted}</p>
<div className="flex items-start gap-2">
<p className="flex-1 text-xs text-indigo-100 font-mono break-all bg-gray-900 rounded px-2 py-1.5">{justMinted}</p>
<button
type="button"
onClick={() => navigator.clipboard?.writeText(justMinted).then(() => {
setTokenCopied(true); setTimeout(() => setTokenCopied(false), 2000);
})}
className="text-xs text-indigo-300 hover:text-indigo-200 px-2 py-1.5 transition-colors shrink-0"
>
{tokenCopied ? "Copied" : "Copy"}
</button>
</div>
<p className="text-xs text-indigo-200 font-mono mt-3 mb-1">
…or run this on a fresh Pi{" "}
{nodeIdForCmd === "node-XXX"
? <>(edit <span className="text-indigo-100">node-XXX</span> and check the broker host)</>
: <>(check the broker host)</>}:
</p>
<div className="flex items-start gap-2">
<pre className="flex-1 text-xs text-indigo-100 font-mono whitespace-pre-wrap break-all bg-gray-900 rounded px-2 py-1.5">{installCmd}</pre>
<button
type="button"
onClick={() => navigator.clipboard?.writeText(installCmd).then(() => {
setCmdCopied(true); setTimeout(() => setCmdCopied(false), 2000);
})}
className="text-xs text-indigo-300 hover:text-indigo-200 px-2 py-1.5 transition-colors shrink-0"
>
{cmdCopied ? "Copied" : "Copy"}
</button>
</div>
<button
type="button"
onClick={() => setJustMinted(null)}
className="text-xs text-indigo-300 hover:text-indigo-200 mt-2 transition-colors"
onClick={() => { setJustMinted(null); setMintedLabel(null); }}
className="text-xs text-indigo-300 hover:text-indigo-200 mt-3 transition-colors"
>
Dismiss
</button>
@@ -105,7 +162,7 @@ function EnrollmentTokensPanel() {
<input
value={label}
onChange={(e) => setLabel(e.target.value)}
placeholder="Label, e.g. 'node-003 field kit'"
placeholder="Node ID, e.g. node-003"
className="flex-1 min-w-[12rem] bg-gray-800 border border-gray-700 rounded-lg px-3 py-1.5 text-white text-sm focus:outline-none focus:border-indigo-500"
/>
<Button type="submit" size="sm" disabled={minting || !label.trim()}>
+15 -12
View File
@@ -5,6 +5,7 @@ import Link from "next/link";
import { createUserWithEmailAndPassword, GoogleAuthProvider, signInWithPopup } from "firebase/auth";
import { auth } from "@/lib/firebase";
import { useRouter } from "next/navigation";
import { describeAuthError } from "@/lib/authErrors";
/**
* Self-serve account creation (SAAS_PLAN.md B4). Only creates the Firebase
@@ -17,6 +18,7 @@ export default function SignupPage() {
const [email, setEmail] = useState("");
const [password, setPassword] = useState("");
const [error, setError] = useState<string | null>(null);
const [misconfigured, setMisconfigured] = useState(false);
const [loading, setLoading] = useState(false);
const router = useRouter();
@@ -24,18 +26,14 @@ export default function SignupPage() {
e.preventDefault();
setLoading(true);
setError(null);
setMisconfigured(false);
try {
await createUserWithEmailAndPassword(auth, email, password);
router.push("/onboarding");
} catch (err: unknown) {
const code = (err as { code?: string })?.code;
if (code === "auth/email-already-in-use") {
setError("An account with this email already exists. Try signing in instead.");
} else if (code === "auth/weak-password") {
setError("Password is too weak — use at least 6 characters.");
} else {
setError("Could not create your account. Check your details and try again.");
}
} catch (err) {
const info = describeAuthError(err, "Could not create your account. Check your details and try again.");
setError(info.message);
setMisconfigured(info.misconfiguration);
} finally {
setLoading(false);
}
@@ -44,11 +42,14 @@ export default function SignupPage() {
async function handleGoogle() {
setLoading(true);
setError(null);
setMisconfigured(false);
try {
await signInWithPopup(auth, new GoogleAuthProvider());
router.push("/onboarding");
} catch {
setError("Google sign-up failed. Try again.");
} catch (err) {
const info = describeAuthError(err, "Google sign-up failed. Try again.");
setError(info.message);
setMisconfigured(info.misconfiguration);
} finally {
setLoading(false);
}
@@ -87,7 +88,9 @@ export default function SignupPage() {
className="w-full bg-gray-800 border border-gray-700 rounded-lg px-3 py-2 text-white text-sm focus:outline-none focus:border-indigo-500"
/>
</div>
{error && <p className="text-red-400 text-xs">{error}</p>}
{error && (
<p className={`text-xs ${misconfigured ? "text-amber-400" : "text-red-400"}`}>{error}</p>
)}
<button
type="submit"
disabled={loading}
+506 -7
View File
@@ -5,7 +5,14 @@ import { useRouter } from "next/navigation";
import { useSystems } from "@/lib/useSystems";
import { c2api } from "@/lib/c2api";
import { useAuth } from "@/components/AuthProvider";
import type { SystemRecord, VocabularyPendingTerm } from "@/lib/types";
import { MachineOutputNotice } from "@/components/ui/MachineOutputNotice";
import type {
AreaContext,
LocalKnowledgeEntry,
SystemRecord,
TalkgroupPending,
VocabularyPendingTerm,
} from "@/lib/types";
// ── P25 structured config types ───────────────────────────────────────────────
@@ -13,6 +20,87 @@ interface TalkgroupEntry {
id: string;
name: string;
tag: string;
// Local knowledge for the transcript corrector (server-26#36). Optional on
// every talkgroup: unset means "inherit the system's", which is the whole
// point of the scope rule — talkgroup narrows, it does not replace.
vocabulary?: string[];
area_context?: AreaContext;
}
/** Comma-separated text field <-> string[], the shape the API stores. */
const listToText = (v?: string[]) => (v ?? []).join(", ");
const textToList = (v: string) =>
v.split(",").map((x) => x.trim()).filter(Boolean);
/**
* Local knowledge is one entry per line, `term — meaning`.
*
* A bare term is valid — half the information is still worth having, and
* forcing a meaning would make people invent one. An em dash, a spaced hyphen
* or an equals sign all separate; whichever comes first wins.
*/
const knowledgeToText = (v?: LocalKnowledgeEntry[]) =>
(v ?? []).map((e) => (e.meaning ? `${e.term} — ${e.meaning}` : e.term)).join("\n");
function textToKnowledge(v: string): LocalKnowledgeEntry[] {
const out: LocalKnowledgeEntry[] = [];
for (const line of v.split("\n")) {
const match = line.match(/^(.*?)\s*(?:—|–|\s-\s|=)\s*(.*)$/);
const term = (match ? match[1] : line).trim();
const meaning = match ? match[2].trim() : "";
if (term) out.push(meaning ? { term, meaning } : { term });
}
return out;
}
/**
* Fold the pre-#36 shape forward on load.
*
* `roads[]` and `landmarks[]` were the original fields and real systems still
* have them stored. Showing them as local-knowledge lines means an operator
* sees what they already entered instead of an empty box, and the next save
* writes them in the new shape.
*/
function migrateArea(a?: AreaContext & { roads?: string[]; landmarks?: string[] }): AreaContext {
if (!a) return {};
const legacy = [...(a.roads ?? []), ...(a.landmarks ?? [])].map((term) => ({ term }));
if (!legacy.length) return a;
const seen = new Set((a.local_knowledge ?? []).map((e) => e.term.toLowerCase()));
const { roads: _roads, landmarks: _landmarks, ...rest } = a;
return {
...rest,
local_knowledge: [
...(a.local_knowledge ?? []),
...legacy.filter((e) => !seen.has(e.term.toLowerCase())),
],
};
}
/**
* Drop an area_context whose every field is blank, so we don't store noise.
*
* Also strips the derived anchor: `center`/`radius_km`/`resolved_*` belong to
* the backend, which geocodes them from the place and merges them back. Sending
* them up would be the frontend deciding what is in a system document.
*/
function cleanArea(a?: AreaContext): AreaContext | undefined {
if (!a) return undefined;
const out: AreaContext = {};
if (a.municipality?.trim()) out.municipality = a.municipality.trim();
if (a.county?.trim()) out.county = a.county.trim();
if (a.state?.trim()) out.state = a.state.trim();
if (a.local_knowledge?.length) out.local_knowledge = a.local_knowledge;
return Object.keys(out).length ? out : undefined;
}
/** What the backend resolved this area to, in one line. */
function anchorLabel(a?: AreaContext): string {
if (!a) return "";
const place = [a.municipality, a.county, a.state].filter(Boolean).join(", ");
if (!place) return "no area set — location checking is off for this scope";
if (a.center && a.radius_km) return `anchored to ${place}, ${Math.round(a.radius_km)} km radius`;
if (a.resolved_from) return `${place} — too wide to check locations against, so that check is skipped`;
return `${place} — not yet resolved`;
}
interface P25Config {
@@ -47,10 +135,15 @@ function recordToP25Config(c: Record<string, unknown>): P25Config {
? (c.voice_channels as number[]).join(", ")
: "",
talkgroups: Array.isArray(c.talkgroups)
? (c.talkgroups as Array<{ id: number; name: string; tag: string }>).map((tg) => ({
? (c.talkgroups as Array<{
id: number; name: string; tag: string;
vocabulary?: string[]; area_context?: AreaContext;
}>).map((tg) => ({
id: String(tg.id),
name: tg.name,
tag: tg.tag ?? "other",
vocabulary: tg.vocabulary ?? [],
area_context: migrateArea(tg.area_context),
}))
: [],
};
@@ -70,7 +163,19 @@ function p25ConfigToRecord(p: P25Config): Record<string, unknown> {
voice_channels: parseFreqs(p.voice_channels),
talkgroups: p.talkgroups
.filter((tg) => tg.id && tg.name)
.map((tg) => ({ id: parseInt(tg.id, 10), name: tg.name, tag: tg.tag })),
.map((tg) => {
const area = cleanArea(tg.area_context);
const vocab = (tg.vocabulary ?? []).filter(Boolean);
return {
id: parseInt(tg.id, 10),
name: tg.name,
tag: tg.tag,
// Omitted rather than written empty: an absent key is what
// resolve_context() reads as "inherit from the system".
...(vocab.length ? { vocabulary: vocab } : {}),
...(area ? { area_context: area } : {}),
};
}),
};
}
@@ -316,6 +421,50 @@ function RRImportModal({
);
}
/** True when this talkgroup overrides anything, so the row can say so. */
function hasLocalKnowledge(tg: TalkgroupEntry): boolean {
return Boolean((tg.vocabulary ?? []).length || cleanArea(tg.area_context));
}
/** One labelled field in the local-knowledge grid. */
function LocalKnowledgeField({
label,
value,
onChange,
placeholder,
multiline,
}: {
label: string;
value: string;
onChange: (v: string) => void;
placeholder?: string;
multiline?: boolean;
}) {
const cls =
"w-full mt-0.5 bg-gray-900 border border-gray-700 rounded px-2 py-1 text-white text-xs focus:outline-none focus:border-indigo-500";
return (
<label className="block">
<span className="text-xs text-gray-500 font-sans">{label}</span>
{multiline ? (
<textarea
value={value}
onChange={(e) => onChange(e.target.value)}
placeholder={placeholder}
rows={4}
className={`${cls} font-mono resize-y`}
/>
) : (
<input
value={value}
onChange={(e) => onChange(e.target.value)}
placeholder={placeholder}
className={cls}
/>
)}
</label>
);
}
// ── Talkgroup table editor ────────────────────────────────────────────────────
function TalkgroupEditor({
@@ -329,12 +478,34 @@ function TalkgroupEditor({
const [pasteText, setPasteText] = useState("");
const [rrSystem, setRrSystem] = useState<RRSystem | null>(null);
const [rrError, setRrError] = useState<string | null>(null);
const [expanded, setExpanded] = useState<number | null>(null);
const rrInputRef = useRef<HTMLInputElement>(null);
function addRow() {
onChange([...talkgroups, { id: "", name: "", tag: "other" }]);
}
function updateArea(i: number, field: "municipality" | "county" | "state", value: string) {
const updated = [...talkgroups];
updated[i] = { ...updated[i], area_context: { ...updated[i].area_context, [field]: value } };
onChange(updated);
}
function updateAreaKnowledge(i: number, value: string) {
const updated = [...talkgroups];
updated[i] = {
...updated[i],
area_context: { ...updated[i].area_context, local_knowledge: textToKnowledge(value) },
};
onChange(updated);
}
function updateRowList(i: number, field: "vocabulary", value: string) {
const updated = [...talkgroups];
updated[i] = { ...updated[i], [field]: textToList(value) };
onChange(updated);
}
function removeRow(i: number) {
onChange(talkgroups.filter((_, idx) => idx !== i));
}
@@ -478,7 +649,8 @@ function TalkgroupEditor({
</thead>
<tbody>
{talkgroups.map((tg, i) => (
<tr key={i} className="border-t border-gray-800 hover:bg-gray-800/30">
<Fragment key={i}>
<tr className="border-t border-gray-800 hover:bg-gray-800/30">
<td className="px-2 py-1">
<input
value={tg.id}
@@ -507,6 +679,16 @@ function TalkgroupEditor({
</select>
</td>
<td className="px-2 py-1 text-center">
<button
type="button"
title="Local knowledge for the transcript corrector"
onClick={() => setExpanded(expanded === i ? null : i)}
className={`transition-colors font-bold mr-2 ${
hasLocalKnowledge(tg) ? "text-indigo-400" : "text-gray-600 hover:text-gray-300"
}`}
>
{expanded === i ? "▾" : "▸"}
</button>
<button
type="button"
onClick={() => removeRow(i)}
@@ -516,6 +698,60 @@ function TalkgroupEditor({
</button>
</td>
</tr>
{/* Local knowledge, collapsed by default: a system can carry 125
talkgroups and most inherit the system's context rather than
override it. */}
{expanded === i && (
<tr className="border-t border-gray-800 bg-gray-900/60">
<td colSpan={4} className="px-3 py-3">
<p className="text-xs text-gray-500 mb-2 font-sans">
Given to the transcript corrector for this talkgroup only, ranked
<span className="text-gray-300"> above</span> the system&apos;s own list.
Leave blank to inherit the system&apos;s.
</p>
<div className="grid grid-cols-1 md:grid-cols-2 gap-2">
<LocalKnowledgeField
label="Municipality"
value={tg.area_context?.municipality ?? ""}
onChange={(v) => updateArea(i, "municipality", v)}
placeholder="Ossining"
/>
<LocalKnowledgeField
label="County"
value={tg.area_context?.county ?? ""}
onChange={(v) => updateArea(i, "county", v)}
placeholder="Westchester"
/>
<LocalKnowledgeField
label="State"
value={tg.area_context?.state ?? ""}
onChange={(v) => updateArea(i, "state", v)}
placeholder="New York"
/>
<div className="md:col-span-2">
<LocalKnowledgeField
label="Local knowledge — one per line, term — what it is"
multiline
value={knowledgeToText(tg.area_context?.local_knowledge)}
onChange={(v) => updateAreaKnowledge(i, v)}
placeholder={
"Snowden Avenue — residential street\nSing Sing — state prison\n11-X-ray — MTA PD patrol unit"
}
/>
</div>
<div className="md:col-span-2">
<LocalKnowledgeField
label="Unit call signs & local terms"
value={listToText(tg.vocabulary)}
onChange={(v) => updateRowList(i, "vocabulary", v)}
placeholder="Post 4, 11-X-ray, Car 7"
/>
</div>
</div>
</td>
</tr>
)}
</Fragment>
))}
</tbody>
</table>
@@ -988,7 +1224,11 @@ function SourceCallPlayer({ callId }: { callId: string }) {
<p className="text-gray-600 italic">No audio</p>
)}
{transcript && (
<p className="text-gray-500 italic line-clamp-2">{transcript}</p>
<>
<p className="text-gray-500 italic line-clamp-2">{transcript}</p>
{/* Gate A / A2 (server-26#46) — this is a raw pipeline transcript. */}
<MachineOutputNotice variant="inline" className="text-[10px]" />
</>
)}
</div>
)}
@@ -996,8 +1236,260 @@ function SourceCallPlayer({ callId }: { callId: string }) {
);
}
// ── Area context panel ────────────────────────────────────────────────────────
/**
* System-wide ground truth for the transcript corrector (server-26#36).
*
* This is the fallback every talkgroup inherits. A talkgroup that covers one
* municipality inside a multi-county system overrides it from the talkgroup
* table in the edit form, and its entries rank above these.
*/
function AreaContextPanel({ systemId }: { systemId: string }) {
const [open, setOpen] = useState(false);
const [area, setArea] = useState<AreaContext | null>(null);
const [loading, setLoading] = useState(false);
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
async function toggle() {
const next = !open;
setOpen(next);
if (!next || area !== null) return;
setLoading(true);
try {
const data = await c2api.getAreaContext(systemId);
setArea(migrateArea(data.area_context));
} catch (e) {
setError(String(e));
} finally {
setLoading(false);
}
}
async function save() {
if (!area) return;
setSaving(true);
setError(null);
try {
// The response carries the anchor this edit produced, so the readout
// below reflects what the backend actually resolved rather than what was
// typed.
const saved = await c2api.updateAreaContext(systemId, area);
setArea(saved.area_context ?? area);
} catch (e) {
setError(String(e));
} finally {
setSaving(false);
}
}
const filled = area
? [area.municipality, area.county, area.state, area.local_knowledge?.length].filter(Boolean).length
: 0;
return (
<div className="mt-3 border-t border-gray-800 pt-3">
<button
onClick={toggle}
className="text-xs text-gray-500 hover:text-gray-300 font-mono transition-colors flex items-center gap-1"
>
<span>{open ? "▲" : "▼"}</span>
<span>
Local Area
{area !== null && (
<span className="text-gray-600 ml-1">
({filled > 0 ? `${filled} field${filled === 1 ? "" : "s"} set` : "not set"})
</span>
)}
</span>
</button>
{open && (
<div className="mt-3 space-y-3 text-xs">
{loading && <p className="text-gray-600 italic font-mono">Loading…</p>}
{area && (
<>
<p className="text-gray-500">
Given to the transcript corrector for every talkgroup on this system.
Whisper mishears local names constantly — naming them here is what lets
them be put back.
</p>
<p className="text-gray-500">
Fill this in <span className="text-gray-300">only if it is true of every
talkgroup</span> on the system. One town, one department — fill it once here.
A system spanning several counties — leave it blank and describe each
talkgroup in the edit form instead.
</p>
<div className="grid grid-cols-1 md:grid-cols-3 gap-2">
<LocalKnowledgeField
label="Municipality"
value={area.municipality ?? ""}
onChange={(v) => setArea({ ...area, municipality: v })}
placeholder="Ossining"
/>
<LocalKnowledgeField
label="County"
value={area.county ?? ""}
onChange={(v) => setArea({ ...area, county: v })}
placeholder="Westchester"
/>
<LocalKnowledgeField
label="State"
value={area.state ?? ""}
onChange={(v) => setArea({ ...area, state: v })}
placeholder="New York"
/>
</div>
<LocalKnowledgeField
label="Local knowledge — one per line, term — what it is"
multiline
value={knowledgeToText(area.local_knowledge)}
onChange={(v) => setArea({ ...area, local_knowledge: textToKnowledge(v) })}
placeholder={
"Route 9 — main north-south highway\nPhelps — Phelps Hospital, Sleepy Hollow\nthe flats — low-lying area by the river"
}
/>
{/* The anchor is the backend's answer to what was typed above, and
it decides whether location checking runs at all — so it is
worth showing rather than leaving as invisible state. */}
<p className="text-gray-600 font-mono">{anchorLabel(area)}</p>
<button
type="button"
onClick={save}
disabled={saving}
className="bg-indigo-700 hover:bg-indigo-600 disabled:opacity-50 text-white px-3 py-1.5 rounded text-xs font-semibold transition-colors"
>
{saving ? "Saving…" : "Save area"}
</button>
</>
)}
{error && <p className="text-red-400 font-mono">{error}</p>}
</div>
)}
</div>
);
}
// ── Vocabulary panel ──────────────────────────────────────────────────────────
/**
* Terms the place verifier and the induction loop proposed, per talkgroup.
*
* Approval is always a person's decision and always lands on the talkgroup that
* proposed it — nothing here promotes a term to the system (server-26#37). A
* wrong term on one channel misleads one channel; the same term system-wide
* misleads every channel on it, including one on the far side of a statewide
* system.
*/
function TalkgroupPendingPanel({ systemId }: { systemId: string }) {
const [open, setOpen] = useState(false);
const [rows, setRows] = useState<TalkgroupPending[] | null>(null);
const [busy, setBusy] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
async function load() {
try {
const data = await c2api.getTalkgroupPending(systemId);
setRows(data.talkgroups ?? []);
} catch (e) {
setError(String(e));
}
}
async function toggle() {
const next = !open;
setOpen(next);
if (next && rows === null) await load();
}
async function act(talkgroupId: number, term: string, approve: boolean) {
setBusy(`${talkgroupId}:${term}`);
setError(null);
try {
if (approve) await c2api.approveTalkgroupTerm(systemId, talkgroupId, term);
else await c2api.dismissTalkgroupTerm(systemId, talkgroupId, term);
await load();
} catch (e) {
setError(String(e));
} finally {
setBusy(null);
}
}
const total = (rows ?? []).reduce((n, r) => n + r.pending.length, 0);
return (
<div className="mt-3 border-t border-gray-800 pt-3">
<button
onClick={toggle}
className="text-xs text-gray-500 hover:text-gray-300 font-mono transition-colors flex items-center gap-1"
>
<span>{open ? "▲" : "▼"}</span>
<span>
Proposed Terms
{rows !== null && (
<span className={total > 0 ? "text-indigo-400 ml-1" : "text-gray-600 ml-1"}>
({total > 0 ? `${total} awaiting review` : "none"})
</span>
)}
</span>
</button>
{open && (
<div className="mt-3 space-y-3 text-xs">
{rows === null && <p className="text-gray-600 italic font-mono">Loading…</p>}
{rows !== null && total === 0 && (
<p className="text-gray-600 italic">
Nothing proposed yet. Terms appear here when a corrected place name
turns out to be real nearby, or when the induction loop spots one in
recent traffic.
</p>
)}
{(rows ?? []).map((row) => (
<div key={row.talkgroup_id}>
<p className="text-gray-400 font-mono mb-1">
{row.talkgroup_name || `TGID ${row.talkgroup_id}`}
</p>
<ul className="space-y-1">
{row.pending.map((p) => (
<li
key={p.term}
className="flex items-start gap-2 bg-gray-900/60 border border-gray-800 rounded px-2 py-1"
>
<span className="flex-1">
<span className="text-white font-mono">{p.term}</span>
{p.meaning && <span className="text-gray-500"> — {p.meaning}</span>}
<span className="text-gray-700 ml-2">{p.source}</span>
</span>
<button
type="button"
disabled={busy === `${row.talkgroup_id}:${p.term}`}
onClick={() => act(row.talkgroup_id, p.term, true)}
className="text-emerald-400 hover:text-emerald-300 disabled:opacity-40 font-semibold"
>
Add
</button>
<button
type="button"
disabled={busy === `${row.talkgroup_id}:${p.term}`}
onClick={() => act(row.talkgroup_id, p.term, false)}
className="text-gray-600 hover:text-red-400 disabled:opacity-40 font-semibold"
>
Drop
</button>
</li>
))}
</ul>
</div>
))}
{error && <p className="text-red-400 font-mono">{error}</p>}
</div>
)}
</div>
);
}
function VocabularyPanel({ systemId }: { systemId: string }) {
const [vocab, setVocab] = useState<string[] | null>(null);
const [pending, setPending] = useState<VocabularyPendingTerm[]>([]);
@@ -1185,13 +1677,18 @@ export default function SystemsPage() {
const { systems, loading } = useSystems();
useEffect(() => {
if (!authLoading && !isAdmin && !isOperator) router.replace("/dashboard");
if (!authLoading && !isAdmin && !isOperator) router.replace("/");
}, [authLoading, isAdmin, isOperator, router]);
if (authLoading || (!isAdmin && !isOperator)) return null;
const [editing, setEditing] = useState<SystemRecord | null | "new">(null);
const [editIsDuplicate, setEditIsDuplicate] = useState(false);
// Every hook must run before this guard. React tracks hooks by call order,
// so returning early on the first render and then reaching a useState on the
// next one is error #310 ("rendered more hooks than during the previous
// render") -- which crashed this whole page to a blank client-exception
// screen the moment auth resolved.
if (authLoading || (!isAdmin && !isOperator)) return null;
async function handleDelete(id: string) {
if (!confirm("Delete this system?")) return;
await c2api.deleteSystem(id);
@@ -1286,6 +1783,8 @@ export default function SystemsPage() {
</div>
<PreferredTokenPanel systemId={s.system_id} initialTokenId={s.preferred_token_id} />
<AiFlagsPanel systemId={s.system_id} initial={(s as unknown as { ai_flags?: SystemAiFlags }).ai_flags ?? {}} />
<AreaContextPanel systemId={s.system_id} />
<TalkgroupPendingPanel systemId={s.system_id} />
<VocabularyPanel systemId={s.system_id} />
</div>
);
+1 -1
View File
@@ -26,7 +26,7 @@ export default function TokensPage() {
const [error, setError] = useState<string | null>(null);
useEffect(() => {
if (!authLoading && !isAdmin && !isOperator) router.replace("/dashboard");
if (!authLoading && !isAdmin && !isOperator) router.replace("/");
}, [authLoading, isAdmin, isOperator, router]);
const refresh = useCallback(async () => {

Some files were not shown because too many files have changed in this diff Show More