Re-adding the same system to a voice channel returns the running stream
untouched (already_on). The form closed as if a stream had been added and
the new label/bot was silently dropped; it now stays open and explains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner-reported on live: adding a radio channel for a server + voice channel
that already had one (different label, system, even bot) overwrote the
existing stream and made its bot leave, because stream_id was guild:channel.
- New streams get guild:channel:<12 hex> ids (leases.new_stream_id); old
guild:channel docs keep working for read/stop/remove/node reports. Route
validators accept both (leases.STREAM_ID_PATTERN).
- start_stream: no stream_id = ADD. Same system already on in that channel
is idempotent (already_on, label/mode untouched); a stopped one of that
system is restarted; anything else is a new doc. stream_id = explicit
restart/edit (admin Start button, reconciler), incl. moving to another bot.
- preferred_token_id is strict: bot already in this server -> 409
preferred_busy naming the stream; on another node -> preferred_other_node;
unusable -> preferred_unavailable. No silent swap. pool_exhausted now says
one-bot-per-server and to add + invite another bot.
- One-token-per-guild was already enforced inside the lease transaction;
now covered by race tests. Per-guild asyncio lock serialises the
already-on / server-cap check with the create (guild half of #223).
- Stop by stream_id (POST /discord/guilds/{id}/stop {stream_id}); admin UI
Stop/Start/Remove act on the row's stream; node picker lists every stream
in a channel and adds beside other systems.
- Bot: /leave stream: (autocomplete) stops one; /leave channel: with several
streams asks which unless all:True; bare /leave still stops the server.
/status and /join name each stream's bot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /nodes/request-update-all (admin) does what Update Node does for each
approved node: flags update_requested_at for the HTTPS updater and nudges it
over MQTT. Pending and rejected nodes are skipped. Admin-only button on the
Nodes page reports which nodes weren't reachable over MQTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DELETE /discord/guilds/{id} deletes channels one by one. If a channel was
restarted mid-removal, delete_stream raised "restarted" and the route
returned 409 with no audit entry, though the channels before it were
already deleted for good. The error now carries the channels already
removed and the route writes discord.guild.remove with outcome
"incomplete" before returning the 409. The server stays linked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- handle_actual logs every node-reported state TRANSITION (not heartbeats)
with stream/node/guild/channel/token ids and the node's error text: info
for connected/connecting/stopped, warning for error, and a dedicated
"radio bot X failed to log in on node N: <reason>" line for login_failed.
The node's reason code is now stored in actual.code and shown next to the
error on Settings > Discord and the node page.
- Remove a radio channel permanently: streams.delete_stream stops it through
desired state first (node republished without it, token -> releasing per
#216, freed on the node's ack), then deletes the doc in a transaction that
refuses if the channel was restarted in between. DELETE
/discord/streams/{id} (platform admin) and DELETE
/discord/nodes/{node}/streams/{id} (admin or org owner, same org + node
only; operators can still stop, not remove). Audited.
- Remove a server: DELETE /discord/guilds/{id} (admin) deletes every radio
channel in it the same way, then tombstones discord_guilds (org_id None +
unlinked_at, #218 semantics, plus removed_at) so it never falls back to
DISCORD_UNLINKED_GUILD_ORG_ID; the admin page hides it, and /drb link or the
admin link brings it back. Optional leave_radio_bots makes pool bots and the
server's org's own BYO bots leave via Discord REST; the DRB server bot stays
until kicked, and the UI says so. Audited.
- UI: Remove (with confirm) on Radio channels, Servers and the node page list.
Refs #213, #221
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- POST /discord/nodes/{id}/streams/stop: stream_id must be guild:channel
snowflakes, so a malformed id is a 422 instead of a Firestore path error.
- Offline footer said channels resume on reconnect; ad-hoc ones stop once
the node is gone past the lease grace (reconcile_once). Say so.
- Picker: say why Start is disabled when the node has no system.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #218 (server-26#213 item 1) unlinks a server by clearing org_id and
stamping unlinked_at, and resolve_guild_org then refuses the fallback org.
The overview showed such a server as "Unlinked · fallback org" and offered it
in Add radio channel, where start_stream would 403. Mirror the rule here
(forward-compatible: unlinked_at does not exist on main yet). Also key each
stream row by the resolved guild id, and drop panel copy that pointed at a
"Needed" figure the page doesn't show / misstated the invite permissions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The card only loads for owners/admins (the list endpoint is owner-only), so a
member saw "Loading…" forever. Show an owner-only note instead. Also assert the
one-server/two-org race really retried, as the PR body already claimed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frontend, built from design-canvas/discord-system-mockup.html ("DRB admin"):
- Settings > Discord (/settings/discord, platform admin only; the tab is
hidden from org owners because every route behind it is admin-only).
KPIs + tabs:
* Radio channels: every discord_streams doc, desired -> actual, the node's
real error, red after 2 min desired-on without connecting. Stop
(POST /discord/guilds/{id}/stop), Start for a stopped channel and
"+ Add radio channel" (POST /discord/streams), with the invite links a
"no bot in this server" refusal carries.
* Servers: linked servers plus servers seen running radio while unlinked
(acting for DISCORD_UNLINKED_GUILD_ORG_ID); Link / Change org
(PUT /discord/guilds/{id}/link).
* Radio bots: valid / check_error, guild membership with a per-server
invite link for each server the bot is missing from, lease (node,
since, releasing), fan-out, Refresh (POST /tokens/{id}/refresh),
Remove, Release all leases (POST /tokens/flush), add with optional BYO org.
* How it works.
- /tokens now renders the same radio-bot table (new token fields + refresh).
- Left out, no backend yet: alert routes, incident forum/roles/plan columns,
per-channel talkgroup chips, "Generate link code" (item 1).
Backend (routers/discord.py):
- GET /discord/admin/overview (platform admin): guilds, streams with node
and bot names, orgs, systems, limits. No token strings, no Discord user
ids, no linking-admin uid.
- PUT /discord/guilds/{id}/link: digits-only id, optional guild_name label,
and re-linking a server to a different org stops its live channels
(the reconciler would otherwise keep re-placing them on the old org's
nodes, using its stored org_id).
- POST /discord/streams: guild_id/channel_id must be digits (the web form
types them in; the bot always sent snowflakes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A server an org explicitly unlinked (unlinked_at) must not fall back to the
default org in the picker either, or the founding org could stream into it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core routers/discord_web.py:
PUT /discord/guilds/{id}/directory (service key): the server bot pushes each
server's name + voice channels; merged onto discord_guilds, never touches org_id.
GET /discord/nodes/{id}/join-options: the node org's servers (same
resolve rule as the bot), their voice channels with any live stream,
radio-bot readiness + invite links, and this node's streams. No tokens.
POST /discord/nodes/{id}/streams: stream pinned to the node; server must be
the node org's, channel must be in the bot's directory, guild + node caps.
POST /discord/nodes/{id}/streams/stop: one stream on this node, or all.
Control = platform admin, org owner, or operator owning the node.
- nodes.py: legacy raw-ID discord_join now refuses a server outside the node's org.
- discord bot: DirectorySync cog pushes on ready (15-min loop), join/leave,
rename, voice/category channel changes (debounced).
- frontend: NodeDiscordRadio panel + picker replaces DiscordJoinModal and the
admin-only Join/Leave buttons.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core: POST /discord/link-codes (org owner/admin) mints a single-use,
10-minute code; minting revokes the org's older codes. The bot redeems it
via POST /discord/guilds/{id}/link/redeem (service key) inside one
Firestore transaction, so two servers racing for one code cannot both win
and a server linked to org A refuses org B's code. GET/DELETE
/discord/linked-guilds lists/unlinks the caller's org's servers; unlink
stops that server's streams. Rate-limited (mint per user, redeem per
server and per Discord user); every mint/link/refusal/unlink audited.
- resolve_guild_org: an explicitly unlinked server no longer falls back to
DISCORD_UNLINKED_GUILD_ORG_ID. Linking away from the fallback org stops
the streams it had running for that org.
- Discord bot: /drb link code: (Manage Server, enforced at runtime too);
GatedTree answers every other command in an unlinked server with how to
link (autocomplete returns nothing). With the default fallback org set,
nothing is gated — unchanged behaviour.
- Frontend: "Linked Discord servers" card on Settings -> Organization —
generate code with countdown + copy, polls for the new link, unlink.
- 16 tests: single-use, two-server race, one-server/two-org race, expiry,
revoked/malformed codes, wrong-org takeover, idempotent relink, rate
limits, fallback stop, scoped list, unlink without fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /tokens and PUT /tokens/{id}/prefer became platform-admin only in #216
(server-26#204 item 6). Three frontend surfaces still offered them to
operators:
- systems page "Preferred Bot Token" panel: now rendered for admins only.
No non-admin listing endpoint was added: the write behind the picker is
admin-only too, so a list would only feed a control that 403s on save.
The panel also no longer turns a fetch error into "No tokens in pool."
- /tokens page: admin-only (operators are sent to /network).
- /network hub: the Bot Tokens card is shown to admins only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remote 'exit 0' came back as ssh exit 1 on the VM, and under bash -e
the step died before the SUPERSEDED check ran (runs 1139, 1144). Capture
the ssh status, check SUPERSEDED first, then fail on a non-zero status.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- reconcile_once only walked desired=on streams, so a token left "releasing"
by /leave while its node was offline (or a deleted node) never returned to
the pool until that node came back (server-26#204 leak shape). Sweep leased
tokens whose node is offline/deleted past the lease grace.
- publish_desired stamped release_seq AFTER building the list, so a stop
landing between the two got a seq whose list still held the stream; an ack
for it could free a token the node was still told to use. Stamp first.
Three regression tests; each fails on the previous commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tip check's git fetch ran before the disk-relief prune that #129 says
must be the first thing a deploy does; on a full disk the fetch could fail
and skip the prune. Prune only removes unreferenced images, so running it
on a run that then turns out superseded is harmless.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Feed/catalog error text (shown to every signed-in user via GET /cameras and
/traffic/511 status fields) carried httpx's full request URL, including
?key= for Ontario 511 and the 511NY DAA key. Query strings are now dropped.
- TfL and Calgary licences are per GEV only (TfL's terms page 403'd), the same
footing as Ontario 511, so they are UNVERIFIED (acknowledgement required)
instead of ungated attribution_required. Attribution still renders.
- The map camera layer clamps its bbox to +/-180/+/-90; zoomed-out bounds
produced a 422 ("could not reach server").
Refs server-26#198
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c2-core
- discord_streams/discord_leases wired in: POST /discord/streams,
POST /discord/guilds/{id}/stop, GET /discord/guilds/{id}/overview,
GET /discord/streams/{id}, PUT /discord/guilds/{id}/link (admin),
GET /discord/node/{node}/tokens/{id} (node key, lease holder only).
- Retained nodes/{id}/discord/desired (token ids, never token strings),
nodes/+/discord/actual subscribed and handled; per-node publish lock keeps
the retained list the newest; failed publishes retried by the reconciler.
- Tokens freed only after the node acks logout (or offline past
discord_lease_offline_release_seconds) - sweeper and C2 startup no longer
release leases.
- /nodes/{id}/command discord_join/leave map onto streams; CommandPayload
keeps preferred_token_id. /tokens: admin-only list, validate on add against
Discord, refresh, flush via transactional force release.
- Guild -> org via discord_guilds, unlinked guilds fall back to
DISCORD_UNLINKED_GUILD_ORG_ID (default "founding").
- tests/fake_firestore.py: optimistic-transaction fake; 27 lease/race tests.
discord bot
- /join, /leave, /status, autocomplete scoped to the calling guild; /leave
never touches another server; Manage Channels default permission;
/join edits its reply with the node's real result; /help copy fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two merges minutes apart ran two "Deploy to VM" jobs on the VM at once
(616/617, 652/656): racing compose pull/up, one run's prune breaking the
other's retry, health checks judging the wrong containers.
- deploy job: concurrency group drb-prod-deploy, cancel-in-progress false
(Gitea >= 1.26 honours it, go-gitea/gitea#32751; server is 1.27.3).
- Deploy step exits clean, before touching the VM, when its SHA is no
longer origin/main's tip; Health check is skipped for a superseded run.
Stops an older run landing after a newer one and rolling prod back.
- deploy-firestore-rules: its own group plus the same tip check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No AI client set a timeout: OpenAI's SDK waits 600s per attempt and retries
twice (timeouts included); google-genai waits forever. A hung provider call
had no upper bound, so recorrelation_sweep's 15-minute pipeline hold-back was
a guess against an unbounded tail.
- config: stt_timeout_seconds (120), openai_timeout_seconds (60),
openai_max_retries (1), gemini_timeout_seconds (60).
- app/internal/ai_clients.py builds every OpenAI/AsyncOpenAI client with them
(transcription, extraction, embeddings, summarizer, vocabulary learner,
trip chat). gemini.py's shared client gets HttpOptions(timeout=ms).
- A timeout raises like any other provider error, so existing fallbacks are
unchanged: rules-only correlation, rules decision on tiebreak, raw
transcript on correction, ai_health "transient".
- MIN_MINUTES_SINCE_PIPELINE_START is now derived from those timeouts
(pipeline_worst_case_seconds, 3-scene budget + 2 min non-AI slack), floor
15: 21 min with the defaults. Loop warns if it reaches the scan window.
- tests/test_ai_timeouts.py: client limits, a real hung-socket request that
gives up in <5s, timeout fallback per tier, the derived bound, and a guard
against any new OpenAI( outside ai_clients.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reject_node only set approval_status, so an approved-then-rejected node kept
its dynsec login, its /upload key and /systems read access.
- reject_node: status -> rejected first (fail-closed), then
dynsec.delete_node_client, then delete node_keys. dynsec failure -> 502,
key kept, retry is idempotent. Re-approve mints a fresh key.
- upload.py: valid key alone is not enough; node must be approved (403),
uncached read so revocation bites immediately.
- dynsec.reconcile_all: deletes rejected nodes' logins on startup, healing
nodes rejected before this change or whose revoke failed.
- Reject confirm text says it revokes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LiveView: localStorage pins carry image URLs, so they rendered (without
credit) after an admin disabled the provider. Pins now show only while
their provider is serving (server-26#198 licence gating).
- MapView: pinned/docked map players drop out once their provider leaves
the serving list; credits are kept from the last answer so docked players
keep attribution while the overlay is hidden.
- WallPanels: the "no camera provider enabled" link to /admin was a dead CTA
for non-admins (bounced to /). Link shown to admins only, deep-linked to
/admin#cameras; the admin page now honours a tab hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend (c2-core):
- app/internal/cameras/: provider interface, licence model, per-provider
catalog cache; one adapter per source in design-canvas/camera-sources.md.
Implemented: 511NY (legacy shape + tolerant XML inventory parser, 511x
endpoints UNVERIFIED), Caltrans, DelDOT, Ontario 511, DriveBC, Calgary,
TfL, NSW, Fintraffic. Stubs: NYC DOT, Westchester DPW, TRANSCOM, CTroads,
511NJ, Austin, TxDOT, MD CHART, WSDOT, Tallinn, Tarktee, Warendorf,
TrafficLand, Windy, EarthCam.
- Every provider defaults OFF (config/camera_providers). Gated licences
(agreement required / unverified / non-commercial / paid) need an explicit
acknowledgement stored with who/when and written to audit_log; the audit
write is a precondition. Stubs and prohibited sources cannot be enabled.
Ack is bound to the licence status it was given for.
- GET /cameras (signed-in), GET/PUT /admin/camera-providers (admin only).
- /traffic/511 follows the ny511 provider switch and returns the DAA credit
+ NYSDOT as-is disclaimer; 511NY event descriptions no longer truncated
("may not alter the substance of a report").
Frontend:
- LiveView becomes the ops wall: Region > Incident > Camera drill-down,
worst-first rail (MapView), KPIs, transcript ticker, scrubber incident
bands, auto-tour, deep links (#inc=&cam=&tour), Esc steps up.
- Incident panel: nearest cameras (path-weighted), fused situational context
(ADS-B, AIS, 511NY events, NEXRAD tile sample), units, call spine.
- MapView keeps every layer; adds Camera Viewsheds (approx.) overlay and a
nearest-camera layer for the selected incident.
- CameraFeed hygiene: hidden-tab/off-screen gating, 2 concurrent HLS cap,
last good frame + stale badge, frozen-frame signature, HLS stall watchdog,
provider attribution rendered on the imagery.
- Camera/event popups show source credit, update time and the disclaimer.
- Admin > Camera providers tab with licence status, health, gated confirm.
- Camera detection is not built (blocked on #199/#200); seam only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- system_config.normalize(): a P25 system with config.mode "conventional"
carries config.channels [{freq MHz, name, nac?}] (1-2, within 1.5 MHz) and
can no longer carry control channels (422). A missing mode is trunked and
passes through untouched, so there is no migration. An analog channel is
mirrored into control_channels for older nodes.
- talkgroups.name_from_system resolves the edge node's synthetic per-channel
id (freq in 100 Hz units) to the channel name.
- The checkin persists radio_channels (conventional channels + encryption).
- Systems editor: Type = P25 trunked / P25 conventional / Analog (NBFM) /
DMR, plus a channel-row editor for conventional and a one-frequency form for
analog. The node page shows conventional channels and an encryption notice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- approve_node: reuses node_keys/{id}.api_key when one exists (still
re-syncs dynsec with it, still never overwrites node_keys). Re-clicking
Approve can no longer lock a live node out. Reissue stays explicit.
- POST /nodes/{id}/request-update (admin): sets update_requested_at on the
node doc AND sends MQTT node_update best-effort, so offline/locked-out
nodes still get it.
- POST /nodes/{id}/update-check (node, inline auth: Bearer api_key for THAT
node or X-Pickup-Secret): host updater's HTTPS check-in. Returns only
update_requested_at; records updater_last_check_at / sha / applied result.
Unknown node -> 401 (no node_id oracle). C2 cannot name a ref, image or
command.
- Frontend: "Update Node Software" (admin only) calls request-update, honest
confirm text; node page shows updater check-in and last update state.
Tests: tests/test_node_provisioning.py (12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any fatal hls.js error set videoFailed permanently, so a brief 511NY feed
drop left the tile on a 60s snapshot until it was reopened. Now fatal
media/network errors get up to 3 in-place recoveries, and a tile that did
fall back retries video every 30s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tiebreaker forced ~all of the 09-27 rollover sink's bad links: idle resets
on every link so a 2.5h, 40-call incident read as fresh, and the rules treated
unit overlap as enough on a county channel where units work jobs back to back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DOT cameras can be pinned at the camera or docked in a strip over the
map; up to 6 open at once, switchable between modes.
- Below zoom 14, 511 cameras/events bucket into a 40px world-pixel grid,
one icon per cell with a count badge; click zooms in.
- Weather radar refreshed by remounting via `key`, and a remounted child of
an unchecked LayersControl.Overlay isn't re-added to the map — radar went
off every 5 min. Now refreshes in place with setUrl + cache-buster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A viewer whose first login ran self-serve signup got an empty org of
their own (org_role owner), so they saw no incidents or calls, and the
earlier fix deliberately never moved a user who already had an org.
PATCH /admin/users/{uid} now moves a user when org_id is passed
explicitly (claims + org_members, audited with left_org_id; the old org
is not deleted). The user list returns org_id/org_role, and the admin
user panel shows the org and a 'Move to my organization' button when it
isn't yours.
Verified: c2-core pytest 496 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /admin/users set a role claim but never an org_id/org_role claim or
an org_members doc. firestore.rules gates every read on the org_id claim,
so a viewer created from the Users page saw no incidents or calls.
New users now join the requested org (default: the acting admin's own,
else 'founding') as org_role 'member', with the same org_members doc
POST /auth/signup writes. PATCH /admin/users/{uid} attaches an org-less
user the same way, which heals accounts created before this fix; it never
silently moves a user who already has an org.
Verified: c2-core pytest 495 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Token colors wrapped in color-mix so opacity modifiers (bg-surface/90 etc.)
generate rules; map overlay panels were rendering see-through.
- Light mode: keep white text on saturated fills (primary/danger buttons were
remapped to navy); map text-gray-200, bg-gray-800/60, border-gray-600.
- Dark mode: lift text-gray-600/700 to gray-500 for contrast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clicking a camera's snapshot pins a 192x108 tile anchored at the camera --
the live HLS stream when the camera has one (hls.js; Safari native), else the
snapshot refreshed every 60s, falling back to the snapshot if the stream dies.
Up to 6 pins, oldest dropped. Players exist only while the DOT Cameras overlay
is on, so hiding it stops every stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/sdr-pins. NodeRecord gains sdr_pins
(service -> serial), sdr_devices and op25_sdr_serial, mirrored from the
node's checkin. PATCH /nodes/{id} validates pins (known services, one
dongle per service) and sends priority/pins as a 'set_sdr_config'
command, never a config re-push. The node restarts OP25 only when OP25's
own dongle changes. The node page's section becomes 'SDRs' with an OP25
SDR dropdown ('Automatic (first SDR)' + detected dongles) and a
per-service dongle dropdown ('Any spare SDR'), plus duplicate-serial and
double-pin warnings.
Verified: c2-core pytest 490 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /traffic/511 serves a bbox slice of the statewide 511NY camera and event
feeds from an in-memory cache (cameras 1h, events 2m TTL, fetched lazily) --
public data, so no Firestore writes. A failed refresh keeps the last good data
and reports the error; a schema change (nothing parses) is an error, not an
empty layer. Frontend: opt-in "DOT Cameras" and "Traffic Events" overlays that
fetch only while shown, with an on-map notice when the feed is down or stale.
NY511_API_KEY is optional in config (the API answers without one today; the
terms require a registered key).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QA (drb-qa-review) blockers: sdr_count defaulted to 1 for nodes that
never sent it, so the panel claimed 'reports 1 SDR (0 spare)'. It is now
None until reported, and the count is only quoted alongside a real
secondary_sdr_running report. Rows read 'Not reported' instead of
'Waiting for SDR' when the node hasn't said what's running (closes#187).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/secondary-sdr-priority. NodeRecord gains
secondary_sdr_priority (ordered; SDRs beyond OP25's run it top-down) and
secondary_sdr_running, both mirrored from the node's checkin.
PATCH /nodes/{id} accepts the priority, validates it, and sends it as a
'set_secondary_priority' MQTT command. A priority-only change never
re-pushes system config, because that restarts OP25. The node detail page
gets a 'Secondary SDRs' section (admin-editable) with enable, reorder,
save, and live Running / Waiting-for-SDR state from the checkin.
Verified: c2-core pytest 482 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QA follow-ups on the ADS-B/AIS overlays:
- Legend gains an aircraft altitude key (tar1090 ramp) while the Aircraft
overlay is on (closes#185).
- AIS aids to navigation (MMSI 99xxxxxxx) render as small yellow buoy
diamonds labelled 'Aid to navigation', no speed; vessels get a 22px
outlined hull, and heading 511/360 ('not available') no longer
rotates the icon (closes#186).
- Aircraft details panel becomes a bottom sheet above the incident drawer
below md, instead of fighting the layers control at the top right.
Verified: tsc --noEmit clean (node:20 on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The popup sat on top of the plane and hid the trail it had just drawn.
Details now render in a panel docked top-right (portaled into the
Leaflet container, click/scroll propagation disabled), so the map can
be panned to follow the path. Clicking empty map or the same plane
deselects; hover tooltip unchanged.
Verified: tsc --noEmit clean (node:20 on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Icons were 16px accent-colored glyphs, indistinguishable from OSM's own
airport symbols. Now a 30px outlined airliner silhouette filled on
tar1090/ADS-B Exchange's altitude hue ramp, with a callsign/altitude
hover tooltip; the selected aircraft grows and gets a white outline.
Clicking an aircraft draws the path heard so far, segment-colored by
altitude. c2-core writes one point per position change to
aircraft/{icao}/positions (deduped in-process, writes now concurrent);
points carry expire_at and a TTL fieldOverride deletes them after ~24h.
Trail reads are gated on the parent aircraft doc's org via get(), so the
query needs no org filter or composite index. The latest stretch without
a 20-min gap counts as the current flight.
Verified: c2-core pytest 479 passed; frontend tsc --noEmit clean (node:20
container on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy-firestore-rules job has failed on every push because
FIREBASE_TOKEN was never set, so rule changes (e.g. aircraft/vessels for
node-26#9) never reached prod. Switch to a dedicated least-privilege
service account whose JSON key lives in FIREBASE_SA_KEY; login:ci tokens
are deprecated and carry their minter's full access. Key is written to
RUNNER_TEMP at 0600 and removed on exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 09-22 replay stop was titled "Traffic Stop at 96 times 5" — a disposition
code read aloud, extracted as the location (server-26#170). clean_location
now rejects "N times N", ten-codes, "signal N", "code N", "condition N".
c2-core: 476 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner: plate reads should become traffic stops. Held-out replay of 09-21
(server-26#170): Ossining's Post 4 stops were read out only as plates
("Frank David Boy, 4514", "Lincoln, Charlie, Robert, 7-4-0-7") and never
became incidents.
The self-initiated backstop now treats two+ phonetic letters followed by
3-7 digits as a stop — but only when extraction found no other event in
the call (a plate on an MVA, tow or parked-car complaint stays with that
event), and never on MTA/rail/bridge/fire/EMS/DPW talkgroups.
c2-core: 475 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A day of replay runs (server-26#170) spent ~$5 of Gemini on ~7 two-hour
windows (~$0.70 per 290 calls) — several dollars a day per live deployment
for correlation alone — and nothing could say where it went (#45). Gemini
3.x thinks by default and bills it as output; the deprecated
google-generativeai SDK these calls used cannot set a thinking level.
- app/internal/gemini.py: every Gemini call (correlation + transcript
correction) goes through google-genai with JSON mode, an explicit
thinking level, and logs in/out/thinking tokens. A model that rejects
the level is retried without it once and remembered, so the tier is
never lost to a config param. API failures still raise for ai_health.
- correlator: thinking_level "minimal" (a link/new/orphan choice).
transcript correction: "low" until a replay shows minimal is safe.
- replay: runs record real Gemini token usage (metrics.gemini_usage),
shown in the Replay tab.
- requirements: google-genai.
c2-core: 474 pass. Frontend typecheck not run (no Node on this box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay of 433b35d (server-26#170): traffic stops now open incidents
("45 Adam" stop, "CM2" stop), but the bridge MVA split 33/56 — one
transmission ("transport complete") was read as scene-resolved at 14:44
and an LLM closure was final, so the rest of the MVA opened a new one.
- LLM closure is now provisional (reopenable), like a timer close: it is
inferred from a single transmission.
- review of 433b35d: backstop only matches a unit's own "on a stop"
self-report (bare "car stop" mentions and "pull over" dropped), never on
MTA/rail/bridge/fire/EMS/DPW talkgroups ("Train 4 holding on the stop"),
negation looks 5 words back, and <=5-word reports ("Adam 3 on a stop")
get a minimal scene instead of being skipped before the backstop.
c2-core: 471 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner: traffic stops should show on the portal — otherwise they are only
visible in the archive. In the 09-22 replay (server-26#170) every Ch 1
stop ("45 Adam on a stop, Eastbound Central Express", "CM2 on the stop,
southbound") came back from extraction untyped, untagged and routine —
read as status traffic after #138 — so the creation gate never opened one.
- prompt: a unit reporting its own activity (on a stop, out with a vehicle
or pedestrian) is a real event: police, tagged, at least minor; the plate
lookups for it belong to it.
- deterministic backstop after extraction: stop / "put me out with"
phrasing adds a "traffic-stop" / "self-initiated" tag (the substance the
creation gate counts), police type if none, minor if routine. Negated
phrasing ("not pull the car over") is left alone; nothing is downgraded.
c2-core: 469 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- only a substantive call after the close reopens a timer-closed incident;
a thin "10-4" (doesn't refresh updated_at) or a sweep link of a call from
before the close rides along without reopening — otherwise the next
sweep closed it again and the portal flickered.
- substantive_call_count counts a call once, not once per scene.
- a timer-closed incident is never adopted as a cross-system parent.
Replay e972cac (correlation-only on 731b54b's extraction) vs the 09-22
answer key: pairwise F1 0.548 -> 0.807 (precision 0.839 -> 0.956, recall
0.407 -> 0.698); the bridge MVA is one 76-call incident instead of two
40-call halves. server-26#170.
c2-core: 468 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hand-labelling the 09-22 10:00-12:00 ET replay window (server-26#170,
answer key replay_groundtruth_0922.json) found ~25 real incidents, of which
only ~5 had an audible clear — most jobs clear by MDT, so the quiet timer is
the close for most incidents and a flat 90 minutes left a lockout or a plate
check "active" on the portal an hour after it ended.
- summarizer: timer close after 30 min quiet for routine/minor, 60 moderate,
90 major/unknown. A timer close is provisional: reopenable=True.
- correlator: reopenable incidents inside incident_reopen_window_minutes
(90, since last substantive call) stay candidates; linking a call to one
reopens it (status active, reopened_count++). The sweep expires the flag
so the reopenable pool stays bounded. Real clears (units_cleared,
llm_closure) are never reopenable.
- cap: incident_max_calls counts substantive calls only
(substantive_call_count). The bridge MVA hit 40 in 32 min with ~40% thin
replies, split in half, and the second half took another job's title.
c2-core: 467 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first clearance fix found it pushes toward early resolve:
- parser cleared units on questions ("45-9, are you clear?"), negations
("not clear yet"), orders ("clear the scene", "clear to transport"),
places/times ("Room 2 clear", "1400 hours, clear") and split "45 9" into
unit 45. Now rejects "?", not/is/are/you, anything after the status word
but sign-offs, place/time words; joins "45 9" -> "45-9".
- a clear from a unit never active on an incident was recorded in
units_cleared and could pass the all-clear gate. Only units actually
active there can clear there now.
- clearance-only calls skip the LLM tier (same as thin calls): only the
rules engine's unit match can say which incident a 10-8 belongs to.
- replay incident view carries srcaddr/srcaddrs for the radio-ID clearance
investigation.
Replay 09-22 10:00-12:00 ET with f0a88d4: real clears 0 -> 2 (both LLM
closure), unit clears still 0 — the parsed clears are right but those
units were never recorded as assigned (Whisper mangles unit IDs at
dispatch), which this commit does not fix.
c2-core: 465 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay of 09-22 10:00-12:00 ET (server-26#170): 0 of 19 incidents resolved
on a clear, 19 on the idle timer, although 25 transmissions said 10-8/clear.
Three independent breaks:
1. Short clears never reached extraction. "45-9, I'm clear." is <=5 words,
so extract_scenes skipped it before GPT and cleared_units stayed empty.
A rule parser now names the unit when it precedes the status word
(never guesses: "10-8, 10-8." / "CMT clear." clear nobody) and returns a
minimal scene that links by unit overlap but cannot open an incident.
2. Clearance compared unit strings exactly, so "11-Adam" clearing never
removed "11 Adam". Now by _normalize_unit key.
3. units_active collected "Desk", "Central", "Division", "unknown", plate
phonetics — none of which ever clear, so all-clear could never pass.
Only units carrying a number (and not a ten-code) are tracked now; the
rest stay in `units` for matching.
c2-core: 463 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first replay (server-26#170) aborted on 5x 'model is unavailable'
from the tiebreak tier. Google's model list shows 2.5 closed to new
projects and no stable Pro model; 3.8-flash is the newest stable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First replay (290 calls, 09-22 10:00-12:00 ET) produced 0 incidents and
no errors: every gpt-4o-mini extraction failed and _sync_extract
swallowed it as "no scenes". Same shape as #169 — and the live extraction
tier in /health/ai had no reporter at all, so this has been invisible in
production too.
- intelligence: API failures propagate out of _sync_extract; extract_scenes
reports them to ai_health ("extraction" tier, billing/dead-model
classified) and still returns [] so the pipeline degrades as before.
- ai_health: inside a replay sandbox, failures go to the run's own sink
instead of being dropped.
- replay: aborts after 5 permanent failures on a tier, naming the cause;
run metrics carry ai_failures; UI shows them.
- replay estimate: audio minutes from started_at/ended_at (no duration
field exists on call docs).
- ReplayTab exposes the loaded run on window.__drbReplay for in-page
analysis.
c2-core: 458 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Correlation has only ever been measured through live AI windows: days of
wall time per change, and the 09-20→22 window was invalidated outright by
unfunded AI accounts (#169). Recordings are kept regardless of AI, so the
traffic to measure against already exists.
- internal/replay.py: runs a time range of real calls through the live
pipeline code in original order, clock pinned per call, into
replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe),
transcripts (re-extract), reuse (correlation only from a prior run's
scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on
virtual time. No alerts, summaries, vocab, AI-health alerts or pending
terms. One run at a time, <=5000 calls, <=7 days.
- firestore.py: ContextVar sandbox redirect for calls/incidents.
- clock.py: ContextVar-pinnable now(), used on the correlation path.
- feature_flags.py: ContextVar flag override so replay runs with live AI off.
- upload.py: scene loop extracted to _extract_and_correlate, shared by the
live pipeline and replay so replay measures the code that runs live.
- resolved_via on every incident resolve, so a real clear can be told
from the idle timeout — live and in replay.
- routers/replay.py + /admin Replay tab: estimate, start, compare runs,
drill into incidents with audio.
Reviewed by drb-correlation-review; its leak and fidelity findings are
fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run
(no Node on the authoring box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incident started_at is an isoformat() string (incident_correlator.py,
routers/incidents.py), not a Firestore timestamp like calls. The range
bounds were Dates, which Firestore compares by type, so any date range
returned zero incidents. Bounds are now UTC ISO strings in the same
"+00:00" shape, which order lexicographically by time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/calls/search scans a 200-row window and returns 50, but the next cursor
was always the last SCANNED row — so with an unfiltered list each page
jumped past the 150 matches it had already read and not shown. Resume
after the last RETURNED row when matches overflow the page; keep the
last-scanned cursor only when the page holds every match (the sparse-
filter case that cursor exists for). Same fix for /calls/eval-queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incidents and Archive get a from/to date range (native date inputs,
local-day bounds). Incidents filters in the Firestore query; Archive
passes date_from/date_to to GET /calls/search, which applies them as a
started_at range — both ride the existing org_id/started_at index.
Also fixes /calls/search and /calls/eval-queue paging: the cursor went to
Firestore as a raw ISO string against a timestamp field, which compares
by type rather than time, so "Load more" re-read the first page. Cursor
and range bounds are now parsed to datetimes (400 on garbage).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incidents page: text search (title, location, summary, units, vehicles,
tags, location mentions), status and type filters, and a Load more button
that pages the Firestore query 100 at a time. Filtering runs over the
loaded window, and the page says so when older incidents exist.
Archive (/calls): readable by every org member, not just admins.
GET /calls/search now takes any Firebase token scoped to the caller's org
— the Firestore rules already let members read every call in their org,
so this widens nothing. Attach/detach stays admin-only (UI and routes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.
GET /calls/eval-queue -- calls with a transcript but no
eval_transcript yet, paged (same bounded-
window-plus-cursor shape as /search)
PUT /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
never touches transcript/transcript_corrected
GET /calls/eval-stats -- eval_count + average word error rate of
the raw and corrected machine transcripts
against the human-verified ones
internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.
Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.
Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add
information", but nothing checked the model's output against its own
rules -- raw["corrected"] was accepted verbatim past a non-empty/changed
check, and the "changed" field it returns was logged for audit and never
validated.
Caught live: the same call's raw vs corrected transcript showed "10-7"
rewritten to "10-13" in one place and "10-4" in another, plus a bare "7"
expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes
Street and 4th and Rowe -> Home Street and Forest Ave, from this system's
own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly
as trustworthy as a correct transcript, which is worse than leaving the
raw mishearing in place.
_code_tokens() extracts every ten-code/signal-shaped token from the
original and corrected text/segments; any change to that set discards
the correction and falls back to raw. Checked independently for the
joined text and for segments, consistent with the existing
all-or-nothing segment-alignment rule.
Does not catch a wrong word swapped for another equally plausible
non-code word -- that class still depends entirely on the model
following its own prompt. Filed server-26#161 for the broader STT/audio
quality initiative this belongs alongside.
Verified: 426 pass, 0 fail (4 new tests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
cat file 2>/dev/null || echo latest only falls back when cat itself fails
(nonzero exit, i.e. the file is missing) -- a file that EXISTS but is EMPTY
makes cat succeed with empty output, so PREV_TAG became "" instead of
"latest". That "" failed the emptiness check further down and exited 1 --
AFTER git pull + docker compose up -d had already succeeded. Worse: exiting
there skips the Health check step (Gitea Actions doesn't run later steps
after a failure), and Health check is the ONLY step that ever writes a real
value to .last_good_tag. Self-perpetuating: once the file went empty, every
future deploy failed the same way forever, with the app itself deploying
fine underneath it every time (confirmed against run 611: deploy log shows
all three images pulled/recreated/started at 66bbf5b, and /health
self-reports that exact GIT_SHA, baked into the image at build time --
two independent signals, same commit).
${VAR:-default} covers empty and unset in one expansion, matching the
fallback behavior the surrounding comment already documented as intended.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_geocode_location's node-distance fallback (geocode_max_km, 40km) is a
proxy for "is this plausible" that only makes sense when the node's own
position is the best guess we have at the area. It doesn't hold for a
citywide/patched feed: node-002 sits in Westchester but relays "New York
City - NYPD Citywide 2 Patch", ~56km from the addresses on it. Every real,
correctly-geocoded address on that talkgroup was rejected by this check,
every time -- location_coords stayed permanently null for the whole
system, which killed the location_proximity correlation signal and let
duplicate incidents form for the same event reported at two nearby
addresses two minutes apart ("Shots Fired at Jackson Avenue" /
"Shots Fired at 1108 Jackson Avenue", 2026-09-20 ~23:00 UTC, merged by
hand via the Archive page's attach/detach while this fix went in).
trust_named_region skips the node-distance rejection exactly when the
query already carries a place name that isn't the node's own position --
operator-set area_context, or a municipality parsed from the talkgroup's
own name. The anchor path is untouched; an anchor's own radius is always
authoritative when one has been resolved.
Also fixes a compounding defect found while verifying: the query was
grafting the node's own COUNTY onto an already-self-named region
("...New York City..., Westchester, New York"), which is self-contradictory
and could degrade the geocode independent of the distance check. The
node's county is now used only when nothing else names the place; state
stays in both branches since it's coarse enough to be correct either way.
Extracted the query-assembly logic into a pure, unit-tested helper
(_location_query_parts) rather than testing it only through the full
extraction pipeline.
Filed #160 as a follow-up: place_verifier.py's verify() has the identical
no-anchor gap for transcript place-name correction, not fixed here.
Verified: 422 pass, 0 fail (9 new tests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The fast/thin path (short acks like "10-4", enabled to attempt linking at
all by 1ffff25) had no identity signal available -- transcript_too_short
skips GPT extraction entirely, so call_units is always empty for this
population. It fell back to "most recently updated incident on this
talkgroup", which silently misattaches a short ack to the wrong incident
whenever two are live on the same busy dispatch channel at once.
metadata_watcher.py already captures the P25 source radio ID (srcaddr) on
every call independent of transcript content, and it already reaches the
call doc (models.py, mqtt_handler.py:245) -- it was just never read by the
correlator. Thread it through _build_context, check it against the
srcaddrs already seen on each TG-matched incident before falling back to
recency, and accumulate it on the incident (_update_incident/_create_incident)
so later calls from the same radio can match.
Verified: 422 pass, 0 fail (4 new tests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The rules/indexes deploy step already existed here, gated on `command -v
firebase` over SSH on the deploy VM. It never found one (no node on the
VM), so it silently warned-and-skipped on every single deploy for weeks —
PR #124 even auto-closed #13/#51 as if this were fixed, when it wasn't.
New standalone deploy-firestore-rules job runs on the Gitea runner itself
(always has node), authenticated via a new FIREBASE_TOKEN secret (from
`firebase login:ci`) instead of anything pre-installed on the VM. It's
independent of the deploy job's health-check/rollback chain on purpose —
a rules deploy failure has nothing to roll back and must not trigger that
logic. notify-failure now distinguishes which job actually failed so the
Discord alert doesn't misreport "production is unchanged" when the app
deployed fine and only the rules push failed.
Needs FIREBASE_TOKEN added as a Gitea Actions secret before this actually
runs — it will fail loudly (by design) until then, which is the whole
point: a loud failure beats a silent skip.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
intelligence.py skips GPT extraction for transcripts <=5 words ("10-8", "show
me clear", a unit check-in) -- real cost/hallucination guard, kept as-is. But
upload.py's no-scenes correlation fallback (the path that thin-links a call
by talkgroup even with zero extracted content) excluded ANY skip_reason,
including transcript_too_short -- so this exact population, brief but real
follow-up and clearance traffic, never even attempted to attach to anything.
Found live: "Live, Ossining." sitting an orphan 6 seconds before a real
incident's founding call, on the same talkgroup.
Now only garbage_transcript (Whisper hallucination, no real content) stays
excluded; transcript_too_short reaches the same thin/fast-path fallback
already trusted for no-transcript calls, gated the same way -- same-talkgroup,
recently-active incident required before anything attaches. No GPT re-invoked,
no new cost.
Verified: 410 pass, 0 fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-26#9, same shape as the ADS-B commit. Adds POST /telemetry/ais
(same node-key auth, same org_id-stamped upsert-by-key pattern, this time
by mmsi into a new `vessels` collection) and its docInMyOrg() firestore
rule. Frontend gets useVessels() (mirrors useAircraft(), longer staleness
window since AIS position reports are minutes apart, not seconds) and an
opt-in "Vessels" map overlay.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-26#9. Adds POST /telemetry/adsb (node-key authed via
require_node_service_or_firebase_token) that upserts one Firestore doc per
icao into a new `aircraft` collection, org_id stamped from the reporting
node the same way upload.py defensively stamps `calls`. firestore.rules
gets a matching docInMyOrg()-gated read rule.
Frontend: useAircraft() mirrors useNodes()'s onSnapshot pattern, filtering
docs older than 2 minutes client-side since nothing prunes a stale aircraft
doc server-side yet. MapView gets an opt-in "Aircraft" overlay (unchecked
by default, like the weather radar layer) rendering a rotated plane glyph
per sighting.
Unverified via typecheck — no Node.js/npm on this authoring box yet (see
CLAUDE.md Testing reality). Server side is pytest-covered (test_telemetry.py).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Server-side half of node-26#9. NodeRecord gains secondary_sdr_mode
(none|adsb|ais|op25_2) and sdr_count; checkin ingestion stores both,
and PATCH /nodes/{id} accepts and re-pushes secondary_sdr_mode the same
way hardware_preset/ppm_override already work, so it isn't wiped by a
system reassignment (see server-26#111 for the pre-existing bug that
pattern avoids repeating).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EXTRACTION_PROMPT's incident_type rule always returned "police" for anything
on a police channel unless contradicted, so pure administrative chatter
(post check-ins, roll call, bare acknowledgements) got a truthy incident_type
and defeated the creation gate's "type" veto (_call_is_substanceless) ~82%
of the time (measured server-26#138/CORRELATION_REVIEW_0914.md, confirmed
against live 9-14 data: 9/9 type-veto examples checked all had severity
"routine" — same population the severity rubric already correctly
identifies as content-free, incident_type just wasn't using that signal).
Rule now checks for actual event content FIRST, on every channel, before
applying the channel-inference defaults; content-free traffic returns
"unknown" (already normalizes to None) instead of a channel default,
letting the existing gate correctly veto it. No correlator/gate code
touched — CORRELATION_REVIEW_0914.md's own recommendation was to fix this
in the prompt, not _call_is_substanceless, to avoid risking a real event
getting gated out.
Verified: 401 pass, 0 fail. Live effect to be confirmed against fresh
traffic (AI features just re-enabled this session after being off since
9/14) — tracking in SESSION_STATE.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent fixes, found by tracing why incidents never actually close
(only 2/281 incidents in 5 correlation debug windows ever got a non-empty
units_cleared; 183 resolved via the 90-min idle sweep instead of a real clear):
1. Pattern B (re-dispatch accept, no explicit 10-8): reassignment=True already
fired correctly and suppressed the unit from re-linking to its prior
incident, but nothing ever released the unit FROM that incident — it just
sat "active" until the idle sweep timed it out. _release_reassigned_units
now scans other active incidents for unit overlap on a reassignment and
clears the unit there, reusing the same units_active/units_cleared merge
(factored out as _apply_unit_clearance) that explicit 10-8 extraction uses.
2. Pattern A (self-clear) extraction was inconsistent for two reasons: no
per-system unit ID format awareness anywhere in the pipeline (formats vary
by department with zero shared convention), and the cleared_units prompt
rule only accepted a unit self-reporting, missing dispatch confirming a
unit's status back to them. Added system.unit_format_hint (owner-authored
free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed
into the extraction prompt, and broadened the cleared_units rule while
still requiring an identifiable unit ID (guards against bare "10-8"/"clear"
noise, including Whisper hallucination runs already caught upstream by
_is_garbage_transcript).
Verified: 401 pass, 0 fail (local Linux venv ~/venvs/drb-5c — see CLAUDE.md
testing-reality note).
server-26#pending — not yet filed, Gitea unreachable from this sandbox.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.
Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.
15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.
recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.
Sandboxed pytest: 381 -> 387.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
- Document call_severity's routine-coercion asymmetry with incident_type
(extraction-said-routine vs extraction-said-nothing look identical).
- Test docstring no longer overclaims the ctx-linkage it doesn't cover;
points to the tests that do (test_consensus_gate.py, test_incident_identity.py).
- scene1 now uses a distinct severity so the test actually exercises both
fields symmetrically; the 'no flat top-level clobber' claim is now
asserted, not just commented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
_call_is_substanceless's "type" veto reads ctx["incident_type"] at decision
time, but that value was never persisted per-scene — only the last-scene-wins
flat field, which #138's window-4 dump analysis couldn't tell apart from
cross-scene contamination without re-guessing from a live dump. Adds
incident_type/severity to _apply_and_log's per-scene write and to admin.py's
_scene_summary allowlist (the debug-dump reader has its own field allowlist,
separate from the write side — silently would not have surfaced otherwise).
Sandboxed pytest: 380 -> 381.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Full removal, not a hardcoded flag: _is_dispatch_channel, _DISPATCH_TG_RE, the is_dispatch parameter, and _call_fits_incident's tactical branch are gone. One evaluation path for every channel.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
is_dispatch was computed from _is_dispatch_channel(talkgroup_name) and picked between two _call_fits_incident evaluation orders: dispatch (requires a positive signal, runs location-conflict/content-divergence vetoes on unit overlap) vs tactical (skips both vetoes, defaults to True on no signal at all within 20 min). Per #133's reasoning, a name not literally containing dispatch/patched/primary got the unvetoed, default-True path solely because of its label. Hardcoded is_dispatch=True at its one real call site; the tactical branch and its own tests stay in place, unreached.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Owner correction from direct scanning experience: a talkgroup named tac/tactical only sees materially different traffic during a real incident, and that's rare -- the bulk of traffic on any monitored channel, including high-risk stops and pursuits, runs on the main channel regardless of what it's named. _is_dispatch_channel's string match on the talkgroup name is a naming-convention guess, not a detector of actual channel behavior; trusting it here meant a busy single-channel department not literally named 'dispatch' would silently get the more permissive 15-minute window and could reproduce #115's original bug (the gate never firing on the channels it targets).
Always use tg_dispatch_thin_idle_minutes (5 min) in the escape hatch, regardless of talkgroup name. Does NOT touch incident_correlator.py's own fast/thin idle-window selection, which uses the same dichotomy for a different, decision-changing purpose -- bigger blast radius, left for its own review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident.
Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).
Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
- the existing flat corr_* fields, unchanged (last-scene-wins, the safe
backward-compatible default for any reader that doesn't know about
`scenes` yet)
- a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
corr_debug}, via doc_set(..., merge=True). Firestore's
DocumentReference.set(data, merge=True) recursively merges nested map
fields by key (documented SDK behaviour, not assumed) — a write to
scenes.1 merges alongside an existing scenes.0 instead of replacing the
whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.
admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.
summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.
Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.
Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.
Fixes server-26#96, server-26#114
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
The 2026-09-12 deploy of #126 failed instantly -- git pull on the VM hit 'No space left on device' before the deploy script could even capture a rollback target. Every deploy leaves 3 freshly SHA-tagged images that only got cleaned up by a prune gated on a failed compose pull; a failure earlier than that (like this one) never reached it. 96 of 100 local images were unreferenced, 23.76GB reclaimable, disk at 100%.
Move an unconditional docker image prune -af to the top of the deploy, before git pull, and upgrade the post-up -d prune from -f (dangling only) to -af (all unused) so stale tagged images stop re-accumulating. Both are safe: prune -a never touches an image a running container references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Three live measurement windows and two consensus-layer fixes (#125, #126)
converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop
iterating the correlator's consensus layer, the actual lever is upstream —
a classifier in scene extraction that recognizes radio housekeeping (roll
call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever
becomes a scene for the correlator to judge.
Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript)
function recognizing two shapes drawn from hand-labeled examples in the
review docs, cross-referenced against the real dumps — not invented regexes.
Deliberately conservative: anything that doesn't cleanly reduce to a known
shape returns (False, None) and the existing pipeline runs unchanged.
SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to
the existing _is_garbage_transcript / transcript_too_short gates and writes
chatter_classifier_verdict / chatter_classifier_reason onto the call doc,
but does not skip extraction. admin.py's correlation-debug _call_summary
surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the
next live window can measure the real-world false-positive rate before
anything is wired to actually skip extraction. TODO(server-26#127) marks
the call site.
Backtest against all three existing dumps (1002 calls): 154 flagged, 0
false positives (no flagged call carries tags, coords, non-routine severity,
or matches any review-doc-named dangerous-to-drop transcript — the major
extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station
subject check, the property-retrieval call, all individually verified).
tests/test_chatter_classifier.py: real transcripts from the dumps/review
docs in both directions. Sandboxed pytest 332 -> 364, green.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
corr_gate_veto was written to corr_debug but the admin endpoint's whitelist (_call_summary + the summary tally) never surfaced it, so the last commit's whole point -- measuring window #4 instead of guessing -- would have produced nothing to read. Add it to both. Also softened the docstring's remaining overclaim: whether the active-only ctx[recent] limitation explains the 2/24 window-3 misses is unanswered, not confirmed -- read corr_gate_veto next window instead of asserting a guess again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Review of #126 found: (1) the escape hatch applied tg_dispatch_thin_idle_minutes (5 min) unconditionally, but incident_correlator's own fast/thin path only uses that on dispatch channels and 15 min on tactical ones via _is_dispatch_channel -- mirrored the same selection here, plus a config.py note flagging the second consumer. (2) the docstring claimed a 'confirmed explanation' for 2 window-3 gate misses that was actually wrong (self-contradictory in its own text); replaced the guess with corr_gate_veto, written into corr_debug on every llm=orphan/rules=new disagreement that escalates, so window #4 can see *why* each one escaped instead of reconstructing it from the raw dump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
_recent_incident_on_same_talkgroup previously treated ANY same-talkgroup
incident within the 2-hour correlation_window_hours lookback as 'recent',
which disabled the whole LLM-orphan consensus gate on busy dispatch
channels: window #3 (CORRELATION_REVIEW_0912.md) measured 0/24 gate fires
against the exact target shape (rules=new, llm=orphan, tiebreak=new), with
22/24 explained by a same-talkgroup incident existing somewhere in the
prior 2h — nearly guaranteed on channels producing 3-13 incidents/2h.
Now the escape hatch only counts an incident as recent within
settings.tg_dispatch_thin_idle_minutes (5 min), reusing the same recency
bound the fast/thin path already uses for the 'dispatch, thin ack 10-30s
later' case this hatch exists for, instead of inventing a new constant.
Investigated the 2 unexplained misses (no same-tg incident found even by
a naive full-collection timestamp scan): confirmed ctx["recent"] is built
from status=="active" incidents with over-capacity incidents dropped
(_build_context / _drop_capped), not a full collection scan — an incident
that has auto-resolved or hit incident_max_calls/incident_max_duration
within the window is invisible to this check even though it is
chronologically recent. This does not explain the 2 misses (a same-tg
incident was absent by both checks there, so some other
_call_is_substanceless condition must be responsible), but it is a real
gap in the check as written. Documented in the docstring with a
TODO(server-26#115); fixing it needs a new, non-active-filtered Firestore
query, out of scope for this pass.
Tests: added a regression test proving an incident inside the old 2h
window but outside the new 5-minute window now correctly gates (fails on
main, passes here), plus a test proving a truly recent (<5min) same-tg
incident still escapes the gate as intended. Sandboxed pytest: 327 -> 329
passed (2 new tests), all green.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
_call_is_substanceless mirrored has_event_substance but not the creation gate's type-resolved short-circuit, so a routine-severity fire/medical call with no coords/tags/vehicles — or a reassignment (unit pulled to a new job) — could be gated to orphan where rules would open an incident. Bail out of the gate on incident_type or reassignment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
The gate added in ca1d8fb checked rules_decision["corr_debug"] for a positive
signal, but that dict is empty at preview time for action=="new" (corr_path is
written at apply time). The check was always False, so the gate fired on real
events — replayed against corr_dump_9-7_pm.json it dropped ~36 linked calls
including a major "extinguishing fire", a moderate fire-alarm, geocoded calls
and pursuit updates.
Gate now runs against ctx (fully populated at preview time). It fires ONLY when
the call is substanceless: routine severity, no vehicle/geocode/tag, and no
incident already running on the same talkgroup. Any of those escalates to the
tiebreak instead. The substance predicate (has_event_substance) is factored out
of incident_correlator's creation gate and shared, so the two cannot diverge.
recorrelation_sweep: a call the gate parked gets a longer link-only retry budget
(10 vs 3) — the gate fires before any incident for the job exists, so the
substantive call that justifies linking can land after the standard ~6 min.
Still create_if_new=False.
incident_correlator location path: evaluate every in-radius candidate and link
the nearest that carries corroboration, instead of the first in an unsorted
`recent`. A unit-overlap location link is now tagged "location_unit_overlap" so
it stops merging into the fast path's bucket in the admin fit-signal histogram.
tests/test_consensus_gate.py: replaced the corr_debug-signal cases with ctx
substance cases (severity, coords, tags, vehicles, same-tg incident); added a
nearest-wins location test; the two location guard tests now assert they reach
the new guard. Full drb-c2-core suite 322 -> 325.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
CORRELATION_REVIEW_0907b.md measured that radio housekeeping (unit
check-ins, roll call, 10-8/10-98 clearings) is being promoted to
incidents. Every case reads corr_llm_action=orphan, corr_rules_action=new,
corr_consensus=tiebreak -> new: the cheap LLM correctly reads "not an
incident", the rules engine says `new` only because there is no incident to
link to, and the smart tiebreaker then sides with rules ~21/21. Reframing
the tiebreaker prompt (#116) did nothing. The fix is a consensus-logic gate,
not another prompt.
Fix 1 (routers/upload.py) - LLM-orphan gate in _correlate_with_consensus:
when the cheap LLM says `orphan` and the rules engine says `new` with NO
positive event signal, resolve to `orphan` and skip the tiebreak call
entirely. "No positive signal" = the rules corr_debug carries neither a
positive corr_path (unit-continuity / location / fast/disambig / fast/single)
nor a positive corr_fit_signal (unit_overlap / location_proximity). When it
does carry one, the existing escalation-to-tiebreak is kept so a genuine
event the LLM misreads as orphan still gets the second look. The resolved
outcome records corr_consensus="llm_orphan_gate" (greppable, distinct from
"tiebreak") and keeps corr_llm_reasoning / corr_rules_action /
corr_llm_action populated.
Fix 2 (incident_correlator.py) - tighten corr_path=location: the location
path linked on a bare sub-location_proximity_km (0.5 km) distance with no
unit or content check, which stitched a vehicle lockout to a station-restroom
slip and merged two different churches an hour apart. A location link now
requires unit overlap with the candidate OR a distance under a tighter bar
(_LOCATION_TIGHT_PROXIMITY_KM = 0.2 km). Pursuit incidents keep their
movement-speed-validated wide radius. A surviving location link now also
writes corr_fit_signal (unit_overlap | location_proximity), consistent with
Fix 1's positive-signal set.
Tests: new tests/test_consensus_gate.py (13 cases) - the gate resolves to
orphan without calling tiebreak on a no-signal disagreement; a unit_overlap /
location_proximity / unit-continuity / fast-disambig rules signal still
escalates; llm=link vs rules=new still escalates; the location path drops a
shared-area candidate with neither unit overlap nor tight proximity, links on
unit overlap, and links on tight proximity alone. Full c2-core suite
309 -> 322 passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
