The picker footer reads "Permissions checked N ago" from checked_at, but
voice_check stamped it with the response time. A denied result is cached
for 10 min, so a user who fixed the permission in Discord and reopened the
picker saw "checked 2s ago · Needs fix" from a stale snapshot. checked_at
now carries the oldest fetched_at behind the answer (now when there is none).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Node-page picker and the admin Add form now show, per voice channel, whether
a radio bot can stream there, from S3's voice-check routes:
- Badge per channel: Ready / Needs fix (names the missing permission) /
Couldn't check / Bots busy / No radio bot / Not a voice channel / Limit N.
- The chosen channel shows "<bot> can't Speak in #dispatch: <what removed it>"
plus c2-core's fix sentence, and an expandable per-bot list.
- Footer: "Permissions checked 40s ago · Re-check" (refresh=1, rate limited
server-side; a limited re-check says so). Invite links when no bot fits.
- Start is disabled only on a definite refusal (needs_fix, bad_channel);
unknown fails open. A channel this system already plays in is judged by
the bot that carries it, so Update/move is never blocked by the other bots.
- Admin Add form disables bots that are definitely denied in the bot list.
- 409 refusals (missing_permissions, preferred_busy, preferred_other_node,
bad_channel) are shown verbatim, multi-line.
Org owners (owner decision Q2) get "Radio bot per system" on
Settings > Organization: their own systems, their own BYO bots, set/clear
through S3's PUT /tokens/{id}/prefer/{system}. New GET /tokens/org/preferred
lists only the caller org's systems and bots (no pool bots, no token
strings; a pool bot an admin chose is flagged, not named).
The prefer route now drops the 5-minute doc_get_cached entry for every
system it rewrites, so a new preference applies to the next start.
Stacked on feat/225-s3-perm-check: must merge after S3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
start_stream now asks Discord (cached REST snapshots, S2) whether each bot
it could pick may View, Connect and Speak in the voice channel, before the
lease. Covers all four entry points: bot POST /discord/streams, web picker,
nodes discord_join, and the reconciler's re-place.
- pick_token: denied bots skipped; confirmed-ok sorts before unknown;
unknown is allowed (fail open); all denied -> missing_permissions naming
each bot, missing permission, trace reason and fix. The reuse branch
falls through when its token is denied.
- Bounded: discord_perm_check_concurrency at once, within
discord_perm_check_budget_seconds; unanswered = unknown.
- Non-voice channel -> 400 bad_channel (not missing_permissions).
- systems/{id}.preferred_token_id is now read (plan finding 1), as a hint.
- Pool bots at discord_pool_bot_guild_cap are never offered as invites;
pool_full when nothing else is left (owner Q3).
- handle_actual drops cached access on missing_permissions, channel_full,
not_in_guild, bad_channel.
- New GET /discord/nodes/{node_id}/guilds/{guild_id}/voice-check (org
member, names only) and GET /discord/guilds/{guild_id}/voice-check
(service key or admin).
- PUT /tokens/{id}/prefer/{system_id}: org owners for their own systems
and own BYO bots; shared systems and pool bots stay admin-only (owner Q2).
Keeps both pointers single-valued; audited.
- Kill switch: DISCORD_PERM_CHECK_ENABLED=false.
- tests/conftest.py: autouse fixture so no test ever reaches discord.com.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New app/internal/discord_rest.py: one shared httpx client with the
DiscordBot User-Agent, per-token rate-limit buckets (X-RateLimit-Bucket +
guild), one retry on a 429 of <=3s, global-429 pause per token, and a
process-wide breaker at 4000 401/403/unshared-429 responses per 10 min so
C2 can never get the shared egress IP banned. Tokens are never logged.
inspect_token / refresh_membership / leave_guild / TokenRejected move off
discord_leases onto it. /users/@me/guilds now follows after= pages (it read
one page of 200 and silently dropped the rest); a failed page is unknown,
never a truncated list.
guild_snapshot(token_doc, guild_id): cached, single-flight read of
/guilds/{g}, the bot's own member and /channels for S3's permission check.
401 marks the token invalid; 10004/10007 drop the server from guild_ids;
50001 is confirmed with one membership read and never changes guild_ids
on its own. TTLs: ok 120s, not-member/invalid 600s, unknown 30s.
config.py gets every setting the #225 slices use (perm-check kill switch,
TTLs, breaker budget, OAuth redirect URI, discord_pool_bot_guild_cap=90).
No behaviour change beyond pagination, User-Agent and the breaker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner-reported on live: adding a radio channel for a server + voice channel
that already had one (different label, system, even bot) overwrote the
existing stream and made its bot leave, because stream_id was guild:channel.
- New streams get guild:channel:<12 hex> ids (leases.new_stream_id); old
guild:channel docs keep working for read/stop/remove/node reports. Route
validators accept both (leases.STREAM_ID_PATTERN).
- start_stream: no stream_id = ADD. Same system already on in that channel
is idempotent (already_on, label/mode untouched); a stopped one of that
system is restarted; anything else is a new doc. stream_id = explicit
restart/edit (admin Start button, reconciler), incl. moving to another bot.
- preferred_token_id is strict: bot already in this server -> 409
preferred_busy naming the stream; on another node -> preferred_other_node;
unusable -> preferred_unavailable. No silent swap. pool_exhausted now says
one-bot-per-server and to add + invite another bot.
- One-token-per-guild was already enforced inside the lease transaction;
now covered by race tests. Per-guild asyncio lock serialises the
already-on / server-cap check with the create (guild half of #223).
- Stop by stream_id (POST /discord/guilds/{id}/stop {stream_id}); admin UI
Stop/Start/Remove act on the row's stream; node picker lists every stream
in a channel and adds beside other systems.
- Bot: /leave stream: (autocomplete) stops one; /leave channel: with several
streams asks which unless all:True; bare /leave still stops the server.
/status and /join name each stream's bot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /nodes/request-update-all (admin) does what Update Node does for each
approved node: flags update_requested_at for the HTTPS updater and nudges it
over MQTT. Pending and rejected nodes are skipped. Admin-only button on the
Nodes page reports which nodes weren't reachable over MQTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DELETE /discord/guilds/{id} deletes channels one by one. If a channel was
restarted mid-removal, delete_stream raised "restarted" and the route
returned 409 with no audit entry, though the channels before it were
already deleted for good. The error now carries the channels already
removed and the route writes discord.guild.remove with outcome
"incomplete" before returning the 409. The server stays linked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- handle_actual logs every node-reported state TRANSITION (not heartbeats)
with stream/node/guild/channel/token ids and the node's error text: info
for connected/connecting/stopped, warning for error, and a dedicated
"radio bot X failed to log in on node N: <reason>" line for login_failed.
The node's reason code is now stored in actual.code and shown next to the
error on Settings > Discord and the node page.
- Remove a radio channel permanently: streams.delete_stream stops it through
desired state first (node republished without it, token -> releasing per
#216, freed on the node's ack), then deletes the doc in a transaction that
refuses if the channel was restarted in between. DELETE
/discord/streams/{id} (platform admin) and DELETE
/discord/nodes/{node}/streams/{id} (admin or org owner, same org + node
only; operators can still stop, not remove). Audited.
- Remove a server: DELETE /discord/guilds/{id} (admin) deletes every radio
channel in it the same way, then tombstones discord_guilds (org_id None +
unlinked_at, #218 semantics, plus removed_at) so it never falls back to
DISCORD_UNLINKED_GUILD_ORG_ID; the admin page hides it, and /drb link or the
admin link brings it back. Optional leave_radio_bots makes pool bots and the
server's org's own BYO bots leave via Discord REST; the DRB server bot stays
until kicked, and the UI says so. Audited.
- UI: Remove (with confirm) on Radio channels, Servers and the node page list.
Refs #213, #221
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- POST /discord/nodes/{id}/streams/stop: stream_id must be guild:channel
snowflakes, so a malformed id is a 422 instead of a Firestore path error.
- Offline footer said channels resume on reconnect; ad-hoc ones stop once
the node is gone past the lease grace (reconcile_once). Say so.
- Picker: say why Start is disabled when the node has no system.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #218 (server-26#213 item 1) unlinks a server by clearing org_id and
stamping unlinked_at, and resolve_guild_org then refuses the fallback org.
The overview showed such a server as "Unlinked · fallback org" and offered it
in Add radio channel, where start_stream would 403. Mirror the rule here
(forward-compatible: unlinked_at does not exist on main yet). Also key each
stream row by the resolved guild id, and drop panel copy that pointed at a
"Needed" figure the page doesn't show / misstated the invite permissions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frontend, built from design-canvas/discord-system-mockup.html ("DRB admin"):
- Settings > Discord (/settings/discord, platform admin only; the tab is
hidden from org owners because every route behind it is admin-only).
KPIs + tabs:
* Radio channels: every discord_streams doc, desired -> actual, the node's
real error, red after 2 min desired-on without connecting. Stop
(POST /discord/guilds/{id}/stop), Start for a stopped channel and
"+ Add radio channel" (POST /discord/streams), with the invite links a
"no bot in this server" refusal carries.
* Servers: linked servers plus servers seen running radio while unlinked
(acting for DISCORD_UNLINKED_GUILD_ORG_ID); Link / Change org
(PUT /discord/guilds/{id}/link).
* Radio bots: valid / check_error, guild membership with a per-server
invite link for each server the bot is missing from, lease (node,
since, releasing), fan-out, Refresh (POST /tokens/{id}/refresh),
Remove, Release all leases (POST /tokens/flush), add with optional BYO org.
* How it works.
- /tokens now renders the same radio-bot table (new token fields + refresh).
- Left out, no backend yet: alert routes, incident forum/roles/plan columns,
per-channel talkgroup chips, "Generate link code" (item 1).
Backend (routers/discord.py):
- GET /discord/admin/overview (platform admin): guilds, streams with node
and bot names, orgs, systems, limits. No token strings, no Discord user
ids, no linking-admin uid.
- PUT /discord/guilds/{id}/link: digits-only id, optional guild_name label,
and re-linking a server to a different org stops its live channels
(the reconciler would otherwise keep re-placing them on the old org's
nodes, using its stored org_id).
- POST /discord/streams: guild_id/channel_id must be digits (the web form
types them in; the bot always sent snowflakes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A server an org explicitly unlinked (unlinked_at) must not fall back to the
default org in the picker either, or the founding org could stream into it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core routers/discord_web.py:
PUT /discord/guilds/{id}/directory (service key): the server bot pushes each
server's name + voice channels; merged onto discord_guilds, never touches org_id.
GET /discord/nodes/{id}/join-options: the node org's servers (same
resolve rule as the bot), their voice channels with any live stream,
radio-bot readiness + invite links, and this node's streams. No tokens.
POST /discord/nodes/{id}/streams: stream pinned to the node; server must be
the node org's, channel must be in the bot's directory, guild + node caps.
POST /discord/nodes/{id}/streams/stop: one stream on this node, or all.
Control = platform admin, org owner, or operator owning the node.
- nodes.py: legacy raw-ID discord_join now refuses a server outside the node's org.
- discord bot: DirectorySync cog pushes on ready (15-min loop), join/leave,
rename, voice/category channel changes (debounced).
- frontend: NodeDiscordRadio panel + picker replaces DiscordJoinModal and the
admin-only Join/Leave buttons.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core: POST /discord/link-codes (org owner/admin) mints a single-use,
10-minute code; minting revokes the org's older codes. The bot redeems it
via POST /discord/guilds/{id}/link/redeem (service key) inside one
Firestore transaction, so two servers racing for one code cannot both win
and a server linked to org A refuses org B's code. GET/DELETE
/discord/linked-guilds lists/unlinks the caller's org's servers; unlink
stops that server's streams. Rate-limited (mint per user, redeem per
server and per Discord user); every mint/link/refusal/unlink audited.
- resolve_guild_org: an explicitly unlinked server no longer falls back to
DISCORD_UNLINKED_GUILD_ORG_ID. Linking away from the fallback org stops
the streams it had running for that org.
- Discord bot: /drb link code: (Manage Server, enforced at runtime too);
GatedTree answers every other command in an unlinked server with how to
link (autocomplete returns nothing). With the default fallback org set,
nothing is gated — unchanged behaviour.
- Frontend: "Linked Discord servers" card on Settings -> Organization —
generate code with countdown + copy, polls for the new link, unlink.
- 16 tests: single-use, two-server race, one-server/two-org race, expiry,
revoked/malformed codes, wrong-org takeover, idempotent relink, rate
limits, fallback stop, scoped list, unlink without fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c2-core
- discord_streams/discord_leases wired in: POST /discord/streams,
POST /discord/guilds/{id}/stop, GET /discord/guilds/{id}/overview,
GET /discord/streams/{id}, PUT /discord/guilds/{id}/link (admin),
GET /discord/node/{node}/tokens/{id} (node key, lease holder only).
- Retained nodes/{id}/discord/desired (token ids, never token strings),
nodes/+/discord/actual subscribed and handled; per-node publish lock keeps
the retained list the newest; failed publishes retried by the reconciler.
- Tokens freed only after the node acks logout (or offline past
discord_lease_offline_release_seconds) - sweeper and C2 startup no longer
release leases.
- /nodes/{id}/command discord_join/leave map onto streams; CommandPayload
keeps preferred_token_id. /tokens: admin-only list, validate on add against
Discord, refresh, flush via transactional force release.
- Guild -> org via discord_guilds, unlinked guilds fall back to
DISCORD_UNLINKED_GUILD_ORG_ID (default "founding").
- tests/fake_firestore.py: optimistic-transaction fake; 27 lease/race tests.
discord bot
- /join, /leave, /status, autocomplete scoped to the calling guild; /leave
never touches another server; Manage Channels default permission;
/join edits its reply with the node's real result; /help copy fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No AI client set a timeout: OpenAI's SDK waits 600s per attempt and retries
twice (timeouts included); google-genai waits forever. A hung provider call
had no upper bound, so recorrelation_sweep's 15-minute pipeline hold-back was
a guess against an unbounded tail.
- config: stt_timeout_seconds (120), openai_timeout_seconds (60),
openai_max_retries (1), gemini_timeout_seconds (60).
- app/internal/ai_clients.py builds every OpenAI/AsyncOpenAI client with them
(transcription, extraction, embeddings, summarizer, vocabulary learner,
trip chat). gemini.py's shared client gets HttpOptions(timeout=ms).
- A timeout raises like any other provider error, so existing fallbacks are
unchanged: rules-only correlation, rules decision on tiebreak, raw
transcript on correction, ai_health "transient".
- MIN_MINUTES_SINCE_PIPELINE_START is now derived from those timeouts
(pipeline_worst_case_seconds, 3-scene budget + 2 min non-AI slack), floor
15: 21 min with the defaults. Loop warns if it reaches the scan window.
- tests/test_ai_timeouts.py: client limits, a real hung-socket request that
gives up in <5s, timeout fallback per tier, the derived bound, and a guard
against any new OpenAI( outside ai_clients.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reject_node only set approval_status, so an approved-then-rejected node kept
its dynsec login, its /upload key and /systems read access.
- reject_node: status -> rejected first (fail-closed), then
dynsec.delete_node_client, then delete node_keys. dynsec failure -> 502,
key kept, retry is idempotent. Re-approve mints a fresh key.
- upload.py: valid key alone is not enough; node must be approved (403),
uncached read so revocation bites immediately.
- dynsec.reconcile_all: deletes rejected nodes' logins on startup, healing
nodes rejected before this change or whose revoke failed.
- Reject confirm text says it revokes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend (c2-core):
- app/internal/cameras/: provider interface, licence model, per-provider
catalog cache; one adapter per source in design-canvas/camera-sources.md.
Implemented: 511NY (legacy shape + tolerant XML inventory parser, 511x
endpoints UNVERIFIED), Caltrans, DelDOT, Ontario 511, DriveBC, Calgary,
TfL, NSW, Fintraffic. Stubs: NYC DOT, Westchester DPW, TRANSCOM, CTroads,
511NJ, Austin, TxDOT, MD CHART, WSDOT, Tallinn, Tarktee, Warendorf,
TrafficLand, Windy, EarthCam.
- Every provider defaults OFF (config/camera_providers). Gated licences
(agreement required / unverified / non-commercial / paid) need an explicit
acknowledgement stored with who/when and written to audit_log; the audit
write is a precondition. Stubs and prohibited sources cannot be enabled.
Ack is bound to the licence status it was given for.
- GET /cameras (signed-in), GET/PUT /admin/camera-providers (admin only).
- /traffic/511 follows the ny511 provider switch and returns the DAA credit
+ NYSDOT as-is disclaimer; 511NY event descriptions no longer truncated
("may not alter the substance of a report").
Frontend:
- LiveView becomes the ops wall: Region > Incident > Camera drill-down,
worst-first rail (MapView), KPIs, transcript ticker, scrubber incident
bands, auto-tour, deep links (#inc=&cam=&tour), Esc steps up.
- Incident panel: nearest cameras (path-weighted), fused situational context
(ADS-B, AIS, 511NY events, NEXRAD tile sample), units, call spine.
- MapView keeps every layer; adds Camera Viewsheds (approx.) overlay and a
nearest-camera layer for the selected incident.
- CameraFeed hygiene: hidden-tab/off-screen gating, 2 concurrent HLS cap,
last good frame + stale badge, frozen-frame signature, HLS stall watchdog,
provider attribution rendered on the imagery.
- Camera/event popups show source credit, update time and the disclaimer.
- Admin > Camera providers tab with licence status, health, gated confirm.
- Camera detection is not built (blocked on #199/#200); seam only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- system_config.normalize(): a P25 system with config.mode "conventional"
carries config.channels [{freq MHz, name, nac?}] (1-2, within 1.5 MHz) and
can no longer carry control channels (422). A missing mode is trunked and
passes through untouched, so there is no migration. An analog channel is
mirrored into control_channels for older nodes.
- talkgroups.name_from_system resolves the edge node's synthetic per-channel
id (freq in 100 Hz units) to the channel name.
- The checkin persists radio_channels (conventional channels + encryption).
- Systems editor: Type = P25 trunked / P25 conventional / Analog (NBFM) /
DMR, plus a channel-row editor for conventional and a one-frequency form for
analog. The node page shows conventional channels and an encryption notice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- approve_node: reuses node_keys/{id}.api_key when one exists (still
re-syncs dynsec with it, still never overwrites node_keys). Re-clicking
Approve can no longer lock a live node out. Reissue stays explicit.
- POST /nodes/{id}/request-update (admin): sets update_requested_at on the
node doc AND sends MQTT node_update best-effort, so offline/locked-out
nodes still get it.
- POST /nodes/{id}/update-check (node, inline auth: Bearer api_key for THAT
node or X-Pickup-Secret): host updater's HTTPS check-in. Returns only
update_requested_at; records updater_last_check_at / sha / applied result.
Unknown node -> 401 (no node_id oracle). C2 cannot name a ref, image or
command.
- Frontend: "Update Node Software" (admin only) calls request-update, honest
confirm text; node page shows updater check-in and last update state.
Tests: tests/test_node_provisioning.py (12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A viewer whose first login ran self-serve signup got an empty org of
their own (org_role owner), so they saw no incidents or calls, and the
earlier fix deliberately never moved a user who already had an org.
PATCH /admin/users/{uid} now moves a user when org_id is passed
explicitly (claims + org_members, audited with left_org_id; the old org
is not deleted). The user list returns org_id/org_role, and the admin
user panel shows the org and a 'Move to my organization' button when it
isn't yours.
Verified: c2-core pytest 496 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /admin/users set a role claim but never an org_id/org_role claim or
an org_members doc. firestore.rules gates every read on the org_id claim,
so a viewer created from the Users page saw no incidents or calls.
New users now join the requested org (default: the acting admin's own,
else 'founding') as org_role 'member', with the same org_members doc
POST /auth/signup writes. PATCH /admin/users/{uid} attaches an org-less
user the same way, which heals accounts created before this fix; it never
silently moves a user who already has an org.
Verified: c2-core pytest 495 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/sdr-pins. NodeRecord gains sdr_pins
(service -> serial), sdr_devices and op25_sdr_serial, mirrored from the
node's checkin. PATCH /nodes/{id} validates pins (known services, one
dongle per service) and sends priority/pins as a 'set_sdr_config'
command, never a config re-push. The node restarts OP25 only when OP25's
own dongle changes. The node page's section becomes 'SDRs' with an OP25
SDR dropdown ('Automatic (first SDR)' + detected dongles) and a
per-service dongle dropdown ('Any spare SDR'), plus duplicate-serial and
double-pin warnings.
Verified: c2-core pytest 490 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /traffic/511 serves a bbox slice of the statewide 511NY camera and event
feeds from an in-memory cache (cameras 1h, events 2m TTL, fetched lazily) --
public data, so no Firestore writes. A failed refresh keeps the last good data
and reports the error; a schema change (nothing parses) is an error, not an
empty layer. Frontend: opt-in "DOT Cameras" and "Traffic Events" overlays that
fetch only while shown, with an on-map notice when the feed is down or stale.
NY511_API_KEY is optional in config (the API answers without one today; the
terms require a registered key).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/secondary-sdr-priority. NodeRecord gains
secondary_sdr_priority (ordered; SDRs beyond OP25's run it top-down) and
secondary_sdr_running, both mirrored from the node's checkin.
PATCH /nodes/{id} accepts the priority, validates it, and sends it as a
'set_secondary_priority' MQTT command. A priority-only change never
re-pushes system config, because that restarts OP25. The node detail page
gets a 'Secondary SDRs' section (admin-editable) with enable, reorder,
save, and live Running / Waiting-for-SDR state from the checkin.
Verified: c2-core pytest 482 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Icons were 16px accent-colored glyphs, indistinguishable from OSM's own
airport symbols. Now a 30px outlined airliner silhouette filled on
tar1090/ADS-B Exchange's altitude hue ramp, with a callsign/altitude
hover tooltip; the selected aircraft grows and gets a white outline.
Clicking an aircraft draws the path heard so far, segment-colored by
altitude. c2-core writes one point per position change to
aircraft/{icao}/positions (deduped in-process, writes now concurrent);
points carry expire_at and a TTL fieldOverride deletes them after ~24h.
Trail reads are gated on the parent aircraft doc's org via get(), so the
query needs no org filter or composite index. The latest stretch without
a 20-min gap counts as the current flight.
Verified: c2-core pytest 479 passed; frontend tsc --noEmit clean (node:20
container on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay of 433b35d (server-26#170): traffic stops now open incidents
("45 Adam" stop, "CM2" stop), but the bridge MVA split 33/56 — one
transmission ("transport complete") was read as scene-resolved at 14:44
and an LLM closure was final, so the rest of the MVA opened a new one.
- LLM closure is now provisional (reopenable), like a timer close: it is
inferred from a single transmission.
- review of 433b35d: backstop only matches a unit's own "on a stop"
self-report (bare "car stop" mentions and "pull over" dropped), never on
MTA/rail/bridge/fire/EMS/DPW talkgroups ("Train 4 holding on the stop"),
negation looks 5 words back, and <=5-word reports ("Adam 3 on a stop")
get a minimal scene instead of being skipped before the backstop.
c2-core: 471 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first clearance fix found it pushes toward early resolve:
- parser cleared units on questions ("45-9, are you clear?"), negations
("not clear yet"), orders ("clear the scene", "clear to transport"),
places/times ("Room 2 clear", "1400 hours, clear") and split "45 9" into
unit 45. Now rejects "?", not/is/are/you, anything after the status word
but sign-offs, place/time words; joins "45 9" -> "45-9".
- a clear from a unit never active on an incident was recorded in
units_cleared and could pass the all-clear gate. Only units actually
active there can clear there now.
- clearance-only calls skip the LLM tier (same as thin calls): only the
rules engine's unit match can say which incident a 10-8 belongs to.
- replay incident view carries srcaddr/srcaddrs for the radio-ID clearance
investigation.
Replay 09-22 10:00-12:00 ET with f0a88d4: real clears 0 -> 2 (both LLM
closure), unit clears still 0 — the parsed clears are right but those
units were never recorded as assigned (Whisper mangles unit IDs at
dispatch), which this commit does not fix.
c2-core: 465 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Correlation has only ever been measured through live AI windows: days of
wall time per change, and the 09-20→22 window was invalidated outright by
unfunded AI accounts (#169). Recordings are kept regardless of AI, so the
traffic to measure against already exists.
- internal/replay.py: runs a time range of real calls through the live
pipeline code in original order, clock pinned per call, into
replay_runs/{run_id}/calls|incidents. Modes: audio (re-transcribe),
transcripts (re-extract), reuse (correlation only from a prior run's
scenes). Simulates the idle-resolve and orphan-recorrelation sweeps on
virtual time. No alerts, summaries, vocab, AI-health alerts or pending
terms. One run at a time, <=5000 calls, <=7 days.
- firestore.py: ContextVar sandbox redirect for calls/incidents.
- clock.py: ContextVar-pinnable now(), used on the correlation path.
- feature_flags.py: ContextVar flag override so replay runs with live AI off.
- upload.py: scene loop extracted to _extract_and_correlate, shared by the
live pipeline and replay so replay measures the code that runs live.
- resolved_via on every incident resolve, so a real clear can be told
from the idle timeout — live and in replay.
- routers/replay.py + /admin Replay tab: estimate, start, compare runs,
drill into incidents with audio.
Reviewed by drb-correlation-review; its leak and fidelity findings are
fixed and covered by tests. c2-core: 456 pass. Frontend typecheck not run
(no Node on the authoring box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/calls/search scans a 200-row window and returns 50, but the next cursor
was always the last SCANNED row — so with an unfiltered list each page
jumped past the 150 matches it had already read and not shown. Resume
after the last RETURNED row when matches overflow the page; keep the
last-scanned cursor only when the page holds every match (the sparse-
filter case that cursor exists for). Same fix for /calls/eval-queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incidents and Archive get a from/to date range (native date inputs,
local-day bounds). Incidents filters in the Firestore query; Archive
passes date_from/date_to to GET /calls/search, which applies them as a
started_at range — both ride the existing org_id/started_at index.
Also fixes /calls/search and /calls/eval-queue paging: the cursor went to
Firestore as a raw ISO string against a timestamp field, which compares
by type rather than time, so "Load more" re-read the first page. Cursor
and range bounds are now parsed to datetimes (400 on garbage).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Incidents page: text search (title, location, summary, units, vehicles,
tags, location mentions), status and type filters, and a Load more button
that pages the Firestore query 100 at a time. Filtering runs over the
loaded window, and the page says so when older incidents exist.
Archive (/calls): readable by every org member, not just admins.
GET /calls/search now takes any Firebase token scoped to the caller's org
— the Firestore rules already let members read every call in their org,
so this widens nothing. Attach/detach stays admin-only (UI and routes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.
GET /calls/eval-queue -- calls with a transcript but no
eval_transcript yet, paged (same bounded-
window-plus-cursor shape as /search)
PUT /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
never touches transcript/transcript_corrected
GET /calls/eval-stats -- eval_count + average word error rate of
the raw and corrected machine transcripts
against the human-verified ones
internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.
Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.
Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
intelligence.py skips GPT extraction for transcripts <=5 words ("10-8", "show
me clear", a unit check-in) -- real cost/hallucination guard, kept as-is. But
upload.py's no-scenes correlation fallback (the path that thin-links a call
by talkgroup even with zero extracted content) excluded ANY skip_reason,
including transcript_too_short -- so this exact population, brief but real
follow-up and clearance traffic, never even attempted to attach to anything.
Found live: "Live, Ossining." sitting an orphan 6 seconds before a real
incident's founding call, on the same talkgroup.
Now only garbage_transcript (Whisper hallucination, no real content) stays
excluded; transcript_too_short reaches the same thin/fast-path fallback
already trusted for no-transcript calls, gated the same way -- same-talkgroup,
recently-active incident required before anything attaches. No GPT re-invoked,
no new cost.
Verified: 410 pass, 0 fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-26#9, same shape as the ADS-B commit. Adds POST /telemetry/ais
(same node-key auth, same org_id-stamped upsert-by-key pattern, this time
by mmsi into a new `vessels` collection) and its docInMyOrg() firestore
rule. Frontend gets useVessels() (mirrors useAircraft(), longer staleness
window since AIS position reports are minutes apart, not seconds) and an
opt-in "Vessels" map overlay.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-26#9. Adds POST /telemetry/adsb (node-key authed via
require_node_service_or_firebase_token) that upserts one Firestore doc per
icao into a new `aircraft` collection, org_id stamped from the reporting
node the same way upload.py defensively stamps `calls`. firestore.rules
gets a matching docInMyOrg()-gated read rule.
Frontend: useAircraft() mirrors useNodes()'s onSnapshot pattern, filtering
docs older than 2 minutes client-side since nothing prunes a stale aircraft
doc server-side yet. MapView gets an opt-in "Aircraft" overlay (unchecked
by default, like the weather radar layer) rendering a rotated plane glyph
per sighting.
Unverified via typecheck — no Node.js/npm on this authoring box yet (see
CLAUDE.md Testing reality). Server side is pytest-covered (test_telemetry.py).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Server-side half of node-26#9. NodeRecord gains secondary_sdr_mode
(none|adsb|ais|op25_2) and sdr_count; checkin ingestion stores both,
and PATCH /nodes/{id} accepts and re-pushes secondary_sdr_mode the same
way hardware_preset/ppm_override already work, so it isn't wiped by a
system reassignment (see server-26#111 for the pre-existing bug that
pattern avoids repeating).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent fixes, found by tracing why incidents never actually close
(only 2/281 incidents in 5 correlation debug windows ever got a non-empty
units_cleared; 183 resolved via the 90-min idle sweep instead of a real clear):
1. Pattern B (re-dispatch accept, no explicit 10-8): reassignment=True already
fired correctly and suppressed the unit from re-linking to its prior
incident, but nothing ever released the unit FROM that incident — it just
sat "active" until the idle sweep timed it out. _release_reassigned_units
now scans other active incidents for unit overlap on a reassignment and
clears the unit there, reusing the same units_active/units_cleared merge
(factored out as _apply_unit_clearance) that explicit 10-8 extraction uses.
2. Pattern A (self-clear) extraction was inconsistent for two reasons: no
per-system unit ID format awareness anywhere in the pipeline (formats vary
by department with zero shared convention), and the cleared_units prompt
rule only accepted a unit self-reporting, missing dispatch confirming a
unit's status back to them. Added system.unit_format_hint (owner-authored
free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed
into the extraction prompt, and broadened the cleared_units rule while
still requiring an identifiable unit ID (guards against bare "10-8"/"clear"
noise, including Whisper hallucination runs already caught upstream by
_is_garbage_transcript).
Verified: 401 pass, 0 fail (local Linux venv ~/venvs/drb-5c — see CLAUDE.md
testing-reality note).
server-26#pending — not yet filed, Gitea unreachable from this sandbox.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.
Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.
15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.
recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.
Sandboxed pytest: 381 -> 387.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
_call_is_substanceless's "type" veto reads ctx["incident_type"] at decision
time, but that value was never persisted per-scene — only the last-scene-wins
flat field, which #138's window-4 dump analysis couldn't tell apart from
cross-scene contamination without re-guessing from a live dump. Adds
incident_type/severity to _apply_and_log's per-scene write and to admin.py's
_scene_summary allowlist (the debug-dump reader has its own field allowlist,
separate from the write side — silently would not have surfaced otherwise).
Sandboxed pytest: 380 -> 381.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Full removal, not a hardcoded flag: _is_dispatch_channel, _DISPATCH_TG_RE, the is_dispatch parameter, and _call_fits_incident's tactical branch are gone. One evaluation path for every channel.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Owner correction from direct scanning experience: a talkgroup named tac/tactical only sees materially different traffic during a real incident, and that's rare -- the bulk of traffic on any monitored channel, including high-risk stops and pursuits, runs on the main channel regardless of what it's named. _is_dispatch_channel's string match on the talkgroup name is a naming-convention guess, not a detector of actual channel behavior; trusting it here meant a busy single-channel department not literally named 'dispatch' would silently get the more permissive 15-minute window and could reproduce #115's original bug (the gate never firing on the channels it targets).
Always use tg_dispatch_thin_idle_minutes (5 min) in the escape hatch, regardless of talkgroup name. Does NOT touch incident_correlator.py's own fast/thin idle-window selection, which uses the same dichotomy for a different, decision-changing purpose -- bigger blast radius, left for its own review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident.
Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).
Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
- the existing flat corr_* fields, unchanged (last-scene-wins, the safe
backward-compatible default for any reader that doesn't know about
`scenes` yet)
- a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
corr_debug}, via doc_set(..., merge=True). Firestore's
DocumentReference.set(data, merge=True) recursively merges nested map
fields by key (documented SDK behaviour, not assumed) — a write to
scenes.1 merges alongside an existing scenes.0 instead of replacing the
whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.
admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.
summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.
Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.
Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.
Fixes server-26#96, server-26#114
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
