Two merges minutes apart ran two "Deploy to VM" jobs on the VM at once
(616/617, 652/656): racing compose pull/up, one run's prune breaking the
other's retry, health checks judging the wrong containers.
- deploy job: concurrency group drb-prod-deploy, cancel-in-progress false
(Gitea >= 1.26 honours it, go-gitea/gitea#32751; server is 1.27.3).
- Deploy step exits clean, before touching the VM, when its SHA is no
longer origin/main's tip; Health check is skipped for a superseded run.
Stops an older run landing after a newer one and rolling prod back.
- deploy-firestore-rules: its own group plus the same tip check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No AI client set a timeout: OpenAI's SDK waits 600s per attempt and retries
twice (timeouts included); google-genai waits forever. A hung provider call
had no upper bound, so recorrelation_sweep's 15-minute pipeline hold-back was
a guess against an unbounded tail.
- config: stt_timeout_seconds (120), openai_timeout_seconds (60),
openai_max_retries (1), gemini_timeout_seconds (60).
- app/internal/ai_clients.py builds every OpenAI/AsyncOpenAI client with them
(transcription, extraction, embeddings, summarizer, vocabulary learner,
trip chat). gemini.py's shared client gets HttpOptions(timeout=ms).
- A timeout raises like any other provider error, so existing fallbacks are
unchanged: rules-only correlation, rules decision on tiebreak, raw
transcript on correction, ai_health "transient".
- MIN_MINUTES_SINCE_PIPELINE_START is now derived from those timeouts
(pipeline_worst_case_seconds, 3-scene budget + 2 min non-AI slack), floor
15: 21 min with the defaults. Loop warns if it reaches the scan window.
- tests/test_ai_timeouts.py: client limits, a real hung-socket request that
gives up in <5s, timeout fallback per tier, the derived bound, and a guard
against any new OpenAI( outside ai_clients.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reject_node only set approval_status, so an approved-then-rejected node kept
its dynsec login, its /upload key and /systems read access.
- reject_node: status -> rejected first (fail-closed), then
dynsec.delete_node_client, then delete node_keys. dynsec failure -> 502,
key kept, retry is idempotent. Re-approve mints a fresh key.
- upload.py: valid key alone is not enough; node must be approved (403),
uncached read so revocation bites immediately.
- dynsec.reconcile_all: deletes rejected nodes' logins on startup, healing
nodes rejected before this change or whose revoke failed.
- Reject confirm text says it revokes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- LiveView: localStorage pins carry image URLs, so they rendered (without
credit) after an admin disabled the provider. Pins now show only while
their provider is serving (server-26#198 licence gating).
- MapView: pinned/docked map players drop out once their provider leaves
the serving list; credits are kept from the last answer so docked players
keep attribution while the overlay is hidden.
- WallPanels: the "no camera provider enabled" link to /admin was a dead CTA
for non-admins (bounced to /). Link shown to admins only, deep-linked to
/admin#cameras; the admin page now honours a tab hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend (c2-core):
- app/internal/cameras/: provider interface, licence model, per-provider
catalog cache; one adapter per source in design-canvas/camera-sources.md.
Implemented: 511NY (legacy shape + tolerant XML inventory parser, 511x
endpoints UNVERIFIED), Caltrans, DelDOT, Ontario 511, DriveBC, Calgary,
TfL, NSW, Fintraffic. Stubs: NYC DOT, Westchester DPW, TRANSCOM, CTroads,
511NJ, Austin, TxDOT, MD CHART, WSDOT, Tallinn, Tarktee, Warendorf,
TrafficLand, Windy, EarthCam.
- Every provider defaults OFF (config/camera_providers). Gated licences
(agreement required / unverified / non-commercial / paid) need an explicit
acknowledgement stored with who/when and written to audit_log; the audit
write is a precondition. Stubs and prohibited sources cannot be enabled.
Ack is bound to the licence status it was given for.
- GET /cameras (signed-in), GET/PUT /admin/camera-providers (admin only).
- /traffic/511 follows the ny511 provider switch and returns the DAA credit
+ NYSDOT as-is disclaimer; 511NY event descriptions no longer truncated
("may not alter the substance of a report").
Frontend:
- LiveView becomes the ops wall: Region > Incident > Camera drill-down,
worst-first rail (MapView), KPIs, transcript ticker, scrubber incident
bands, auto-tour, deep links (#inc=&cam=&tour), Esc steps up.
- Incident panel: nearest cameras (path-weighted), fused situational context
(ADS-B, AIS, 511NY events, NEXRAD tile sample), units, call spine.
- MapView keeps every layer; adds Camera Viewsheds (approx.) overlay and a
nearest-camera layer for the selected incident.
- CameraFeed hygiene: hidden-tab/off-screen gating, 2 concurrent HLS cap,
last good frame + stale badge, frozen-frame signature, HLS stall watchdog,
provider attribution rendered on the imagery.
- Camera/event popups show source credit, update time and the disclaimer.
- Admin > Camera providers tab with licence status, health, gated confirm.
- Camera detection is not built (blocked on #199/#200); seam only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- system_config.normalize(): a P25 system with config.mode "conventional"
carries config.channels [{freq MHz, name, nac?}] (1-2, within 1.5 MHz) and
can no longer carry control channels (422). A missing mode is trunked and
passes through untouched, so there is no migration. An analog channel is
mirrored into control_channels for older nodes.
- talkgroups.name_from_system resolves the edge node's synthetic per-channel
id (freq in 100 Hz units) to the channel name.
- The checkin persists radio_channels (conventional channels + encryption).
- Systems editor: Type = P25 trunked / P25 conventional / Analog (NBFM) /
DMR, plus a channel-row editor for conventional and a one-frequency form for
analog. The node page shows conventional channels and an encryption notice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- approve_node: reuses node_keys/{id}.api_key when one exists (still
re-syncs dynsec with it, still never overwrites node_keys). Re-clicking
Approve can no longer lock a live node out. Reissue stays explicit.
- POST /nodes/{id}/request-update (admin): sets update_requested_at on the
node doc AND sends MQTT node_update best-effort, so offline/locked-out
nodes still get it.
- POST /nodes/{id}/update-check (node, inline auth: Bearer api_key for THAT
node or X-Pickup-Secret): host updater's HTTPS check-in. Returns only
update_requested_at; records updater_last_check_at / sha / applied result.
Unknown node -> 401 (no node_id oracle). C2 cannot name a ref, image or
command.
- Frontend: "Update Node Software" (admin only) calls request-update, honest
confirm text; node page shows updater check-in and last update state.
Tests: tests/test_node_provisioning.py (12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any fatal hls.js error set videoFailed permanently, so a brief 511NY feed
drop left the tile on a 60s snapshot until it was reopened. Now fatal
media/network errors get up to 3 in-place recoveries, and a tile that did
fall back retries video every 30s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tiebreaker forced ~all of the 09-27 rollover sink's bad links: idle resets
on every link so a 2.5h, 40-call incident read as fresh, and the rules treated
unit overlap as enough on a county channel where units work jobs back to back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DOT cameras can be pinned at the camera or docked in a strip over the
map; up to 6 open at once, switchable between modes.
- Below zoom 14, 511 cameras/events bucket into a 40px world-pixel grid,
one icon per cell with a count badge; click zooms in.
- Weather radar refreshed by remounting via `key`, and a remounted child of
an unchecked LayersControl.Overlay isn't re-added to the map — radar went
off every 5 min. Now refreshes in place with setUrl + cache-buster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A viewer whose first login ran self-serve signup got an empty org of
their own (org_role owner), so they saw no incidents or calls, and the
earlier fix deliberately never moved a user who already had an org.
PATCH /admin/users/{uid} now moves a user when org_id is passed
explicitly (claims + org_members, audited with left_org_id; the old org
is not deleted). The user list returns org_id/org_role, and the admin
user panel shows the org and a 'Move to my organization' button when it
isn't yours.
Verified: c2-core pytest 496 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /admin/users set a role claim but never an org_id/org_role claim or
an org_members doc. firestore.rules gates every read on the org_id claim,
so a viewer created from the Users page saw no incidents or calls.
New users now join the requested org (default: the acting admin's own,
else 'founding') as org_role 'member', with the same org_members doc
POST /auth/signup writes. PATCH /admin/users/{uid} attaches an org-less
user the same way, which heals accounts created before this fix; it never
silently moves a user who already has an org.
Verified: c2-core pytest 495 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Token colors wrapped in color-mix so opacity modifiers (bg-surface/90 etc.)
generate rules; map overlay panels were rendering see-through.
- Light mode: keep white text on saturated fills (primary/danger buttons were
remapped to navy); map text-gray-200, bg-gray-800/60, border-gray-600.
- Dark mode: lift text-gray-600/700 to gray-500 for contrast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clicking a camera's snapshot pins a 192x108 tile anchored at the camera --
the live HLS stream when the camera has one (hls.js; Safari native), else the
snapshot refreshed every 60s, falling back to the snapshot if the stream dies.
Up to 6 pins, oldest dropped. Players exist only while the DOT Cameras overlay
is on, so hiding it stops every stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/sdr-pins. NodeRecord gains sdr_pins
(service -> serial), sdr_devices and op25_sdr_serial, mirrored from the
node's checkin. PATCH /nodes/{id} validates pins (known services, one
dongle per service) and sends priority/pins as a 'set_sdr_config'
command, never a config re-push. The node restarts OP25 only when OP25's
own dongle changes. The node page's section becomes 'SDRs' with an OP25
SDR dropdown ('Automatic (first SDR)' + detected dongles) and a
per-service dongle dropdown ('Any spare SDR'), plus duplicate-serial and
double-pin warnings.
Verified: c2-core pytest 490 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /traffic/511 serves a bbox slice of the statewide 511NY camera and event
feeds from an in-memory cache (cameras 1h, events 2m TTL, fetched lazily) --
public data, so no Firestore writes. A failed refresh keeps the last good data
and reports the error; a schema change (nothing parses) is an error, not an
empty layer. Frontend: opt-in "DOT Cameras" and "Traffic Events" overlays that
fetch only while shown, with an on-map notice when the feed is down or stale.
NY511_API_KEY is optional in config (the API answers without one today; the
terms require a registered key).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QA (drb-qa-review) blockers: sdr_count defaulted to 1 for nodes that
never sent it, so the panel claimed 'reports 1 SDR (0 spare)'. It is now
None until reported, and the count is only quoted alongside a real
secondary_sdr_running report. Rows read 'Not reported' instead of
'Waiting for SDR' when the node hasn't said what's running (closes#187).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pairs with node-26 feat/secondary-sdr-priority. NodeRecord gains
secondary_sdr_priority (ordered; SDRs beyond OP25's run it top-down) and
secondary_sdr_running, both mirrored from the node's checkin.
PATCH /nodes/{id} accepts the priority, validates it, and sends it as a
'set_secondary_priority' MQTT command. A priority-only change never
re-pushes system config, because that restarts OP25. The node detail page
gets a 'Secondary SDRs' section (admin-editable) with enable, reorder,
save, and live Running / Waiting-for-SDR state from the checkin.
Verified: c2-core pytest 482 passed; frontend tsc --noEmit clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QA follow-ups on the ADS-B/AIS overlays:
- Legend gains an aircraft altitude key (tar1090 ramp) while the Aircraft
overlay is on (closes#185).
- AIS aids to navigation (MMSI 99xxxxxxx) render as small yellow buoy
diamonds labelled 'Aid to navigation', no speed; vessels get a 22px
outlined hull, and heading 511/360 ('not available') no longer
rotates the icon (closes#186).
- Aircraft details panel becomes a bottom sheet above the incident drawer
below md, instead of fighting the layers control at the top right.
Verified: tsc --noEmit clean (node:20 on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The popup sat on top of the plane and hid the trail it had just drawn.
Details now render in a panel docked top-right (portaled into the
Leaflet container, click/scroll propagation disabled), so the map can
be panned to follow the path. Clicking empty map or the same plane
deselects; hover tooltip unchanged.
Verified: tsc --noEmit clean (node:20 on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Icons were 16px accent-colored glyphs, indistinguishable from OSM's own
airport symbols. Now a 30px outlined airliner silhouette filled on
tar1090/ADS-B Exchange's altitude hue ramp, with a callsign/altitude
hover tooltip; the selected aircraft grows and gets a white outline.
Clicking an aircraft draws the path heard so far, segment-colored by
altitude. c2-core writes one point per position change to
aircraft/{icao}/positions (deduped in-process, writes now concurrent);
points carry expire_at and a TTL fieldOverride deletes them after ~24h.
Trail reads are gated on the parent aircraft doc's org via get(), so the
query needs no org filter or composite index. The latest stretch without
a 20-min gap counts as the current flight.
Verified: c2-core pytest 479 passed; frontend tsc --noEmit clean (node:20
container on radio-box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy-firestore-rules job has failed on every push because
FIREBASE_TOKEN was never set, so rule changes (e.g. aircraft/vessels for
node-26#9) never reached prod. Switch to a dedicated least-privilege
service account whose JSON key lives in FIREBASE_SA_KEY; login:ci tokens
are deprecated and carry their minter's full access. Key is written to
RUNNER_TEMP at 0600 and removed on exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 09-22 replay stop was titled "Traffic Stop at 96 times 5" — a disposition
code read aloud, extracted as the location (server-26#170). clean_location
now rejects "N times N", ten-codes, "signal N", "code N", "condition N".
c2-core: 476 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner: plate reads should become traffic stops. Held-out replay of 09-21
(server-26#170): Ossining's Post 4 stops were read out only as plates
("Frank David Boy, 4514", "Lincoln, Charlie, Robert, 7-4-0-7") and never
became incidents.
The self-initiated backstop now treats two+ phonetic letters followed by
3-7 digits as a stop — but only when extraction found no other event in
the call (a plate on an MVA, tow or parked-car complaint stays with that
event), and never on MTA/rail/bridge/fire/EMS/DPW talkgroups.
c2-core: 475 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A day of replay runs (server-26#170) spent ~$5 of Gemini on ~7 two-hour
windows (~$0.70 per 290 calls) — several dollars a day per live deployment
for correlation alone — and nothing could say where it went (#45). Gemini
3.x thinks by default and bills it as output; the deprecated
google-generativeai SDK these calls used cannot set a thinking level.
- app/internal/gemini.py: every Gemini call (correlation + transcript
correction) goes through google-genai with JSON mode, an explicit
thinking level, and logs in/out/thinking tokens. A model that rejects
the level is retried without it once and remembered, so the tier is
never lost to a config param. API failures still raise for ai_health.
- correlator: thinking_level "minimal" (a link/new/orphan choice).
transcript correction: "low" until a replay shows minimal is safe.
- replay: runs record real Gemini token usage (metrics.gemini_usage),
shown in the Replay tab.
- requirements: google-genai.
c2-core: 474 pass. Frontend typecheck not run (no Node on this box).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay of 433b35d (server-26#170): traffic stops now open incidents
("45 Adam" stop, "CM2" stop), but the bridge MVA split 33/56 — one
transmission ("transport complete") was read as scene-resolved at 14:44
and an LLM closure was final, so the rest of the MVA opened a new one.
- LLM closure is now provisional (reopenable), like a timer close: it is
inferred from a single transmission.
- review of 433b35d: backstop only matches a unit's own "on a stop"
self-report (bare "car stop" mentions and "pull over" dropped), never on
MTA/rail/bridge/fire/EMS/DPW talkgroups ("Train 4 holding on the stop"),
negation looks 5 words back, and <=5-word reports ("Adam 3 on a stop")
get a minimal scene instead of being skipped before the backstop.
c2-core: 471 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner: traffic stops should show on the portal — otherwise they are only
visible in the archive. In the 09-22 replay (server-26#170) every Ch 1
stop ("45 Adam on a stop, Eastbound Central Express", "CM2 on the stop,
southbound") came back from extraction untyped, untagged and routine —
read as status traffic after #138 — so the creation gate never opened one.
- prompt: a unit reporting its own activity (on a stop, out with a vehicle
or pedestrian) is a real event: police, tagged, at least minor; the plate
lookups for it belong to it.
- deterministic backstop after extraction: stop / "put me out with"
phrasing adds a "traffic-stop" / "self-initiated" tag (the substance the
creation gate counts), police type if none, minor if routine. Negated
phrasing ("not pull the car over") is left alone; nothing is downgraded.
c2-core: 469 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- only a substantive call after the close reopens a timer-closed incident;
a thin "10-4" (doesn't refresh updated_at) or a sweep link of a call from
before the close rides along without reopening — otherwise the next
sweep closed it again and the portal flickered.
- substantive_call_count counts a call once, not once per scene.
- a timer-closed incident is never adopted as a cross-system parent.
Replay e972cac (correlation-only on 731b54b's extraction) vs the 09-22
answer key: pairwise F1 0.548 -> 0.807 (precision 0.839 -> 0.956, recall
0.407 -> 0.698); the bridge MVA is one 76-call incident instead of two
40-call halves. server-26#170.
c2-core: 468 pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
