02b5b7b5a5b6997363d2c2b342f9e49dcc09b6f5
91
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Logan CusanoandClaude Opus 5.5
|
40014a47a3 |
c2-core: Archive "Load more" skipped 150 of every 200 calls
/calls/search scans a 200-row window and returns 50, but the next cursor was always the last SCANNED row — so with an unfiltered list each page jumped past the 150 matches it had already read and not shown. Resume after the last RETURNED row when matches overflow the page; keep the last-scanned cursor only when the page holds every match (the sparse- filter case that cursor exists for). Same fix for /calls/eval-queue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Opus 5.5
|
6479174022 |
frontend: date range picker on Incidents and Archive; fix Archive paging
Incidents and Archive get a from/to date range (native date inputs, local-day bounds). Incidents filters in the Firestore query; Archive passes date_from/date_to to GET /calls/search, which applies them as a started_at range — both ride the existing org_id/started_at index. Also fixes /calls/search and /calls/eval-queue paging: the cursor went to Firestore as a raw ISO string against a timestamp field, which compares by type rather than time, so "Load more" re-read the first page. Cursor and range bounds are now parsed to datetimes (400 on garbage). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Opus 5.5
|
fa194e0f0a |
frontend: search, filters and load-more on Incidents; open Archive to viewers
Incidents page: text search (title, location, summary, units, vehicles, tags, location mentions), status and type filters, and a Load more button that pages the Firestore query 100 at a time. Filtering runs over the loaded window, and the page says so when older incidents exist. Archive (/calls): readable by every org member, not just admins. GET /calls/search now takes any Firebase token scoped to the caller's org — the Firestore rules already let members read every call in their org, so this widens nothing. Attach/detach stays admin-only (UI and routes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Sonnet 5
|
5f85a878fa |
admin: STT eval harness — record human-verified transcripts, measure real WER (#163)
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.
GET /calls/eval-queue -- calls with a transcript but no
eval_transcript yet, paged (same bounded-
window-plus-cursor shape as /search)
PUT /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
never touches transcript/transcript_corrected
GET /calls/eval-stats -- eval_count + average word error rate of
the raw and corrected machine transcripts
against the human-verified ones
internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.
Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.
Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Sonnet 5
|
1ffff25cd2 |
upload: let short transcripts (<=5 words) attempt correlation instead of never linking at all
intelligence.py skips GPT extraction for transcripts <=5 words ("10-8", "show
me clear", a unit check-in) -- real cost/hallucination guard, kept as-is. But
upload.py's no-scenes correlation fallback (the path that thin-links a call
by talkgroup even with zero extracted content) excluded ANY skip_reason,
including transcript_too_short -- so this exact population, brief but real
follow-up and clearance traffic, never even attempted to attach to anything.
Found live: "Live, Ossining." sitting an orphan 6 seconds before a real
incident's founding call, on the same talkgroup.
Now only garbage_transcript (Whisper hallucination, no real content) stays
excluded; transcript_too_short reaches the same thin/fast-path fallback
already trusted for no-transcript calls, gated the same way -- same-talkgroup,
recently-active incident required before anything attaches. No GPT re-invoked,
no new cost.
Verified: 410 pass, 0 fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Sonnet 5
|
3d2b722c64 |
Wire AIS end to end: telemetry ingestion + live map overlay
node-26#9, same shape as the ADS-B commit. Adds POST /telemetry/ais (same node-key auth, same org_id-stamped upsert-by-key pattern, this time by mmsi into a new `vessels` collection) and its docInMyOrg() firestore rule. Frontend gets useVessels() (mirrors useAircraft(), longer staleness window since AIS position reports are minutes apart, not seconds) and an opt-in "Vessels" map overlay. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Sonnet 5
|
5537b095df |
Wire ADS-B end to end: telemetry ingestion + live map overlay
node-26#9. Adds POST /telemetry/adsb (node-key authed via require_node_service_or_firebase_token) that upserts one Firestore doc per icao into a new `aircraft` collection, org_id stamped from the reporting node the same way upload.py defensively stamps `calls`. firestore.rules gets a matching docInMyOrg()-gated read rule. Frontend: useAircraft() mirrors useNodes()'s onSnapshot pattern, filtering docs older than 2 minutes client-side since nothing prunes a stale aircraft doc server-side yet. MapView gets an opt-in "Aircraft" overlay (unchecked by default, like the weather radar layer) rendering a rotated plane glyph per sighting. Unverified via typecheck — no Node.js/npm on this authoring box yet (see CLAUDE.md Testing reality). Server side is pytest-covered (test_telemetry.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Sonnet 5
|
8892e824fc |
Add second-SDR fields: secondary_sdr_mode + sdr_count on NodeRecord
Server-side half of node-26#9. NodeRecord gains secondary_sdr_mode
(none|adsb|ais|op25_2) and sdr_count; checkin ingestion stores both,
and PATCH /nodes/{id} accepts and re-pushes secondary_sdr_mode the same
way hardware_preset/ppm_override already work, so it isn't wiped by a
system reassignment (see server-26#111 for the pre-existing bug that
pattern avoids repeating).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Sonnet 5
|
fb0bb15c22 |
correlator/intelligence: close the incident-clearance gap (dispatch-to-10-8 lifecycle)
Two independent fixes, found by tracing why incidents never actually close
(only 2/281 incidents in 5 correlation debug windows ever got a non-empty
units_cleared; 183 resolved via the 90-min idle sweep instead of a real clear):
1. Pattern B (re-dispatch accept, no explicit 10-8): reassignment=True already
fired correctly and suppressed the unit from re-linking to its prior
incident, but nothing ever released the unit FROM that incident — it just
sat "active" until the idle sweep timed it out. _release_reassigned_units
now scans other active incidents for unit overlap on a reassignment and
clears the unit there, reusing the same units_active/units_cleared merge
(factored out as _apply_unit_clearance) that explicit 10-8 extraction uses.
2. Pattern A (self-clear) extraction was inconsistent for two reasons: no
per-system unit ID format awareness anywhere in the pipeline (formats vary
by department with zero shared convention), and the cleared_units prompt
rule only accepted a unit self-reporting, missing dispatch confirming a
unit's status back to them. Added system.unit_format_hint (owner-authored
free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed
into the extraction prompt, and broadened the cleared_units rule while
still requiring an identifiable unit ID (guards against bare "10-8"/"clear"
noise, including Whisper hallucination runs already caught upstream by
_is_garbage_transcript).
Verified: 401 pass, 0 fail (local Linux venv ~/venvs/drb-5c — see CLAUDE.md
testing-reality note).
server-26#pending — not yet filed, Gitea unreachable from this sandbox.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Sonnet 5
|
8dd636af8f |
correlator: stop the re-correlation sweep racing an in-flight upload (#131)
server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical extracted data, ~2% of linked calls across 3 live dumps. Root cause: the sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't tell 'never processed' apart from 'real-time pipeline is still mid-flight' -- a call whose STT/scene-extraction/correlation chain (routers/upload.py _run_intelligence_pipeline) hasn't finished yet has none of those fields set, so the sweep picks it up and correlates it independently, sometimes onto a different incident than the real-time path lands on. Confirmed in review: _update_incident/_create_incident append call_id to an incident's call_ids unconditionally, with no cross-incident dedup guard -- preventing the second correlation attempt is the only lever available at this layer. Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc before any slow step; the sweep holds back any call whose marker is under 15 minutes old (raised from an initial 5 -- see below), regardless of how orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the marker write itself failed) is not held back -- absence isn't evidence of an in-flight pipeline, and that's #131's own pre-existing population. Also covers the /calls/{id}/reprocess path, which calls the same _run_intelligence_pipeline. 15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was a guess against an unbounded tail -- drb-correlation-review flagged this. Raising it is free on the recovery side: a call that finished processing (linked or genuinely orphaned) always has corr_path set and is already excluded by the sweep's other filter, so this constant only ever delays calls that are still actually running. DEFERRED.md row 52 (outside this repo, Version 5C root) updated to flag its ~6 min timing figure as stale. recorrelation_sweep.py had zero test coverage before this. New file covers the guard function's boundary (age < threshold vs exactly-at vs old vs missing vs unparseable) and one integration-shaped test proving a racing call never reaches correlate_call while a genuinely-orphaned call still does. Sandboxed pytest: 381 -> 387. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
7717fcccdd |
correlator/admin: capture per-scene incident_type + severity (#139)
_call_is_substanceless's "type" veto reads ctx["incident_type"] at decision time, but that value was never persisted per-scene — only the last-scene-wins flat field, which #138's window-4 dump analysis couldn't tell apart from cross-scene contamination without re-guessing from a live dump. Adds incident_type/severity to _apply_and_log's per-scene write and to admin.py's _scene_summary allowlist (the debug-dump reader has its own field allowlist, separate from the write side — silently would not have surfaced otherwise). Sandboxed pytest: 380 -> 381. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
b1884852d5 |
correlator: remove is_dispatch and the tactical fit path entirely (#134)
Full removal, not a hardcoded flag: _is_dispatch_channel, _DISPATCH_TG_RE, the is_dispatch parameter, and _call_fits_incident's tactical branch are gone. One evaluation path for every channel. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
c50bfda8db |
correlator: drop the dispatch/tactical name-guess from the escape hatch, use one window (#115)
Owner correction from direct scanning experience: a talkgroup named tac/tactical only sees materially different traffic during a real incident, and that's rare -- the bulk of traffic on any monitored channel, including high-risk stops and pursuits, runs on the main channel regardless of what it's named. _is_dispatch_channel's string match on the talkgroup name is a naming-convention guess, not a detector of actual channel behavior; trusting it here meant a busy single-channel department not literally named 'dispatch' would silently get the more permissive 15-minute window and could reproduce #115's original bug (the gate never firing on the channels it targets). Always use tg_dispatch_thin_idle_minutes (5 min) in the escape hatch, regardless of talkgroup name. Does NOT touch incident_correlator.py's own fast/thin idle-window selection, which uses the same dichotomy for a different, decision-changing purpose -- bigger blast radius, left for its own review. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
0fe6d3b567 |
correlator: delete stale scenes on re-extraction instead of leaving them to rot (#96/#114)
Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident. Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
fae84a45c3 |
correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix
Every scene of a multi-scene call correlates independently in upload.py's scene loop, but every scene's corr_debug was written flat onto the same shared call doc — scene 2's write silently clobbered scene 1's corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's raw transcript per linked call, mixing text from scenes the incident had nothing to do with, while ignoring transcript_corrected entirely (#114). Fix: thread a scene_index from both `for scene in scenes:` loops in upload.py down through _correlate_with_consensus -> incident_correlator.preview_correlation/correlate_call -> _build_context -> ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the same Firestore call: - the existing flat corr_* fields, unchanged (last-scene-wins, the safe backward-compatible default for any reader that doesn't know about `scenes` yet) - a new nested `scenes.<scene_index>` entry with {transcript, incident_id, corr_debug}, via doc_set(..., merge=True). Firestore's DocumentReference.set(data, merge=True) recursively merges nested map fields by key (documented SDK behaviour, not assumed) — a write to scenes.1 merges alongside an existing scenes.0 instead of replacing the whole `scenes` map. scene_index defaults to 0 for every caller with no scene concept (the recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain single-scene call still gets a one-entry `scenes` map equivalent to reading its flat fields today. admin.py's _call_summary exposes the new `scenes` list per call (each entry carrying the same corr_* field names as the flat fields, so the two shapes are interchangeable to the tally); the summary tally now iterates each call's scenes-if-present, else its own flat fields, so a 2-scene call with two different corr_path values counts as two data points instead of one blend. New `scene_decision_count` sits next to `linked_call_count` to make that distinction visible. summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map to find the scene(s) whose corr_debug recorded a link into the specific incident being summarized, joining more than one if several scenes landed in the same incident. Falls back to transcript_corrected-or-transcript for a call doc with no `scenes` field (predates this change) — the one-liner half of #114, worth doing regardless since it stops raw-transcript summaries even for old-schema docs. Does not touch #80/#95/#102's existing ctx-threading fixes (embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of scope here. Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py, additions to test_admin_debug_correlation.py) covering the merge shape, last-scene-wins flat-field backward compat, the admin tally's per-scene vs per-call counting (including old-schema fallback), and the summarizer's scene-specific text selection (including old-schema fallback). Full sandboxed suite: 364 -> 378 passed, all green. Fixes server-26#96, server-26#114 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
3ae0bb2d5b |
intelligence: run the chatter classifier before the too-short skip, not after (#127)
82% of the classifier's backtest flags were <=5-word transcripts that already exit at skip_reason=transcript_too_short before the classifier ever ran, so shadow mode was on track to observe roughly a fifth of the real catch rate. Compute the verdict once, ahead of that check, and fold it into whichever doc_set already runs (no extra Firestore write). Also add a chatter_classifier_flagged/reason tally spanning both linked calls AND orphans in admin.py's summary block -- the target population is non-events, which land as orphans or single-call incidents, so linked alone undercounts it the same way corr_gate_veto would have without the #126 fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
05ddec8284 |
intelligence: shadow-mode upstream dispatch-vs-chatter classifier (server-26#127)
Three live measurement windows and two consensus-layer fixes (#125, #126) converged on one decision (CORRELATION_REVIEW_0907b.md, _0912.md): stop iterating the correlator's consensus layer, the actual lever is upstream — a classifier in scene extraction that recognizes radio housekeeping (roll call, bare 10-4/10-8/98 acknowledgements, unit check-ins) before it ever becomes a scene for the correlator to judge. Adds app/internal/chatter_classifier.py: a pure classify_chatter(transcript) function recognizing two shapes drawn from hand-labeled examples in the review docs, cross-referenced against the real dumps — not invented regexes. Deliberately conservative: anything that doesn't cleanly reduce to a known shape returns (False, None) and the existing pipeline runs unchanged. SHADOW MODE ONLY. intelligence.extract_scenes computes the verdict next to the existing _is_garbage_transcript / transcript_too_short gates and writes chatter_classifier_verdict / chatter_classifier_reason onto the call doc, but does not skip extraction. admin.py's correlation-debug _call_summary surfaces both fields, same pattern as corr_gate_veto (#115/#126), so the next live window can measure the real-world false-positive rate before anything is wired to actually skip extraction. TODO(server-26#127) marks the call site. Backtest against all three existing dumps (1002 calls): 154 flagged, 0 false positives (no flagged call carries tags, coords, non-routine severity, or matches any review-doc-named dangerous-to-drop transcript — the major extinguishing-fire call, geocoded calls, pursuit updates, the Pelham Station subject check, the property-retrieval call, all individually verified). tests/test_chatter_classifier.py: real transcripts from the dumps/review docs in both directions. Sandboxed pytest 332 -> 364, green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
83beb2bf35 |
admin: surface corr_gate_veto on the correlation-debug endpoint (#115)
corr_gate_veto was written to corr_debug but the admin endpoint's whitelist (_call_summary + the summary tally) never surfaced it, so the last commit's whole point -- measuring window #4 instead of guessing -- would have produced nothing to read. Add it to both. Also softened the docstring's remaining overclaim: whether the active-only ctx[recent] limitation explains the 2/24 window-3 misses is unanswered, not confirmed -- read corr_gate_veto next window instead of asserting a guess again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
598054746a |
correlator: mirror the dispatch/tactical idle split in the escape hatch, record the gate-veto reason (#115)
Review of #126 found: (1) the escape hatch applied tg_dispatch_thin_idle_minutes (5 min) unconditionally, but incident_correlator's own fast/thin path only uses that on dispatch channels and 15 min on tactical ones via _is_dispatch_channel -- mirrored the same selection here, plus a config.py note flagging the second consumer. (2) the docstring claimed a 'confirmed explanation' for 2 window-3 gate misses that was actually wrong (self-contradictory in its own text); replaced the guess with corr_gate_veto, written into corr_debug on every llm=orphan/rules=new disagreement that escalates, so window #4 can see *why* each one escaped instead of reconstructing it from the raw dump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
400b74b519 |
correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)
_recent_incident_on_same_talkgroup previously treated ANY same-talkgroup incident within the 2-hour correlation_window_hours lookback as 'recent', which disabled the whole LLM-orphan consensus gate on busy dispatch channels: window #3 (CORRELATION_REVIEW_0912.md) measured 0/24 gate fires against the exact target shape (rules=new, llm=orphan, tiebreak=new), with 22/24 explained by a same-talkgroup incident existing somewhere in the prior 2h — nearly guaranteed on channels producing 3-13 incidents/2h. Now the escape hatch only counts an incident as recent within settings.tg_dispatch_thin_idle_minutes (5 min), reusing the same recency bound the fast/thin path already uses for the 'dispatch, thin ack 10-30s later' case this hatch exists for, instead of inventing a new constant. Investigated the 2 unexplained misses (no same-tg incident found even by a naive full-collection timestamp scan): confirmed ctx["recent"] is built from status=="active" incidents with over-capacity incidents dropped (_build_context / _drop_capped), not a full collection scan — an incident that has auto-resolved or hit incident_max_calls/incident_max_duration within the window is invisible to this check even though it is chronologically recent. This does not explain the 2 misses (a same-tg incident was absent by both checks there, so some other _call_is_substanceless condition must be responsible), but it is a real gap in the check as written. Documented in the docstring with a TODO(server-26#115); fixing it needs a new, non-active-filtered Firestore query, out of scope for this pass. Tests: added a regression test proving an incident inside the old 2h window but outside the new 5-minute window now correctly gates (fails on main, passes here), plus a test proving a truly recent (<5min) same-tg incident still escapes the gate as intended. Sandboxed pytest: 327 -> 329 passed (2 new tests), all green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
15a9d10666 |
correlator: keep the tiebreak for typed / reassignment calls in the orphan gate (#115)
_call_is_substanceless mirrored has_event_substance but not the creation gate's type-resolved short-circuit, so a routine-severity fire/medical call with no coords/tags/vehicles — or a reassignment (unit pulled to a new job) — could be gated to orphan where rules would open an incident. Bail out of the gate on incident_type or reassignment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
dd426572fc |
correlator: fix consensus orphan-gate to test call substance, not empty corr_debug (#115)
The gate added in
|
||
|
Logan CusanoandClaude Sonnet 5
|
ca1d8fbdae |
correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)
CORRELATION_REVIEW_0907b.md measured that radio housekeeping (unit check-ins, roll call, 10-8/10-98 clearings) is being promoted to incidents. Every case reads corr_llm_action=orphan, corr_rules_action=new, corr_consensus=tiebreak -> new: the cheap LLM correctly reads "not an incident", the rules engine says `new` only because there is no incident to link to, and the smart tiebreaker then sides with rules ~21/21. Reframing the tiebreaker prompt (#116) did nothing. The fix is a consensus-logic gate, not another prompt. Fix 1 (routers/upload.py) - LLM-orphan gate in _correlate_with_consensus: when the cheap LLM says `orphan` and the rules engine says `new` with NO positive event signal, resolve to `orphan` and skip the tiebreak call entirely. "No positive signal" = the rules corr_debug carries neither a positive corr_path (unit-continuity / location / fast/disambig / fast/single) nor a positive corr_fit_signal (unit_overlap / location_proximity). When it does carry one, the existing escalation-to-tiebreak is kept so a genuine event the LLM misreads as orphan still gets the second look. The resolved outcome records corr_consensus="llm_orphan_gate" (greppable, distinct from "tiebreak") and keeps corr_llm_reasoning / corr_rules_action / corr_llm_action populated. Fix 2 (incident_correlator.py) - tighten corr_path=location: the location path linked on a bare sub-location_proximity_km (0.5 km) distance with no unit or content check, which stitched a vehicle lockout to a station-restroom slip and merged two different churches an hour apart. A location link now requires unit overlap with the candidate OR a distance under a tighter bar (_LOCATION_TIGHT_PROXIMITY_KM = 0.2 km). Pursuit incidents keep their movement-speed-validated wide radius. A surviving location link now also writes corr_fit_signal (unit_overlap | location_proximity), consistent with Fix 1's positive-signal set. Tests: new tests/test_consensus_gate.py (13 cases) - the gate resolves to orphan without calling tiebreak on a no-signal disagreement; a unit_overlap / location_proximity / unit-continuity / fast-disambig rules signal still escalates; llm=link vs rules=new still escalates; the location path drops a shared-area candidate with neither unit overlap nor tight proximity, links on unit overlap, and links on tight proximity alone. Full c2-core suite 309 -> 322 passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix |
||
|
Logan CusanoandClaude Sonnet 5
|
ef1e3d7f9d |
correlator: LLM tier reads the scene's transcript, not the whole call (server-26#102)
The last leg of the #80/#95 scene-context leak. llm_correlator._call_block read call_doc's whole-call transcript for every scene, so on a multi-scene call every scene's cheap-tier and tiebreaker decision was made against text that also contained the other scenes. - intelligence.py: each processed[] scene now carries its own "transcript" — transcript_corrected, else this scene's segments joined, else (single scene) the whole transcript. - _build_context / preview_correlation / correlate_call: take a `transcript` param; _build_context resolves ctx["scene_transcript"] from it, falling back to the call doc (sweep, single-scene, tests) — the fallback is kept here, unlike embedding/severity, because a scene always has real text. - upload.py: both scene loops pass scene["transcript"]. - llm_correlator._call_block: reads ctx["scene_transcript"] (call-doc fallback retained for test-built ctx). - recorrelation_sweep: passes the call doc's text explicitly. - +1 regression test. Full c2-core suite green (296 passed, sandboxed venv). NOT for merge until the running correlation measurement window closes and its dump is analysed — deploying a correlator change mid-window would mix old and new behaviour in the sample. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Sonnet 5
|
a9d1d2475a |
correlator: judge each scene on its own embedding + severity (server-26#80, #95)
intelligence.py writes only the primary scene's embedding and severity to
calls/{id}. _build_context read them back off the call doc, so every
non-primary scene of a multi-scene call was correlated against scene 1's
semantic vector and severity rung: a scene about a different event scored
on the embedding path against the wrong incident, and could inherit a
minor/moderate/major severity it never had, clearing the creation gate on
borrowed weight. Same defect and same fix as the #87 coords leak.
- _build_context / preview_correlation / correlate_call: take embedding and
severity as params; drop the call_doc.get() fallbacks. A scene that
passes none has none, and is judged thin on its own signal.
- upload.py: both scene loops pass scene["embedding"] / scene["severity"];
_correlate_with_consensus forwards them. The no-scene unclassified branch
passes neither (correct: no scene, judged thin).
- recorrelation_sweep: passes the call doc's stored values explicitly
(whole-call re-link, link-only, so a borrowed severity cannot create).
- intelligence.py: SCENE DETECTION prompt tightened toward one scene
(server-26#5, partial) - MULTIPLE only for genuinely separate events,
"when unsure, one scene", plus a not-a-new-scene list.
- test_incident_identity.py: +2 regression tests mirroring the #87 test.
Full c2-core suite green (295 passed). #5 prompt change is unmeasured -
needs a scoped correlation-only window. Known remaining legs, tracked
separately: llm_correlator._call_block still reads the whole-call
transcript per scene; content-divergence veto skips on a None embedding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
8b6c170265 |
Close viewer-triggerable OpenAI spend on incident summarize (server-26#81)
POST /incidents/{id}/summarize was gated by require_service_or_firebase_token,
which accepts any authenticated Firebase user including role "viewer". That
route spends OpenAI credits via the background summarizer. The call-side
equivalent was already moved to require_admin_token; this brings the incident
side in line with it.
The frontend's two "summarize now" buttons on the incident detail page are
already gated behind isAdmin, so this backend change matches existing UI
behavior exactly and does not break any viewer/operator surface — it only
closes direct-API access for non-admins.
Swept every other route in incidents.py: list/get are reads with no spend and
correctly stay open to any signed-in user; create/update/delete/link/unlink
were already require_admin_token. No other sibling route needed changing.
Adds test_incident_summarize_auth.py pinning the dependency wiring directly
(the convention used in test_admin_feature_flags.py), so a future revert back
to the weak dependency fails a test immediately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
865b5b4317 |
Close the /admin/features side-door that needed a container shell to flip AI spend
Board minutes #62 Decision 2 (server-26#64), due 2026-08-31. CTO draft #60 finding 1 and CISO draft #61 finding 3 reached this independently. GET/PUT /admin/features accepted only a Firebase admin token, so the unattended runbook had no headless path and SSHed into the c2-core container to write config/ai_features with the admin SDK. Moving a platform-wide AI cost switch required a full container shell, and set_flags() wrote no audit entry either way, so a flag flip was unattributable however it happened. - New agent_service_key (AGENT_SERVICE_KEY), deliberately separate from the Discord bot's service_key. Sharing one key would collapse two principals into a single unattributable identity in every log line, and the bot has no business flipping AI flags regardless. - require_agent_key_or_admin accepts the agent key or a Firebase admin, and rejects the Discord key. The "key is configured" guard is load-bearing: compare_digest("", "") is a match, so a deployment that never set the key would otherwise accept an empty credential. - set_flags() writes an audit_log entry with before/after values and the actor, wrapped so an audit failure cannot lose the flag write or 500 the route. - Cascade helper sets the global doc and every system carrying an ai_flags override in one call. A global False already beats everything, but a system False beats a global True, so turning AI *on* could half-apply and leave a radio system hot after shutoff. It scans for the override rather than hardcoding the two known system IDs, so a new system cannot silently defeat it. - cascade defaults to False. PUT /systems/{id}/ai-flags and the AiFlagsPanel toggle mean a per-system override is deliberate operator intent; cascading by default would erase it on any unrelated global flip. The runbook opts in. Issue items 5 and 6 (retiring the SSH path from drb-worksession.md) are NOT done here and the runbook is untouched. The credential does not exist in production yet, so the SSH path is still the only one that works; retiring it now would break the next unattended run. Owner activation is recorded on #64. Tests 273 -> 289. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Logan Cusano
|
d18e4f0743 |
Make "AI is off" true, and stop the transcript PATCH from destroying calls
config/ai_features was not the switch it was documented to be. Three paths spent money with it off, and one path read it wrong, so per-system opt-outs did not opt anything out. - Correlation in the ingest pipeline tested the raw global flag instead of the per-system resolution. With a system opted out, extraction was skipped but the no-scenes fallback still correlated the call with empty tags, taking the thin/recency path and attaching it to whatever incident was most recent on that system. The opt-out did not disable correlation, it disabled good correlation and left the worst kind running. (#75) - Transcript correction ran on every transcribed call gated only by an env var, spending Gemini tokens and a Places lookup per proposed location. An "STT-only" window was never STT-only and its cost could not be attributed. Now behind transcript_correction_enabled. (#76) - _run_extraction_pipeline and the vocabulary learner, both reachable from PATCH /calls/{id}/transcript, checked no flags at all. (#76, #81) The flag resolver now lives in feature_flags.resolve_flags() rather than as a local helper in upload.py. Three copies of that logic is how #75 happened. PATCH /calls/{id}/transcript now refuses with 409 when correlation is off. That route wipes tags, severity, location, units, embedding and unlinks the call from every incident before queueing re-extraction. Gating extraction alone would have made it destructive-only in the standing flags-off configuration: the call left blank and orphaned forever, with the route still answering 200. The wipe and the rebuild are one transaction in intent, so it refuses before the first write. Also: the summarizer's stale-incident sweep is no longer behind summaries_enabled. It is pure Firestore with no model call in it, and gating it meant nothing auto-resolved while AI was off - so every incident stayed active forever and the candidate set every correlation reads kept growing. transcript_correction_enabled is documented as NOT a pure cost lever. The corrector is also the noise gate that sets not_speech; with it off, recogniser noise reaches extraction as a real transcript, comes back thin, and auto-attaches. Never open an evaluation window with correction off and correlation on. 14 tests added covering flag precedence, both pipeline paths, the 409, the correction gate and the summarizer no-op. Suite: 264 passed. Refs #75, #76, #81, #45. |
||
|
Logan CusanoandClaude Opus 5
|
964343c819 |
area_context v2 + Maps place verification (server-26#36, #37)
#36 — the correction pass shipped in
|
||
|
Logan CusanoandClaude Opus 5
|
58efdbd6eb |
Correct the transcript before anything reads it
Correction existed, but as a line in intelligence.py's EXTRACTION_PROMPT --
which put it in the wrong place twice over. The same model call that extracted
units, location and severity emitted the correction afterwards, so extraction
reasoned over text already known to be wrong; and it sat behind
correlation_enabled, so during a cost-controlled STT-only window nothing was
ever corrected at all. That is the normal state during development.
internal/transcript_correction.py is now its own pass, between the degenerate
filter and the Firestore write. It receives an already-produced transcript plus
a reference list, so unlike a Whisper prompt it has no series to extend -- the
distinction that keeps vocabulary out of the recogniser's prompt, where an
enumerated ten-code list once made it hallucinate ten-code runs.
Reference data is merged from the talkgroup and the system, TALKGROUP FIRST. A
system spanning several counties can have a talkgroup covering one
municipality, and that municipality's streets must not be buried under a
county-wide list. A single-municipality system is the degenerate case: populate
the system level and every talkgroup inherits it. Area context is now SET --
municipality, county, roads, landmarks, on both scopes -- rather than guessed
from talkgroup names, which is what vocabulary_learner did and which is close
to useless across multiple counties.
Segments are corrected too, not just the joined text. extract_scenes builds its
prompt from numbered segments whenever there is more than one, so a correction
that only fixed the transcript would have been discarded on exactly the
multi-transmission calls carrying the most content. Alignment is enforced: an
array of the wrong length or type is dropped whole, because scenes map back to
transmissions by index and a shifted array would misattribute audio silently.
Whisper is also retried once on degenerate output. Call e49ea32c produced a
56-word ten-code counting run on one attempt and ordinary speech on the next --
same clip, same temperature=0 -- so a hallucination is a coin-flip, and
discarding on the first bad roll threw away a recoverable transcript.
Two things found on the way:
PUT /systems/{id} wiped ten_codes on every save. The systems form sends only
{name, type, config}, and model_dump() wrote every omitted field as its default
over the top. Now exclude_unset. area_context would have been the next victim,
which is why it gets its own route alongside ten-codes rather than a field on
that payload.
Closes server-26#36.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
1bfa856d1b |
Serve audio as whatever it actually is
audio/mpeg was hardcoded at both points call audio is written and served, from back when the node produced nothing but 16 kbps MP3. It now uploads FLAC, and a browser will not play a FLAC body labelled audio/mpeg. storage.py grows one extension -> Content-Type map, used by the GCS upload and by /media. Keyed off the object's real extension, so every existing .mp3 recording keeps working with no migration -- and _safe_audio_filename already accepted .flac, so object naming needed nothing. Also flags what this costs: /media sends the whole body with Accept-Ranges: none, which was fine at ~60 KB per call and is not fine at ~1.3 MB/min. Noted at the header and in DEFERRED.md, whose stated reason for deferring Range support was the old file size. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Opus 5
|
140dfbfc74 |
Give the archive a real read, and the debug view a verdict
Three backend pieces the /calls page needs, plus the fix for a debug view that
hid its data exactly when it was wanted.
GET /calls/search — paged, filterable call archive. GET /calls returns every
call in one unordered shot: fine for a node's handful of active calls, useless
as an archive. Only the org scope and the started_at ordering go to Firestore,
since that pair is the one composite index that exists; the rest filters in
Python over a bounded window, the same shape admin.py's debug route uses. The
cursor advances over the scanned window rather than the returned page, or a
sparse filter would re-scan from the same place forever.
Manual attribution. POST /incidents/{id}/calls/{id} only ever wrote the legacy
scalar incident_id, never incident_ids -- which is what the correlator writes
and what the frontend queries with array-contains. A manually attached call was
therefore invisible on the incident page it had just been attached to. It now
maintains both and marks the summary stale. DELETE is new: there was no way to
undo an attachment at all, so a wrong link was permanent.
The debug view no longer filters to AI-enabled systems by default. That filter
emptied the view the moment the flags went off, which is precisely when a
window gets reviewed -- on 2026-08-23 it fell from 100 incidents to 6 between
switching correlation off and opening the tab. ai_systems_only=true restores it.
It also returns a summary block now: corr_path / fit_signal / consensus /
llm_action tallies, transcript coverage on both linked and orphaned calls,
single-call and median-calls-per-incident for fragmentation, max span and
anything past the server-26#22 caps for merging, and the count of incidents
still carrying a fallback "— TGID" title. All of it was being recomputed by
hand from the raw payload on every review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
039a06dc72 |
Let C2 name a talkgroup it already knows
84 of the 100 incidents in the 2026-08-23 dump were titled "Ems — TGID 9048"
or "Other — TGID 9600" -- the fallback, not a description. The title is the
incident's name everywhere it appears: list rows, map pins, Discord alerts.
_create_incident builds it from a content tag and a talkgroup label, and the
label was collapsing to "TGID {id}" because talkgroup_name arrived as None.
It is a plain form field on /upload, forwarded untouched into correlation, and
the node only sends it when OP25 had the name in its loaded tags file -- which
is exactly the case C2 can cover from its own systems collection, where all 125
talkgroup definitions live.
The lookup already existed, on the other path: mqtt_handler resolved it from
the system config on call_start. So the call document held the right name while
the pipeline that titles the incident ignored it. That asymmetry is the bug.
internal/talkgroups.py is now the one implementation -- caller's hint, then the
call document, then the system config -- and both paths use it.
_run_intelligence_pipeline resolves once at the funnel /upload and
/calls/{id}/reprocess share, so the dispatch-channel test, scene extraction and
the title all see a real name. When the call document was the thing missing it,
the resolved name is written back, so the archive and the orphan panel stop
showing a bare TGID too.
Also gives fast/thin a corr_fit_signal. It is 63% of all links and was the only
path writing none, so corr_fit_signal was absent on 295 of 309 calls and the
admin debug view's distribution panel read empty -- looking broken when it was
faithfully reporting that the dominant path records nothing. It now says
thin_recency, which is what actually decided it.
Closes server-26#34. Refs server-26#35 -- the tier's 3.5% invocation rate is a
cost/benefit question, not a bug, and stays open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan Cusano
|
c7be6416f2 |
Surface LLM correlation fields in debug view; fix unit-continuity path
/admin/debug/correlation stripped corr_consensus and the corr_llm_* fields
that upload.py's consensus correlator writes onto the call doc, making it
the one tool built to answer "is the LLM correlation tier alive" unable to
answer it (2026-08-19 dump had to infer LLM state from commit dates instead
of reading it off the data). admin.py's _call_summary() now includes
corr_consensus, corr_llm_reasoning, corr_llm_action, corr_rules_action.
The unit-continuity correlation path never wrote corr_matched_units, unlike
fast/single and fast/disambig, so the debug view showed null for a match
that was in fact unit-driven by construction. Now populated unconditionally
on that path (server-26#16).
Also traced the negative corr_incident_idle_min (-4.1 observed) to its root
cause: the re-correlation sweep anchors `now` to the linking call's own
started_at, and that back-dated value was being written straight into the
incident's updated_at, letting it land before the incident's own
started_at. Added _floor_at_started_at() so updated_at can never precede
started_at. (commit
|
||
|
Logan Cusano
|
5355095c48 |
Compare node API keys in constant time on /upload
/upload compared the per-node API key with a plain !=, which short-circuits on the first differing byte and so leaks a little information about how much of a guess was correct. The reason to fix it is less the timing channel itself -- an HTTP round trip is noisy -- than the inconsistency: enrollment.py and dynsec.py both went out of their way to use secrets.compare_digest for the same class of credential, so the codebase contradicted itself on whether this mattered. Now it does not. Also coalesces a missing api_key field to "" so compare_digest is never handed None, which would raise TypeError and turn a malformed node_keys document into a 500 instead of a 401. Closes logan/server-26#12 |
||
|
Logan Cusano
|
70d63abeaa |
Re-evaluate incident severity on link, stamp resolved_at at every resolution site
#17: severity was written once at _create_incident and never touched again, so an incident that opened routine and escalated to a working fire stayed routine forever. _update_incident now merges call_severity into the incident via _max_severity() on every link. Severity is monotonic: it only ever rises, never falls. An incident briefly assessed "major" genuinely was major at that moment; a later, calmer-sounding call is evidence the situation is winding down, not that the earlier read was wrong. status/resolved_at exist to retire an incident — severity should stay as the high-water mark so the worst-first rail, "Major only" filter, and map colouring never bury a call that was genuinely major. See _max_severity's docstring in incident_correlator.py for the full argument. #18: none of the resolution sites wrote resolved_at, so an incident's lifespan couldn't be reconstructed for the history-scrub feature. Added resolved_at alongside status="resolved" at all six sites that flip it: - incident_correlator.py _update_incident (signal-based: units all cleared) - incident_correlator.py maybe_resolve_parent (master auto-resolve) - summarizer.py _stale_sweep (90-minute auto-resolve) - upload.py, both scene-resolution loops (single- and multi-scene) - calls.py reprocess/correction path (_update_incident's signal-resolve and maybe_resolve_parent's master-resolve weren't named in the issue's four call sites, but they set status the same way and were missing resolved_at too.) No backfill: existing resolved incidents keep resolved_at = null, which means "resolved before this field existed," not "never resolved." Backfilling from updated_at would be a guess dressed up as data. Tests: added to tests/test_correlator_gate.py, which needs no Firestore for the pure _max_severity cases and patches fstore for the _update_incident/ maybe_resolve_parent writes. Covers the escalation case (routine -> major), the no-downgrade case, and resolved_at on both the signal-resolve and master-resolve paths. 52/52 passing in that file; 83 passed / 10 pre-existing failures for drb-c2-core overall (baseline was 69/10 — the +14 is exactly the new tests, no regressions). Fixes #17, #18. |
||
|
Logan CusanoandClaude Opus 5
|
bc191fb59f |
Stop one malformed call document 500ing the whole debug view
/admin/debug/correlation built its call lookup as {doc["call_id"]: doc}, which
raises KeyError on any stored call missing that field -- and at least one in
production is missing it. One bad document took down the entire view rather
than dropping a single call from it.
The document id is authoritative and always present; the call_id *field* is
written by the upload path and evidently has not always been. Keying off the id
we asked for removes the dependency on the field entirely.
Found while generating a correlation dump server-side, because the UI route this
serves has been unusable tonight.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan CusanoandClaude Opus 5
|
90a0412066 |
Bound the correlation debug reads so the view stops hanging
/admin/debug/correlation read every incident ever created, sorted them in Python and kept 20, and separately pulled every call in the orphan window with no cap. That worked while the collections were small. They are not small now: Firestore kills an unbounded scan with a 503 and the request never returns, so the debug view simply spins -- which is also what made the org backfill script fail earlier tonight, same cause, different caller. Incidents now come back pre-sorted from Firestore with a limit, and the orphan scan is capped at 3000 documents. Both queries order on the single field they already filter or sort by (updated_at, ended_at), so neither needs a composite index -- worth preserving, since the index file from the tenancy work has not been deployed. Capping introduces a way to be wrong quietly: a truncated window looks exactly like a quiet night. The payload now carries incidents_window_exhausted and orphan_scan_truncated so a short result announces itself instead of being read as a correlation improvement. The AI-system filter still runs in Python, so the incident window is 10x the requested limit rather than the limit itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Logan CusanoandClaude Opus 5
|
a3681ea698 |
Stamp org_id everywhere and gate every route that leaked across tenants
The previous commit shipped Firestore rules that reference an org_id claim
nothing issues yet, and an org_id filter nothing writes yet - this is the
commit that makes both real. Backend half of SAAS_PLAN.md B2/B2b/B2c.
Data model: organizations/{org_id} and org_members/{uid} are new
collections (models.py OrganizationRecord/OrgMember). org_id is now an
Optional field on NodeRecord, SystemRecord, CallRecord, IncidentRecord,
AlertRule, and AlertEvent - optional because every existing document
predates it; scripts/backfill_org_id.py (written, not run - it touches
production Firestore and Firebase Auth claims) is what closes that gap
later. plan_id/subscription_status/stripe_* on OrganizationRecord are
deliberately None: no billing or pricing model has been decided, so this is
a seam, not a promise. app/internal/tenancy.py holds FOUNDING_ORG_ID, the
org every pre-tenancy document and every legacy enrollment path resolves
into.
Where org_id comes from, end to end: a customer's node enrolls with a
per-org token (new enrollment_tokens/{token_hash} collection, minted via
POST /org/enrollment-tokens - new routers/org.py) instead of the old
fleet-wide ENROLLMENT_TOKEN, which still works as a fallback that resolves
to FOUNDING_ORG_ID so an already-deployed node's .env doesn't start failing
today. The node's org_id then flows onto every call it produces
(mqtt_handler.py's call_start/call_end, upload.py's /upload handler all
resolve it from the node doc), and onto every incident correlated from
those calls (incident_correlator.py's _create_incident/_create_master_incident).
That last one is the part that isn't just a read filter: _build_context's
`all_active = collection_list("incidents", status="active")` fed every
correlation candidate - fast-path talkgroup match, unit-continuity,
disambiguation - from the entire incidents collection, unscoped. Without
scoping it to the call's own org_id, a call from org A could link into an
incident org B already owns, which is a cross-tenant data merge at
correlation time, not just an over-broad read. Same shape of bug in
alerter.py: rule matching pulled every enabled alert_rule regardless of
org, so org A's keyword rule could fire (and POST org A's Discord webhook)
on org B's radio traffic. Both now resolve org_id from the call doc itself
rather than threading a new parameter through every caller.
Every list/get route gained org scoping via a new resolve_caller_org_id()
helper in internal/auth.py, which handles the three credential shapes those
routes accept (service key, node api_key, Firebase user) uniformly and
returns None (unrestricted) for the service key and platform admins -
preserving today's single-org behaviour exactly while closing the leak for
everyone else: GET /nodes, /systems, /calls, /incidents, /alerts,
/alert-rules. Write routes for nodes/systems (approve, create, delete, etc.)
deliberately stay platform-admin-only for now rather than being loosened to
org-owner/operator - that's a real gap called out in SAAS_PLAN.md 2.4's
"should be" column, but it's a separate authorization redesign the 12-item
build order doesn't actually enumerate, and doing it half-considered here
risked being exactly the "half-applied filter is worse than none" failure
mode the plan warns about. Today's founding org keeps working unchanged;
loosening node/system management to org owners is follow-up work, flagged
rather than guessed at.
Also closed the four spend/access-attack routes SAAS_PLAN.md B2c called out
by file and line: POST /calls/{id}/reprocess is now admin-only (was any
signed-in viewer looping the Whisper+Gemini pipeline for free - DEFERRED.md
had this as a live, independent-of-SaaS exploit) plus a per-call rate
limiter as a second guard; POST /alerts/{id}/acknowledge now checks the
alert's org_id; GET /admin/features moved from require_firebase_token to
require_admin_token; and trips.py's four unauthenticated mutation routes
(create_trip, update_trip_tags, create_event, update_event) are now
restricted to the founding org (or the bot's service key, or a platform
admin) - trips has no org_id of its own and isn't getting one, since
[[trips-feature-intentional]] says it's an internal utility riding along on
this stack, not a tenant-scoped product surface.
New public-but-scoped seam: POST /auth/signup (routers/links.py, alongside
the existing /auth/link* routes) provisions an organizations doc and an
owner org_members doc for a just-created Firebase user, then sets their
org_id/org_role claims - idempotent, so a double-submit doesn't create two
orgs. This is the only route that turns "has a Firebase account" into "can
read anything," which is what the frontend AuthProvider no-claim guard
(next commit) is built around.
Also new: GET/PATCH /org for the organization profile (closes the disabled
"Save changes" button noted in DEFERRED.md - there was no organizations
concept to save into before this), and POST /waitlist (public, source-IP
rate-limited, not coupled to any plan or tier - the commercial model is
still an open decision per SAAS_PLAN.md section 6).
Verified: all touched files py_compile clean; c2-core pytest is 69
passed / 10 failed, matching the documented pre-existing baseline exactly
(DEFERRED.md - mqtt_handler/node_sweeper test-vs-code drift, unrelated to
this change) - no new failures. flake8 --max-line-length=120 shows no new
violations in any touched file (checked each new E501/E221/E30x against
`git diff` to confirm it predates this commit); c2-core has no CI lint gate
regardless (CLAUDE.md - flake8 only runs in Client CI).
No new environment variables. Firestore composite indexes for the queries
this introduces were already shipped in the previous commit
(infra/firestore/firestore.indexes.json).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan Cusano
|
97013e1505 |
Stop Whisper hallucinations and dedupe recordings across nodes
Two independent sources of garbage in the AI pipeline, both visible in the 2026-08-16 correlation dump. 1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt text as preceding transcript, so on noisy or silent audio it continued the series, emitting transcripts that count upward from 10-4 to 10-99. The existing no_speech_prob filter could not catch these: the model is highly confident in text it invented by continuing a pattern. The prompt no longer contains a series to extend, and _is_degenerate() rejects the three shapes this failure takes: ascending ten-code runs, one phrase looping, and near-identical segments across a whole recording. Verified against 13 transcripts from production: all four known hallucinations rejected, all nine real ones kept, including terse traffic containing legitimate codes. 2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one dump. Each was transcribed, billed and correlated twice, and the resulting incident listed two units where there was one. Canonical selection is by earliest started_at, tie-broken on call_id, NOT by upload order: upload order varies with encode time and network latency, so it would make the authoritative recording non-deterministic. Call documents are created from MQTT call_start before uploads arrive, so both nodes independently reach the same verdict. The loser keeps its audio (it may be the cleaner capture) but is excluded from STT, correlation, the re-correlation sweep and the orphan debug view. Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is missing, where the caller unpacks two values. A missing key surfaced as a misleading "Transcription failed" instead of the real warning. Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an injected callable so it stays importable without firebase-admin present. |
||
|
Logan Cusano
|
a2cd2c57ca |
Serve call audio through c2-core instead of GCS signed URLs
upload_audio() could only sign a URL when GCP_CREDENTIALS_PATH pointed at a service-account key file. The deployed VM runs on Application Default Credentials with no key file, so every upload silently took the fallback branch and returned a bare gs:// URI. That broke two things at once: * Browsers cannot fetch a gs:// URI, so no recording was ever playable. * _public_url_to_gcs_uri() only matched https://storage.googleapis.com/ and returned None for it, so `if gcs_uri:` in the upload path was always false and transcription never ran. Nothing was logged, which is why this looked like an OpenAI credits problem rather than a storage one. The fallback also interpolated the client-supplied filename instead of the call_id-derived safe name, so the URI did not even name the object written. Calls now store only the canonical gs:// location. A short-lived playback link is minted per read as an HMAC over (call_id, expiry) keyed by SERVICE_KEY, and audio is served from the private bucket by the new /media route. An <audio src> cannot carry an Authorization header, so the link has to be the credential; that router is therefore public with the check done inline, as enrollment.py already does. Signing GCS URLs from the VM would have needed a serviceAccountTokenCreator grant on its own service account — this avoids the IAM change entirely and keeps the bucket private. gcs_uri_for_call() reconstructs the object name from call_id, so recordings made before this fix are reachable again without a data migration. Frontend rows come straight from Firestore via onSnapshot and never see a server-minted field, so CallRow fetches the link lazily on expand. Also removes the last long-lived (1 year) signed URL and the log line that printed it. |
||
|
Logan CusanoandClaude Opus 5
|
ee633cbe46 |
Secure the broker for public exposure: TLS and per-node credentials
Edge nodes are deployed to arbitrary locations by arbitrary people, so the
broker has to be reachable from the internet and secured on its own merits
rather than by a VPN.
Three defects made that impossible. The broker only had a plaintext 1883
listener; every node shared one drb-node password; and the ACL pattern used
%c, the client-supplied client id, so any holder of that shared password
could set client_id to another node and take over its namespace. The comment
claiming this cryptographically prevented cross-node access was wrong and is
gone.
Authentication now uses mosquitto 2.x's built-in dynamic-security plugin on
the stock eclipse-mosquitto image. c2-core administers it over the control
topic, creating each node's client on approval with username=<node_id> and
password=<its node_keys api_key>, attached to a role whose ACL is nodes/%u/#
against the authenticated username. One credential, one revocation point.
An HTTP-callback plugin was implemented first and rejected: that project is
archived upstream, which is not an acceptable dependency on an
internet-facing broker.
Because dynsec state is a second source of truth alongside Firestore,
approve/reissue/delete now write to the broker first and surface a 502
rather than drifting, and c2-core reconciles every approved node into dynsec
on startup.
Adds node self-enrollment (POST /nodes/enroll, GET /nodes/{id}/credentials)
so a new node can obtain its key over HTTPS without an operator handling
secrets by hand. Enrolling an already-approved node_id is refused on the
fleet token alone — otherwise a leaked token plus a guessable id would let
an attacker steal a live node's key before the real node asked for it.
Pickup secrets are stored hashed and returned once, and the endpoint is rate
limited per source IP.
Infrastructure: an 8883 TLS listener fed by Caddy's certificate via a
systemd path unit, a firewall rule for it, and Caddy now 404s /internal/*
so the api vhost cannot proxy internal routes.
Also fixes CORS, which allowed https://app.<domain> while the frontend is
served on the bare domain — every call from the portal would have failed —
and widens the vault gitignore to a glob, since ansible-vault leaves
backup siblings that the exact-name rule left committable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Logan Cusano
|
c42bd1902c | feat: Add local system override with 24h timeout support | ||
|
Logan
|
3defdf18dc | stale calls fix | ||
|
Logan
|
1f17b6c0d2 |
feat: add role-based user management, audit log, and session tracking
Introduces a full user management system with three roles (admin, operator,
viewer), an audit log, and per-session login history.
Backend:
- app/internal/audit.py: write_audit() helper → audit_log Firestore collection
- app/internal/auth.py: get_role() helper; require_admin_token accepts both
legacy admin:true claim and new role:"admin" claim for backward compat
- app/routers/users.py: CRUD under /admin/users — list, create (returns
one-time invite link), get (with sessions), patch role/nodes/name,
disable, enable, delete; operator role requires ≥1 owned node
- app/routers/links.py: POST /auth/session records sign-in events to
user_sessions Firestore collection
- app/routers/admin.py: GET /admin/audit paginated endpoint
- app/main.py: register users router
Frontend:
- AuthProvider: exposes role, isAdmin, isOperator, ownedNodeIds from claims
- Nav: role-gated links — viewers get dashboard/calls/incidents/map/alerts/
trips; operators add nodes/systems/tokens; admins add admin
- admin/page.tsx: new Users tab (list table, create modal, inline edit panel
with role/nodes editor, disable/enable/delete, login history) and Audit
Log tab (paginated, color-coded actions)
- login/page.tsx: calls recordSession() on email and Google sign-in
- nodes, systems, tokens pages: role guards redirect viewers to dashboard
- profile/page.tsx: shows accurate role badge and label
- lib/types.ts: UserRole, UserRecord, UserSession, AuditEntry types
- lib/c2api.ts: user management methods + recordSession
Firestore collections added: user_profiles, audit_log, user_sessions
Firebase custom claims schema: { role, owned_node_ids, admin (legacy) }
|
||
|
Logan
|
961cc6f36e | add button to clear stale 'active' calls | ||
|
Logan
|
6ae4d398f8 | add trips permissions | ||
|
Logan
|
981f03ac06 | allow overlap (note) tags | ||
|
Logan
|
4dd3343026 | add event editing | ||
|
Logan
|
fce189d8c9 | assistant updates |