logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

Admin Replay: re-run the pipeline over past calls in a sandbox, so correlation changes are measured without live AI windows #170

New Issue
Open
opened 2026-09-26 15:15:54 -04:00 by logan · 6 comments
logan commented 2026-09-26 15:15:54 -04:00
Owner
Copy Link
Copy Source

Owner ruling 2026-09-26: correlation work can't keep depending on live AI windows (days of wall time, burns credits, and the 09-20→22 window was invalidated by #169). Recordings are kept regardless of AI, so replay past traffic instead — as a permanent admin section, not a one-off script.

Built on branch feat/replay (Server):

  • app/internal/replay.py — runs a time range of real calls through the live pipeline code (upload._extract_and_correlate, now shared with the live path), original order, clock pinned per call, into replay_runs/{run_id}/calls|incidents (ContextVar redirect in firestore.py). Modes: audio (re-Whisper), transcripts (re-extract), reuse (correlation only, reuses a prior run's scenes — isolates correlator changes from extraction noise). Simulates idle-resolve + recorrelation sweeps on virtual time. No alerts/summaries/vocab. One run at a time, ≤5000 calls, ≤7 days.
  • app/routers/replay.py — /admin/replay estimate/start/list/get/cancel/delete/incidents.
  • New resolved_via field on every incident resolve (units_cleared, llm_closure, reassignment, children_resolved, idle_timeout, emptied_by_*) — also useful live: first time a real clear can be told from a timeout without inference.
  • Frontend: /admin → Replay tab (form + estimate, runs table with comparable metrics, incident drill-down with audio).

Next after merge: replay 09-22 11:30–19:00Z (transcripts mode) once Gemini/OpenAI are funded, as the baseline row. Follow-up worth filing separately: hand-labelled ground truth for one afternoon so runs get a real score, not just shape metrics.

Owner ruling 2026-09-26: correlation work can't keep depending on live AI windows (days of wall time, burns credits, and the 09-20→22 window was invalidated by #169). Recordings are kept regardless of AI, so replay past traffic instead — as a permanent admin section, not a one-off script. Built on branch `feat/replay` (Server): - `app/internal/replay.py` — runs a time range of real calls through the live pipeline code (`upload._extract_and_correlate`, now shared with the live path), original order, clock pinned per call, into `replay_runs/{run_id}/calls|incidents` (ContextVar redirect in `firestore.py`). Modes: audio (re-Whisper), transcripts (re-extract), reuse (correlation only, reuses a prior run's scenes — isolates correlator changes from extraction noise). Simulates idle-resolve + recorrelation sweeps on virtual time. No alerts/summaries/vocab. One run at a time, ≤5000 calls, ≤7 days. - `app/routers/replay.py` — `/admin/replay` estimate/start/list/get/cancel/delete/incidents. - New `resolved_via` field on every incident resolve (units_cleared, llm_closure, reassignment, children_resolved, idle_timeout, emptied_by_*) — also useful live: first time a real clear can be told from a timeout without inference. - Frontend: /admin → Replay tab (form + estimate, runs table with comparable metrics, incident drill-down with audio). Next after merge: replay 09-22 11:30–19:00Z (transcripts mode) once Gemini/OpenAI are funded, as the baseline row. Follow-up worth filing separately: hand-labelled ground truth for one afternoon so runs get a real score, not just shape metrics.
logan commented 2026-09-26 17:57:30 -04:00
Author
Owner
Copy Link
Copy Source

Replay iterations, 09-22 10:00-12:00 ET (290 calls), 2026-09-26:

run SHA incidents 1-call orphans real clears / timeouts
baseline 6e82ee8 19 15.8% 94 0 / 19
clearance fixes f0a88d4 21 19% 107 2 / 19
clearance review 731b54b 19 10.5% 107 1 / 18

Fixed along the way: extraction failures were silent (#172), gemini-2.5-pro gone -> 3.8-flash (6e82ee8), short "45-9, I'm clear." never reached extraction, clearance matched by exact string, junk non-units held incidents open, parser false positives (731b54b).

Findings not fixed:

  1. Unit-clear lifecycle is capped by unit-ID transcription. Parsed clears are correct ("45-9, I'm clear.") but that unit was never recorded as assigned — Whisper renders the same unit as 45-8 / 45-acre / 45 Baker at dispatch. unit_format_hint per system (owner input) is the cheap lever; STT quality is the real one.
  2. Radio-ID (srcaddr) clearance can't carry it. Ch 1 (Patched) 0/83 calls have srcaddr, NYC patch 0/17 (analog patches). P25 channels ~70% coverage; only 4 of 8 clear transmissions there matched an incident's srcaddrs.
  3. Many jobs never say 10-8 on air at all (MDT clears). Proposal: adaptive idle-resolve by severity (e.g. 30/60/90 min) plus reopen-on-link for timeout-resolved incidents, so status tracks reality without depending on a spoken clear.
  4. Channel absorption. Busy single-channel departments (Ch 1): one incident absorbs the channel for an hour (40-call cap) — fast/thin tier-1 links content-free replies to whatever was touched in the last 30s, and unit continuity chains a unit's consecutive jobs. Cross-talkgroup links are minor (12/~180).
Replay iterations, 09-22 10:00-12:00 ET (290 calls), 2026-09-26: | run | SHA | incidents | 1-call | orphans | real clears / timeouts | |---|---|---|---|---|---| | baseline | 6e82ee8 | 19 | 15.8% | 94 | 0 / 19 | | clearance fixes | f0a88d4 | 21 | 19% | 107 | 2 / 19 | | clearance review | 731b54b | 19 | 10.5% | 107 | 1 / 18 | Fixed along the way: extraction failures were silent (#172), gemini-2.5-pro gone -> 3.8-flash (6e82ee8), short "45-9, I'm clear." never reached extraction, clearance matched by exact string, junk non-units held incidents open, parser false positives (731b54b). Findings not fixed: 1. **Unit-clear lifecycle is capped by unit-ID transcription.** Parsed clears are correct ("45-9, I'm clear.") but that unit was never recorded as assigned — Whisper renders the same unit as 45-8 / 45-acre / 45 Baker at dispatch. unit_format_hint per system (owner input) is the cheap lever; STT quality is the real one. 2. **Radio-ID (srcaddr) clearance can't carry it.** Ch 1 (Patched) 0/83 calls have srcaddr, NYC patch 0/17 (analog patches). P25 channels ~70% coverage; only 4 of 8 clear transmissions there matched an incident's srcaddrs. 3. **Many jobs never say 10-8 on air at all** (MDT clears). Proposal: adaptive idle-resolve by severity (e.g. 30/60/90 min) plus reopen-on-link for timeout-resolved incidents, so status tracks reality without depending on a spoken clear. 4. **Channel absorption.** Busy single-channel departments (Ch 1): one incident absorbs the channel for an hour (40-call cap) — fast/thin tier-1 links content-free replies to whatever was touched in the last 30s, and unit continuity chains a unit's consecutive jobs. Cross-talkgroup links are minor (12/~180).
logan commented 2026-09-26 19:50:18 -04:00
Author
Owner
Copy Link
Copy Source

Answer key: hand-labelled 09-22 14:00-16:00Z (25 real incidents) at Version 5C/replay_groundtruth_0922.json, scored pairwise over labelled calls.

run P R F1 found
6e82ee8 baseline 0.825 0.404 0.543 19/25
f0a88d4 clearance 0.878 0.410 0.559 18/25
731b54b clearance review 0.839 0.407 0.548 19/25
e972cac timer+reopen+substantive cap (reuse 731b54b) 0.956 0.698 0.807 19/25

Only ~5/25 real incidents had an audible clear, which is why severity-scaled timer + reopen-on-link (e972cac, review fixes 737bdf0) is the close mechanism, not unit 10-8s.

Remaining misses: MVA tail (15:56 follow-up opened a new incident and absorbed a separate disabled-vehicle job), small traffic stops / DOT cutout never become incidents (creation gate), NYPD patch missing-person traffic fragmented (garbled STT). Not yet validated on a second window — the answer key is one 2h slice, so treat 0.807 as in-sample.

Answer key: hand-labelled 09-22 14:00-16:00Z (25 real incidents) at `Version 5C/replay_groundtruth_0922.json`, scored pairwise over labelled calls. | run | P | R | F1 | found | |---|---|---|---|---| | 6e82ee8 baseline | 0.825 | 0.404 | 0.543 | 19/25 | | f0a88d4 clearance | 0.878 | 0.410 | 0.559 | 18/25 | | 731b54b clearance review | 0.839 | 0.407 | 0.548 | 19/25 | | e972cac timer+reopen+substantive cap (reuse 731b54b) | 0.956 | 0.698 | **0.807** | 19/25 | Only ~5/25 real incidents had an audible clear, which is why severity-scaled timer + reopen-on-link (e972cac, review fixes 737bdf0) is the close mechanism, not unit 10-8s. Remaining misses: MVA tail (15:56 follow-up opened a new incident and absorbed a separate disabled-vehicle job), small traffic stops / DOT cutout never become incidents (creation gate), NYPD patch missing-person traffic fragmented (garbled STT). Not yet validated on a second window — the answer key is one 2h slice, so treat 0.807 as in-sample.
logan commented 2026-09-26 21:23:12 -04:00
Author
Owner
Copy Link
Copy Source

Traffic stops (owner: should show on the portal) + provisional LLM closure:

run P R F1 note
433b35d traffic stops (fresh extraction) 0.786 0.432 0.557 stops appear; MVA split 33/56 by a final LLM closure ("transport complete" at 14:44)
266c958 reuse of 433b35d extraction 0.926 0.715 0.807 LLM closure now reopenable -> MVA one 76-call incident

Stops now opening incidents: 45 Adam (Central Express), CM2 (southbound), plus the West Main / BRP->CCC / Parsons stops.

Measurement note: fresh extraction is noisy — the trespass incident fragmented in 433b35d purely from different extracted units. Compare correlator changes with reuse mode on a fixed extraction; judge extraction changes on their targeted outcome, not one F1 number.

Traffic stops (owner: should show on the portal) + provisional LLM closure: | run | P | R | F1 | note | |---|---|---|---|---| | 433b35d traffic stops (fresh extraction) | 0.786 | 0.432 | 0.557 | stops appear; MVA split 33/56 by a final LLM closure ("transport complete" at 14:44) | | 266c958 reuse of 433b35d extraction | 0.926 | 0.715 | **0.807** | LLM closure now reopenable -> MVA one 76-call incident | Stops now opening incidents: 45 Adam (Central Express), CM2 (southbound), plus the West Main / BRP->CCC / Parsons stops. **Measurement note:** fresh extraction is noisy — the trespass incident fragmented in 433b35d purely from different extracted units. Compare correlator changes with reuse mode on a fixed extraction; judge extraction changes on their targeted outcome, not one F1 number.
logan commented 2026-09-27 01:52:41 -04:00
Author
Owner
Copy Link
Copy Source

Second window validates the in-sample number. Sat 09-21 18:00-20:00 ET (204 calls), hand-labelled blind before viewing the system grouping: Version 5C/replay_groundtruth_0921.json (23 real incidents).

window run P R F1 found
09-22 10-12 ET (in-sample) 65705bf reuse 0.941 0.724 0.818 20/25
09-21 18-20 ET (held-out) 65705bf fresh 0.883 0.787 0.833 17/23

Held-out misses: Ossining Post 4 plate-read stops ("Post 4. 52-62... Frank David Boy 4514" — a stop phrased as a plate read, not "on a stop"), a harassment case-number request, a DEC/MTA ID check, and two small Dobbs/Ch1 calls. The knife-menacing (17 calls) and Pine & Lincoln detention (9 calls) incidents were tracked end to end.

Gemini accounting (65705bf, reuse run, 09-22 window): correlation 3.6-flash 124 calls in 211k/out 9k/thinking 0; tiebreak 3.8-flash 60 calls in 105k/out 4k/thinking 39k.

**Second window validates the in-sample number.** Sat 09-21 18:00-20:00 ET (204 calls), hand-labelled blind before viewing the system grouping: `Version 5C/replay_groundtruth_0921.json` (23 real incidents). | window | run | P | R | F1 | found | |---|---|---|---|---|---| | 09-22 10-12 ET (in-sample) | 65705bf reuse | 0.941 | 0.724 | 0.818 | 20/25 | | 09-21 18-20 ET (held-out) | 65705bf fresh | 0.883 | 0.787 | **0.833** | 17/23 | Held-out misses: Ossining Post 4 plate-read stops ("Post 4. 52-62... Frank David Boy 4514" — a stop phrased as a plate read, not "on a stop"), a harassment case-number request, a DEC/MTA ID check, and two small Dobbs/Ch1 calls. The knife-menacing (17 calls) and Pine & Lincoln detention (9 calls) incidents were tracked end to end. Gemini accounting (65705bf, reuse run, 09-22 window): correlation 3.6-flash 124 calls in 211k/out 9k/thinking 0; tiebreak 3.8-flash 60 calls in 105k/out 4k/thinking 39k.
logan commented 2026-09-27 11:01:27 -04:00
Author
Owner
Copy Link
Copy Source

Plate reads -> traffic stops (e90a73f), both windows, fresh extraction:

window P R F1 found vs before
09-21 18-20 ET (held-out) 0.937 0.810 0.869 18/23 0.833
09-22 10-12 ET 0.947 0.695 0.801 19/25 0.818 reuse (within fresh-extraction noise)

Ossining Post 4 plate-read stops (Hemlock, Ramapo) now open incidents. Nit seen: one stop titled "Traffic Stop at 96 times 5" (a ten-code signal extracted as a location).

Plate reads -> traffic stops (e90a73f), both windows, fresh extraction: | window | P | R | F1 | found | vs before | |---|---|---|---|---|---| | 09-21 18-20 ET (held-out) | 0.937 | 0.810 | **0.869** | 18/23 | 0.833 | | 09-22 10-12 ET | 0.947 | 0.695 | 0.801 | 19/25 | 0.818 reuse (within fresh-extraction noise) | Ossining Post 4 plate-read stops (Hemlock, Ramapo) now open incidents. Nit seen: one stop titled "Traffic Stop at 96 times 5" (a ten-code signal extracted as a location).
logan commented 2026-10-04 01:15:58 -04:00
Author
Owner
Copy Link
Copy Source

Live-AI window, measured 2026-10-04. AI went dry on billing (#196): Gemini correlation from 09-29 15:07Z, OpenAI STT from 09-30 12:15Z. Healthy window is 09-27 15Z → 09-29 15:07Z. The 09-29 15:08Z → 09-30 12:15Z stretch is a free control: same code, extraction alive, correlation rules-only. Data: live call docs via /calls/search + /incidents/{id}, slim dump Version 5C/corr_dump_2026-10-04.json. debug/correlation can't reach this far back (#197).

segment hrs incidents/hr 1-call median / max calls orphan % (transcribed) traffic-stop incidents
A healthy, pre-#189 (6658c26) 12 6.7 32.5% 3 / 40 39.5 7
B healthy, #189 prompt (e236409) 36 10.5 32.2% 3 / 37 49.4 40
C rules-only control 21 19.2 36.8% 2 / 33 47.1 38

Reading:

  1. The LLM tier earns its keep. Rules-only creates about 1.8x the incidents per hour (19.2 vs 10.5). It also has 2.4x the same-location incident pairs within 45 min (17 vs 7, a rough over-split heuristic).
  2. The #189 sink didn't recur after e009090. Biggest B incidents: 37 calls/75 min (9A MVA), 34/120 (CIP), 27/102 (disabled train). None as large as the 40-call/2h34m Rollover in A.
  3. Single-call incidents are 32%, against 10–19% in replay. But 65 of 148 singletons (A+B) are NYPD Citywide 2 Patch (8026), which is 59% singletons and mostly one-shot NYC fragments. Excluding 8026, the singleton rate is 24%. Separately, 47 incidents (10%) have generic "Police — " titles with no type or location; 23 of them are MTA 6/7/11.
  4. Traffic stops do show as incidents (owner ruling met): 47 in A+B.
  5. Orphans in B: 1,375 transcribed calls. 52% are under 40 characters (chatter, correctly orphaned) and 189 are over 120 characters (candidates for missed links).
  6. Scene corr_path A+B: none 1891, fast/thin 928, new 460, disambig 120, unit-continuity 28. fast/thin is still about 60% of links (#134).

No live F1. There's no hand-labelled live window, and replay can't run while AI is dry. So better/worse vs 0.80–0.87 is unproven: the shape is consistent with replay, low confidence.

Instrument check: #96 is closed and these use the per-scene scenes map, so corr_path counts are clean. #155 (units_cleared rarely populates) taints closure stats only: resolved_via is ~90% idle_timeout. #169 doesn't apply, since the LLM answered in about 45% of healthy scene decisions. Traffic-stop and same-location counts are title/location regex heuristics.

**Live-AI window, measured 2026-10-04.** AI went dry on billing (#196): Gemini correlation from **09-29 15:07Z**, OpenAI STT from **09-30 12:15Z**. Healthy window is 09-27 15Z → 09-29 15:07Z. The 09-29 15:08Z → 09-30 12:15Z stretch is a free control: same code, extraction alive, correlation rules-only. Data: live call docs via `/calls/search` + `/incidents/{id}`, slim dump `Version 5C/corr_dump_2026-10-04.json`. debug/correlation can't reach this far back (#197). | segment | hrs | incidents/hr | 1-call | median / max calls | orphan % (transcribed) | traffic-stop incidents | |---|---|---|---|---|---|---| | A healthy, pre-#189 (6658c26) | 12 | 6.7 | 32.5% | 3 / 40 | 39.5 | 7 | | B healthy, #189 prompt (e236409) | 36 | 10.5 | 32.2% | 3 / 37 | 49.4 | 40 | | C rules-only control | 21 | 19.2 | 36.8% | 2 / 33 | 47.1 | 38 | Reading: 1. **The LLM tier earns its keep.** Rules-only creates about 1.8x the incidents per hour (19.2 vs 10.5). It also has 2.4x the same-location incident pairs within 45 min (17 vs 7, a rough over-split heuristic). 2. **The #189 sink didn't recur after e009090.** Biggest B incidents: 37 calls/75 min (9A MVA), 34/120 (CIP), 27/102 (disabled train). None as large as the 40-call/2h34m Rollover in A. 3. **Single-call incidents are 32%, against 10–19% in replay.** But 65 of 148 singletons (A+B) are NYPD Citywide 2 Patch (8026), which is 59% singletons and mostly one-shot NYC fragments. Excluding 8026, the singleton rate is 24%. Separately, 47 incidents (10%) have generic "Police — <talkgroup>" titles with no type or location; 23 of them are MTA 6/7/11. 4. Traffic stops do show as incidents (owner ruling met): 47 in A+B. 5. Orphans in B: 1,375 transcribed calls. 52% are under 40 characters (chatter, correctly orphaned) and 189 are over 120 characters (candidates for missed links). 6. Scene corr_path A+B: none 1891, fast/thin 928, new 460, disambig 120, unit-continuity 28. fast/thin is still about 60% of links (#134). **No live F1.** There's no hand-labelled live window, and replay can't run while AI is dry. So **better/worse vs 0.80–0.87 is unproven**: the shape is consistent with replay, low confidence. **Instrument check:** #96 is closed and these use the per-scene `scenes` map, so corr_path counts are clean. #155 (units_cleared rarely populates) taints closure stats only: resolved_via is ~90% idle_timeout. #169 doesn't apply, since the LLM answered in about 45% of healthy scene decisions. Traffic-stop and same-location counts are title/location regex heuristics.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
 minutes:draft
One exec pre-read, closed by CEO
 minutes:final
CEO ruling; open until enacted
 role:ceo
Filed by the CEO
 role:cfo
Filed by the CFO
 role:cio
Filed by the CIO
 role:ciso
Filed by the CISO
 role:cmo
Filed by the CMO
 role:coo
Filed by the COO
 role:cto
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/correlation area/frontend
priority
p1
status
in-progress
type
feature
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#170