logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

P0: standing correlation-quality evaluation procedure (scheduled AI window) #57

New Issue
Closed
opened 2026-08-24 23:03:42 -04:00 by logan · 3 comments
logan commented 2026-08-24 23:03:42 -04:00
Owner
Copy Link
Copy Source

P0 — standing procedure, runs every evening session.

Correlation quality is the product. The unit of value is the INCIDENT, not the call, so
correlation gets a measured evaluation on a schedule rather than only when something looks wrong.

The full runbook lives at Version 5C/.claude/scheduled/drb-worksession.md (STEP 5) and is
executed by the twice-daily scheduled work session. This issue is the tracker for it.

Procedure

  1. Static pass, every run — drb-correlation-review over incident_correlator.py,
    llm_correlator.py, intelligence.py, and the upload router scene loop.
  2. Live AI window, evening run only, and only if a correlation-affecting commit is live on
    /health and has not yet been evaluated against real traffic. Never open a window "just to look".
  3. Arm a server-side failsafe first (sleep 4200 + /opt/drb/ai_off.py) so a crashed agent
    cannot leave the window open. If arming fails, do not open the window.
  4. Turn on stt_enabled + correlation_enabled at BOTH levels — global config/ai_features
    and the per-system ai_flags on Suffolk a342a757-8c1a-460d-b857-252d2c5e6e94 and Westchester
    f3bc494e-0951-4a61-b16b-7adc5eb157b1, which survive a global off-switch. Record the prior
    values of summaries_enabled / vocabulary_learning_enabled and restore them after.
  5. Wait ~60 min, polling in 9-minute increments (a single long sleep exceeds the tool timeout).
  6. Turn off and verify the read-back shows False, then kill the failsafe.
  7. Dump and review — debug_correlation(limit=100, orphan_hours=2) in-process, saved as
    Version 5C/correlation-logs-<date>.json so runs are comparable. Trend beats snapshot.

No browser required

PUT /admin/features needs a Firebase admin ID token (require_admin_token in
app/internal/auth.py does not accept the service key), which is why earlier automation needed
Chrome. Writing Firestore directly from inside the c2-core container over SSH avoids the HTTP
route entirely and is fully headless.

What each window must answer

  • Over-splitting rate (#5) — same scene opening two incidents.
  • Over-merging — one incident absorbing a whole talkgroup.
  • Orphan count, and whether orphans are still all sweep-"exhausted" (#27).
  • corr_path / fit_signal distribution, and whether corr_consensus / corr_llm_action populate.
  • Whether map pins land on verified places (#28).
  • tier is DISABLED in the logs — means a model ID or the billing died again. Highest-priority finding.

Related: #5, #27, #28, #35, #6.

Follow-up worth doing

/admin/features accepts only a Firebase admin token. Switching it to
require_service_key_or_admin would make automated toggling first-class instead of a
Firestore side-door. Not required for the procedure above — noted so the side-door is a
deliberate choice rather than an accident.

**P0 — standing procedure, runs every evening session.** Correlation quality is the product. The unit of value is the INCIDENT, not the call, so correlation gets a measured evaluation on a schedule rather than only when something looks wrong. The full runbook lives at `Version 5C/.claude/scheduled/drb-worksession.md` (STEP 5) and is executed by the twice-daily scheduled work session. This issue is the tracker for it. ### Procedure 1. **Static pass, every run** — `drb-correlation-review` over `incident_correlator.py`, `llm_correlator.py`, `intelligence.py`, and the upload router scene loop. 2. **Live AI window, evening run only, and only if** a correlation-affecting commit is live on `/health` and has not yet been evaluated against real traffic. Never open a window "just to look". 3. **Arm a server-side failsafe first** (`sleep 4200` + `/opt/drb/ai_off.py`) so a crashed agent cannot leave the window open. If arming fails, do not open the window. 4. **Turn on** `stt_enabled` + `correlation_enabled` at BOTH levels — global `config/ai_features` and the per-system `ai_flags` on Suffolk `a342a757-8c1a-460d-b857-252d2c5e6e94` and Westchester `f3bc494e-0951-4a61-b16b-7adc5eb157b1`, which survive a global off-switch. Record the prior values of `summaries_enabled` / `vocabulary_learning_enabled` and restore them after. 5. **Wait ~60 min**, polling in 9-minute increments (a single long sleep exceeds the tool timeout). 6. **Turn off and verify the read-back shows False**, then kill the failsafe. 7. **Dump and review** — `debug_correlation(limit=100, orphan_hours=2)` in-process, saved as `Version 5C/correlation-logs-<date>.json` so runs are comparable. Trend beats snapshot. ### No browser required `PUT /admin/features` needs a Firebase admin ID token (`require_admin_token` in `app/internal/auth.py` does not accept the service key), which is why earlier automation needed Chrome. Writing Firestore directly from inside the `c2-core` container over SSH avoids the HTTP route entirely and is fully headless. ### What each window must answer - Over-splitting rate (#5) — same scene opening two incidents. - Over-merging — one incident absorbing a whole talkgroup. - Orphan count, and whether orphans are still all sweep-"exhausted" (#27). - `corr_path` / `fit_signal` distribution, and whether `corr_consensus` / `corr_llm_action` populate. - Whether map pins land on verified places (#28). - `tier is DISABLED` in the logs — means a model ID or the billing died again. Highest-priority finding. Related: #5, #27, #28, #35, #6. ### Follow-up worth doing `/admin/features` accepts only a Firebase admin token. Switching it to `require_service_key_or_admin` would make automated toggling first-class instead of a Firestore side-door. Not required for the procedure above — noted so the side-door is a deliberate choice rather than an accident.
logan commented 2026-08-25 21:31:40 -04:00
Author
Owner
Copy Link
Copy Source

Window run, 2026-08-25 evening (unattended run). Recording it here since this issue owns the procedure.

Trigger: six correlation-affecting commits landed on main between 2026-08-20 and 2026-08-23 (33a247d…cc038e6, touching incident_correlator.py, intelligence.py, llm_correlator.py). All postdate the CORRELATION_REVIEW_0820.md baseline and none had been evaluated against live traffic.

Procedure as actually executed, which is closer to what this issue should specify than what it currently says:

  1. A server-side failsafe was armed before anything was turned on — a detached 70-minute timer on the server that closes the window regardless of whether the agent survives. Arming it is a precondition; if it fails, the window does not open.
  2. Flags set at both the global config/ai_features level and the per-system ai_flags level, stt_enabled and correlation_enabled only.
  3. summaries_enabled and vocabulary_learning_enabled were already false and were deliberately left untouched, so there is nothing to restore and no learned-state confound in the results.
  4. Read-back confirmed before starting the clock.

Two mechanical notes worth folding into the written procedure:

  • The scripts live at /opt/drb/aiwin/ on the server and are run via compose run --rm --no-deps with PYTHONPATH=/app and the directory bind-mounted. There is no bind mount of the app source into the running c2-core container in either compose file, so the exec -T c2-core python /app/ai_off.py form assumed by the runbook cannot work as written — the runbook is wrong on that point and should be corrected.
  • feature_flags caches for 30s but per-system docs are read through doc_get_cached with a 300s TTL. Turning a system on can therefore take up to five minutes to take effect. Turning off is fast, because the global flag is the master switch and beats the per-system value.

Also: tier is DISABLED is emitted once per model per process (llm_correlator.py:296). A container restart re-arms it, so the absence of that line after a restart proves nothing. Any future run of this procedure should record the c2-core container's start time alongside the log grep, or the check is not evidence.

**Window run, 2026-08-25 evening (unattended run).** Recording it here since this issue owns the procedure. Trigger: six correlation-affecting commits landed on `main` between 2026-08-20 and 2026-08-23 (`33a247d`…`cc038e6`, touching `incident_correlator.py`, `intelligence.py`, `llm_correlator.py`). All postdate the `CORRELATION_REVIEW_0820.md` baseline and none had been evaluated against live traffic. Procedure as actually executed, which is closer to what this issue should specify than what it currently says: 1. A server-side failsafe was armed **before** anything was turned on — a detached 70-minute timer on the server that closes the window regardless of whether the agent survives. Arming it is a precondition; if it fails, the window does not open. 2. Flags set at both the global `config/ai_features` level and the per-system `ai_flags` level, `stt_enabled` and `correlation_enabled` only. 3. `summaries_enabled` and `vocabulary_learning_enabled` were already `false` and were deliberately left untouched, so there is nothing to restore and no learned-state confound in the results. 4. Read-back confirmed before starting the clock. Two mechanical notes worth folding into the written procedure: - The scripts live at `/opt/drb/aiwin/` on the server and are run via `compose run --rm --no-deps` with `PYTHONPATH=/app` and the directory bind-mounted. There is **no** bind mount of the app source into the running `c2-core` container in either compose file, so the `exec -T c2-core python /app/ai_off.py` form assumed by the runbook cannot work as written — the runbook is wrong on that point and should be corrected. - `feature_flags` caches for 30s but per-system docs are read through `doc_get_cached` with a **300s** TTL. Turning a system *on* can therefore take up to five minutes to take effect. Turning off is fast, because the global flag is the master switch and beats the per-system value. Also: `tier is DISABLED` is emitted **once per model per process** (`llm_correlator.py:296`). A container restart re-arms it, so the absence of that line after a restart proves nothing. Any future run of this procedure should record the c2-core container's start time alongside the log grep, or the check is not evidence.
logan commented 2026-09-01 02:59:24 -04:00
Author
Owner
Copy Link
Copy Source

First saved correlation dump under this procedure. Unattended run 2026-09-01, against live b722223. Saved as Version 5C/correlation-logs-2026-09-01.json — it is the only one on disk, so this run establishes the baseline and there is no trend to read yet. Every future run compares to this.

Taken with no AI window opened — this is a read of existing state, not spend. AI stayed off throughout.

The numbers

Measure Value
Incidents 100
Linked calls 187
Orphaned calls 22 (orphan_scan_truncated: false)
Single-call incidents 58 of 100
Median calls/incident 1 · max 7 · max span 25.6 min
Incidents over cap 0 (caps: 40 calls / 120 min)
Incident types police 70 · other 14 · ems 11 · accident 3 · fire 2

corr_path: new 101 (54.0%) · fast/thin 56 (29.9%) · fast/disambig 21 (11.2%) · fast/single 4 · unit-continuity 4 · location 1.

corr_fit_signal: none 150 · unit_overlap 20 · thin_recency 12 · location_proximity 5.

corr_consensus: tiebreak 60 · agreed 55 · rules_only 35 · none 37. corr_llm_action: none 127 · orphan 49 · link 11.

Verdicts against the standing failure modes

  • Over-merge: not present. Max 7 calls in one incident, zero incidents at either cap, max span 25.6 minutes against a 120-minute ceiling. Nothing is absorbing a talkgroup. The mechanism-B feedback loop from CORRELATION_REVIEW_0820.md is not firing in this window.
  • Over-split (#5): not detected. Zero calls appear in more than one incident. Note this probe cannot see the scene-level over-split #5 actually describes, only the call-level consequence — and per #96 the per-scene record is unreliable, so absence of evidence is weak here.
  • Orphans: 22, and the interesting part is why. orphans_with_transcript: 0 — every single orphan has no transcript. These are not correlation failures at all; correlation was never given anything to work with. That reframes the orphan problem as a transcription-coverage problem and points at #7 (bulk reprocess calls stored without a transcript), not at the correlator. linked_calls_without_transcript is 6.
  • Pins: 53 of 100 incidents carry a location field, 47 carry none. fallback_titled_incidents: 26 — a quarter of incidents have no real title.
  • The LLM tier is doing far more than #35 says. 60 of 187 links (32%) have a populated corr_llm_action, and consensus shows 60 tiebreaks. #35 states 3.5%. Do not treat this as a refutation of #35 — per #96 both figures are computed off a per-call blend of scene decisions, so the honest statement is that the instrument disagrees with itself. What is solid: the tier is being consulted, is not disabled, and votes orphan 49 times against link 11 — it is acting predominantly as a brake on linking, not a linker.
  • No tier is DISABLED condition. Model IDs and billing are alive.

Procedure defect found while running this

The dump could not be taken the way this issue specifies. docker compose run on the VM resolves a stale image, so it executed code older than production and died on a signature change. Filed as #101 — the serious part is that non-signature drift would have produced clean, wrong numbers instead of an error. This dump was taken against the live container instead and is trustworthy; earlier dumps taken via run are of unverified provenance. #57's procedure text needs updating before the next run.

**First saved correlation dump under this procedure. Unattended run 2026-09-01, against live `b722223`.** Saved as `Version 5C/correlation-logs-2026-09-01.json` — it is the only one on disk, so this run establishes the **baseline** and there is no trend to read yet. Every future run compares to this. Taken with **no AI window opened** — this is a read of existing state, not spend. AI stayed off throughout. ## The numbers | Measure | Value | |---|---| | Incidents | 100 | | Linked calls | 187 | | Orphaned calls | 22 (`orphan_scan_truncated: false`) | | Single-call incidents | 58 of 100 | | Median calls/incident | 1 · max **7** · max span **25.6 min** | | Incidents over cap | **0** (caps: 40 calls / 120 min) | | Incident types | police 70 · other 14 · ems 11 · accident 3 · fire 2 | `corr_path`: new 101 (54.0%) · fast/thin 56 (29.9%) · fast/disambig 21 (11.2%) · fast/single 4 · unit-continuity 4 · location 1. `corr_fit_signal`: none 150 · unit_overlap 20 · thin_recency 12 · location_proximity 5. `corr_consensus`: tiebreak 60 · agreed 55 · rules_only 35 · none 37. `corr_llm_action`: none 127 · **orphan 49 · link 11**. ## Verdicts against the standing failure modes - **Over-merge: not present.** Max 7 calls in one incident, zero incidents at either cap, max span 25.6 minutes against a 120-minute ceiling. Nothing is absorbing a talkgroup. The mechanism-B feedback loop from `CORRELATION_REVIEW_0820.md` is not firing in this window. - **Over-split (#5): not detected.** Zero calls appear in more than one incident. Note this probe cannot see the *scene*-level over-split #5 actually describes, only the call-level consequence — and per #96 the per-scene record is unreliable, so absence of evidence is weak here. - **Orphans: 22, and the interesting part is why.** `orphans_with_transcript: 0` — **every single orphan has no transcript.** These are not correlation failures at all; correlation was never given anything to work with. That reframes the orphan problem as a transcription-coverage problem and points at #7 (bulk reprocess calls stored without a transcript), not at the correlator. `linked_calls_without_transcript` is 6. - **Pins:** 53 of 100 incidents carry a location field, 47 carry none. `fallback_titled_incidents: 26` — a quarter of incidents have no real title. - **The LLM tier is doing far more than #35 says.** 60 of 187 links (**32%**) have a populated `corr_llm_action`, and consensus shows 60 tiebreaks. #35 states 3.5%. **Do not treat this as a refutation of #35** — per #96 both figures are computed off a per-call blend of scene decisions, so the honest statement is that the instrument disagrees with itself. What is solid: **the tier is being consulted, is not disabled, and votes `orphan` 49 times against `link` 11** — it is acting predominantly as a brake on linking, not a linker. - **No `tier is DISABLED` condition.** Model IDs and billing are alive. ## Procedure defect found while running this The dump could not be taken the way this issue specifies. `docker compose run` on the VM resolves a **stale image**, so it executed code older than production and died on a signature change. Filed as **#101** — the serious part is that non-signature drift would have produced clean, wrong numbers instead of an error. This dump was taken against the live container instead and is trustworthy; earlier dumps taken via `run` are of unverified provenance. **#57's procedure text needs updating before the next run.**
logan added the area/correlation
priority
p0
type
tracking
 labels 2026-10-05 02:04:31 -04:00
logan commented 2026-10-27 19:27:09 -04:00
Author
Owner
Copy Link
Copy Source

Closing (2026-10-27): this depended on the twice-daily scheduled session, which has been dead since the Linux move, plus live AI windows. Admin Replay (#170) replaces it as the way to measure correlation.

Closing (2026-10-27): this depended on the twice-daily scheduled session, which has been dead since the Linux move, plus live AI windows. Admin Replay (#170) replaces it as the way to measure correlation.
logan closed this issue 2026-10-27 19:27:10 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
 minutes:draft
One exec pre-read, closed by CEO
 minutes:final
CEO ruling; open until enacted
 role:ceo
Filed by the CEO
 role:cfo
Filed by the CFO
 role:cio
Filed by the CIO
 role:ciso
Filed by the CISO
 role:cmo
Filed by the CMO
 role:coo
Filed by the COO
 role:cto
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/correlation
priority
p0
type
tracking
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#57