Incidents never actually clear — units_cleared almost never populates, 90-min idle sweep does the resolving instead #155

New Issue

2026-09-20T14:33:01-04:00

logan commented

2026-09-20 14:33:01 -04:00

Found while auditing whether incident correlation work has actually improved incident lifecycle tracking (dispatch → final unit 10-8), not just junk/duplicate suppression.

Evidence (5 correlation debug windows, 281 in-window incidents, 1721 calls scanned): only 2/281 incidents ever had a non-empty units_cleared. Of ~183 calls with a clearance-like phrase in the transcript, only ~19 produced a cleared_units extraction. incident_correlator.py's resolve-on-clear condition (units_cleared and not units_active → resolved) essentially never fires — 183/281 incidents in-window were instead resolved by the 90-minute idle sweep (summarizer.py), i.e. incidents time out, they don't get marked cleared by a real event.

Root cause — two independent mechanisms, not one (confirmed against corr_dump_9-12/9-13/9-14.json):

Pattern B — re-dispatch accept, no explicit 10-8 (dispatch: "are you able to clear and take a run at X" / unit: "10-4"). reassignment=True already fired correctly and already suppressed the unit from re-linking to its prior incident (upload.py sets corr_units=[] on reassignment) — but nothing ever released the unit FROM that prior incident. It just sat "active" until the idle sweep. Structural gap, not an extraction miss.
Pattern A — self-clear, inconsistent extraction. No per-system unit-ID format awareness existed anywhere (vocabulary_learner.py's "known local terms" is a flat glossary, not a structured format) — departments use incompatible conventions (Yorktown: 5-David, sometimes spoken as bare David; County: SAM-1/airport-3/parks-4, a location word + number) with zero shared pattern. Also, intelligence.py's cleared_units prompt rule only accepted a unit self-reporting, missing dispatch confirming a unit's status back to them (e.g. unit asks "how do you show me", dispatch replies "showing you available") — confirmed misses where a known unit ID sits directly next to a clear phrase matching the prompt's own example pattern and still got dropped.

Fixed in fb0bb15 (pushed to main, deploying now):

_release_reassigned_units (incident_correlator.py): on reassignment=True, scans other active incidents for unit overlap and releases the unit there via the same units_active/units_cleared merge explicit-10-8 extraction uses (factored out as _apply_unit_clearance).
system.unit_format_hint (owner-authored free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed into the extraction prompt; broadened the cleared_units rule to accept dispatch-confirmed availability while still requiring an identifiable unit ID (guards against bare 10-8/clear noise and Whisper-hallucination runs already caught upstream by _is_garbage_transcript).

Tests: 401 pass, 0 fail (verified against a Linux venv this session set up at ~/venvs/drb-5c — note added to CLAUDE.md's testing-reality section since the documented Windows-venv path isn't the only option going forward).

Not yet done: owner needs to populate unit_format_hint for the systems actually in use (Yorktown, County) via the new PUT route — no default value, no UI field yet (API-only for now, consistent with how ten-codes/area-context started). Auto-induction of unit-ID format (mirroring vocabulary_learner's induction loop) deliberately deferred — bigger lever, not needed yet for a 2-3-system personal-use scope.

Left open pending the post-deploy live review: plan is to measure cleared_units/units_active population and the reassignment-release path against real traffic for a 2-hour window post-deploy, then reassess before closing.

Related but distinct: #88 (resolved/reassignment are prompt-only invariants with no validator — a parsing-robustness gap, not this lifecycle gap).

Filed by Claude Code following up on a correlation-review audit, 2026-09-20.

**Found while auditing whether incident correlation work has actually improved incident lifecycle tracking (dispatch → final unit 10-8), not just junk/duplicate suppression.** Evidence (5 correlation debug windows, 281 in-window incidents, 1721 calls scanned): only 2/281 incidents ever had a non-empty `units_cleared`. Of ~183 calls with a clearance-like phrase in the transcript, only ~19 produced a `cleared_units` extraction. `incident_correlator.py`'s resolve-on-clear condition (`units_cleared and not units_active → resolved`) essentially never fires — 183/281 incidents in-window were instead resolved by the 90-minute idle sweep (`summarizer.py`), i.e. incidents time out, they don't get marked cleared by a real event. **Root cause — two independent mechanisms, not one** (confirmed against corr_dump_9-12/9-13/9-14.json): 1. **Pattern B — re-dispatch accept, no explicit 10-8** (dispatch: "are you able to clear and take a run at X" / unit: "10-4"). `reassignment=True` already fired correctly and already suppressed the unit from re-linking to its prior incident (`upload.py` sets `corr_units=[]` on reassignment) — but nothing ever released the unit FROM that prior incident. It just sat "active" until the idle sweep. Structural gap, not an extraction miss. 2. **Pattern A — self-clear, inconsistent extraction.** No per-system unit-ID format awareness existed anywhere (`vocabulary_learner.py`'s "known local terms" is a flat glossary, not a structured format) — departments use incompatible conventions (Yorktown: `5-David`, sometimes spoken as bare `David`; County: `SAM-1`/`airport-3`/`parks-4`, a location word + number) with zero shared pattern. Also, `intelligence.py`'s `cleared_units` prompt rule only accepted a unit self-reporting, missing dispatch confirming a unit's status back to them (e.g. unit asks "how do you show me", dispatch replies "showing you available") — confirmed misses where a known unit ID sits directly next to a clear phrase matching the prompt's own example pattern and still got dropped. **Fixed in `fb0bb15`** (pushed to main, deploying now): 1. `_release_reassigned_units` (`incident_correlator.py`): on `reassignment=True`, scans other active incidents for unit overlap and releases the unit there via the same `units_active`/`units_cleared` merge explicit-10-8 extraction uses (factored out as `_apply_unit_clearance`). 2. `system.unit_format_hint` (owner-authored free text, `GET`/`PUT /systems/{id}/unit-format`, no auto-induction yet) fed into the extraction prompt; broadened the `cleared_units` rule to accept dispatch-confirmed availability while still requiring an identifiable unit ID (guards against bare `10-8`/`clear` noise and Whisper-hallucination runs already caught upstream by `_is_garbage_transcript`). Tests: 401 pass, 0 fail (verified against a Linux venv this session set up at `~/venvs/drb-5c` — note added to CLAUDE.md's testing-reality section since the documented Windows-venv path isn't the only option going forward). **Not yet done:** owner needs to populate `unit_format_hint` for the systems actually in use (Yorktown, County) via the new PUT route — no default value, no UI field yet (API-only for now, consistent with how ten-codes/area-context started). Auto-induction of unit-ID format (mirroring `vocabulary_learner`'s induction loop) deliberately deferred — bigger lever, not needed yet for a 2-3-system personal-use scope. **Left open pending the post-deploy live review**: plan is to measure `cleared_units`/`units_active` population and the reassignment-release path against real traffic for a 2-hour window post-deploy, then reassess before closing. Related but distinct: #88 (resolved/reassignment are prompt-only invariants with no validator — a parsing-robustness gap, not this lifecycle gap). Filed by Claude Code following up on a correlation-review audit, 2026-09-20.

logan referenced this issue

2026-09-20 14:38:54 -04:00

deploy.yml: /opt/drb/.last_good_tag reads empty (not missing), so PREV_TAG is blank and every deploy's Main Deploy step fails — rollback safety net has been dead since at least run #602 #156

logan referenced this issue

2026-09-26 14:48:53 -04:00

Gemini correlation tier returned nothing for the whole 09-20→09-22 AI run — 377/377 decisions rules-only #169

logan referenced this issue

2026-09-27 19:26:37 -04:00

Long-lived Major incident becomes a sink: 'Rollover Entrapment at TSP' absorbed 40 calls / 2h34m / 37 km path on Ch1 #189

logan referenced this issue

2026-10-04 01:15:58 -04:00

Admin Replay: re-run the pipeline over past calls in a sandbox, so correlation changes are measured without live AI windows #170

logan added the area/ai-pipeline area/correlation

labels 2026-10-05 02:04:53 -04:00

logan referenced this issue

2026-10-05 02:08:36 -04:00

Map: history mode — make the TimeScrubber playhead move (resolved_at already exists; #18 closed) #240

logan commented

2026-10-05 02:08:39 -04:00

Map persona panel 2026-10-05: 7 of 8 personas (TOC, NOC, RCC, GSOC, OEM, NEWS, FD) named this as a reason they would not keep the map on a wall — 'cleared/stood down' is the signal that releases a hold or crew, and false Majors at the top of the worst-first rail train the room to ignore red. It also taints history replay: nearly every resolved_at is the idle sweep, so replayed end times run up to 90 min late. Ranked top-5 map pain point; no new issue filed.

Map persona panel 2026-10-05: 7 of 8 personas (TOC, NOC, RCC, GSOC, OEM, NEWS, FD) named this as a reason they would not keep the map on a wall — 'cleared/stood down' is the signal that releases a hold or crew, and false Majors at the top of the worst-first rail train the room to ignore red. It also taints history replay: nearly every `resolved_at` is the idle sweep, so replayed end times run up to 90 min late. Ranked top-5 map pain point; no new issue filed.

logan referenced this issue

2026-10-05 02:08:40 -04:00

Severity never de-escalates on stand-down: 'Flames at the house' stays Major/active after units report a fireplace #190

logan referenced this issue

2026-10-05 02:09:11 -04:00

Map roadmap — persona panel 2026-10-05 #247

logan referenced this issue

2026-10-27 20:03:50 -04:00

Correlation: rules first, one cheap LLM call only when ambiguous; remove the cheap/smart consensus tiebreak #299

logan referenced this issue

2026-11-08 01:07:48 -05:00

Correlation: make the P25 radio ID (srcaddr) a first-class link, and measure how often calls carry it #304