Incidents never actually clear — units_cleared almost never populates, 90-min idle sweep does the resolving instead #155

Open
opened 2026-09-20 14:33:01 -04:00 by logan · 0 comments
Owner

Found while auditing whether incident correlation work has actually improved incident lifecycle tracking (dispatch → final unit 10-8), not just junk/duplicate suppression.

Evidence (5 correlation debug windows, 281 in-window incidents, 1721 calls scanned): only 2/281 incidents ever had a non-empty units_cleared. Of ~183 calls with a clearance-like phrase in the transcript, only ~19 produced a cleared_units extraction. incident_correlator.py's resolve-on-clear condition (units_cleared and not units_active → resolved) essentially never fires — 183/281 incidents in-window were instead resolved by the 90-minute idle sweep (summarizer.py), i.e. incidents time out, they don't get marked cleared by a real event.

Root cause — two independent mechanisms, not one (confirmed against corr_dump_9-12/9-13/9-14.json):

  1. Pattern B — re-dispatch accept, no explicit 10-8 (dispatch: "are you able to clear and take a run at X" / unit: "10-4"). reassignment=True already fired correctly and already suppressed the unit from re-linking to its prior incident (upload.py sets corr_units=[] on reassignment) — but nothing ever released the unit FROM that prior incident. It just sat "active" until the idle sweep. Structural gap, not an extraction miss.
  2. Pattern A — self-clear, inconsistent extraction. No per-system unit-ID format awareness existed anywhere (vocabulary_learner.py's "known local terms" is a flat glossary, not a structured format) — departments use incompatible conventions (Yorktown: 5-David, sometimes spoken as bare David; County: SAM-1/airport-3/parks-4, a location word + number) with zero shared pattern. Also, intelligence.py's cleared_units prompt rule only accepted a unit self-reporting, missing dispatch confirming a unit's status back to them (e.g. unit asks "how do you show me", dispatch replies "showing you available") — confirmed misses where a known unit ID sits directly next to a clear phrase matching the prompt's own example pattern and still got dropped.

Fixed in fb0bb15 (pushed to main, deploying now):

  1. _release_reassigned_units (incident_correlator.py): on reassignment=True, scans other active incidents for unit overlap and releases the unit there via the same units_active/units_cleared merge explicit-10-8 extraction uses (factored out as _apply_unit_clearance).
  2. system.unit_format_hint (owner-authored free text, GET/PUT /systems/{id}/unit-format, no auto-induction yet) fed into the extraction prompt; broadened the cleared_units rule to accept dispatch-confirmed availability while still requiring an identifiable unit ID (guards against bare 10-8/clear noise and Whisper-hallucination runs already caught upstream by _is_garbage_transcript).

Tests: 401 pass, 0 fail (verified against a Linux venv this session set up at ~/venvs/drb-5c — note added to CLAUDE.md's testing-reality section since the documented Windows-venv path isn't the only option going forward).

Not yet done: owner needs to populate unit_format_hint for the systems actually in use (Yorktown, County) via the new PUT route — no default value, no UI field yet (API-only for now, consistent with how ten-codes/area-context started). Auto-induction of unit-ID format (mirroring vocabulary_learner's induction loop) deliberately deferred — bigger lever, not needed yet for a 2-3-system personal-use scope.

Left open pending the post-deploy live review: plan is to measure cleared_units/units_active population and the reassignment-release path against real traffic for a 2-hour window post-deploy, then reassess before closing.

Related but distinct: #88 (resolved/reassignment are prompt-only invariants with no validator — a parsing-robustness gap, not this lifecycle gap).

Filed by Claude Code following up on a correlation-review audit, 2026-09-20.

**Found while auditing whether incident correlation work has actually improved incident lifecycle tracking (dispatch → final unit 10-8), not just junk/duplicate suppression.** Evidence (5 correlation debug windows, 281 in-window incidents, 1721 calls scanned): only 2/281 incidents ever had a non-empty `units_cleared`. Of ~183 calls with a clearance-like phrase in the transcript, only ~19 produced a `cleared_units` extraction. `incident_correlator.py`'s resolve-on-clear condition (`units_cleared and not units_active → resolved`) essentially never fires — 183/281 incidents in-window were instead resolved by the 90-minute idle sweep (`summarizer.py`), i.e. incidents time out, they don't get marked cleared by a real event. **Root cause — two independent mechanisms, not one** (confirmed against corr_dump_9-12/9-13/9-14.json): 1. **Pattern B — re-dispatch accept, no explicit 10-8** (dispatch: "are you able to clear and take a run at X" / unit: "10-4"). `reassignment=True` already fired correctly and already suppressed the unit from re-linking to its prior incident (`upload.py` sets `corr_units=[]` on reassignment) — but nothing ever released the unit FROM that prior incident. It just sat "active" until the idle sweep. Structural gap, not an extraction miss. 2. **Pattern A — self-clear, inconsistent extraction.** No per-system unit-ID format awareness existed anywhere (`vocabulary_learner.py`'s "known local terms" is a flat glossary, not a structured format) — departments use incompatible conventions (Yorktown: `5-David`, sometimes spoken as bare `David`; County: `SAM-1`/`airport-3`/`parks-4`, a location word + number) with zero shared pattern. Also, `intelligence.py`'s `cleared_units` prompt rule only accepted a unit self-reporting, missing dispatch confirming a unit's status back to them (e.g. unit asks "how do you show me", dispatch replies "showing you available") — confirmed misses where a known unit ID sits directly next to a clear phrase matching the prompt's own example pattern and still got dropped. **Fixed in `fb0bb15`** (pushed to main, deploying now): 1. `_release_reassigned_units` (`incident_correlator.py`): on `reassignment=True`, scans other active incidents for unit overlap and releases the unit there via the same `units_active`/`units_cleared` merge explicit-10-8 extraction uses (factored out as `_apply_unit_clearance`). 2. `system.unit_format_hint` (owner-authored free text, `GET`/`PUT /systems/{id}/unit-format`, no auto-induction yet) fed into the extraction prompt; broadened the `cleared_units` rule to accept dispatch-confirmed availability while still requiring an identifiable unit ID (guards against bare `10-8`/`clear` noise and Whisper-hallucination runs already caught upstream by `_is_garbage_transcript`). Tests: 401 pass, 0 fail (verified against a Linux venv this session set up at `~/venvs/drb-5c` — note added to CLAUDE.md's testing-reality section since the documented Windows-venv path isn't the only option going forward). **Not yet done:** owner needs to populate `unit_format_hint` for the systems actually in use (Yorktown, County) via the new PUT route — no default value, no UI field yet (API-only for now, consistent with how ten-codes/area-context started). Auto-induction of unit-ID format (mirroring `vocabulary_learner`'s induction loop) deliberately deferred — bigger lever, not needed yet for a 2-3-system personal-use scope. **Left open pending the post-deploy live review**: plan is to measure `cleared_units`/`units_active` population and the reassignment-release path against real traffic for a 2-hour window post-deploy, then reassess before closing. Related but distinct: #88 (resolved/reassignment are prompt-only invariants with no validator — a parsing-robustness gap, not this lifecycle gap). Filed by Claude Code following up on a correlation-review audit, 2026-09-20.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#155