Consensus tiebreaker manufactures incidents — resolves ~all disagreements as "new", overriding the LLM's "link" #115

New Issue

2026-09-07T04:42:16-04:00

logan commented

2026-09-07 04:42:16 -04:00

From the 2026-09-07 correlation measurement window (CORRELATION_REVIEW_0907.md, corr_dump_9-7_0437am.json). This — not scene detection (#5) — is the dominant over-split driver.

Data (88 calls in a 5h, 2-node window)

corr_consensus × cheap-LLM corr_llm_action × resulting corr_path:

consensus	cheap LLM said	count	→ `new`
agreed	—	27	20
rules_only	—	19	3
tiebreak	link	10	8
tiebreak	new	5	0
tiebreak	orphan	25	24

tiebreak ran on 40 / 88 calls (45%) — nearly every call hits the expensive smart model. Large Gemini spend on its own.
Cheap LLM wanted link → tiebreaker chose "new" 8/10.
Cheap LLM wanted orphan → tiebreaker opened a new incident 24/25.
32 of 55 window "new" incidents came out of a tiebreak where the cheap LLM wanted to link or orphan.

The tiebreaker's own _RULES (shared in llm_correlator.py) say "orphan when in doubt — conservative is always correct" and "new only if the call has a clear incident_type AND describes a distinct, identifiable scene." Its behaviour is the opposite.

Concrete cases

f53119a5 — same TG-383 event ("female backed up the ramp and left", 05:12–05:16) landed in two incidents (daa58459 "Vehicle Leaving Scene", e55de831 "Vehicle Check at ramp from Midland"). corr_consensus: tiebreak, corr_llm_action: link → corr_path: new + double-linked.
Mohegan Park Ave (TG 9560): 4e4a7864 "Nuisance Alarm" (07:54:57) + 0a95dc40 "Noise Complaint" (07:55:52) — same car-alarms call 55s apart. 2nd call corr_llm_action: link → corr_path: new.

Where to look

llm_correlator.tiebreak() + _build_tiebreak_prompt() — is the prompt biased toward "new"? Does it actually instruct the model that resolving a link/new disagreement toward link is the conservative choice for an over-splitting system?
_inc_summary() — the active-incident block the tiebreaker sees is id | type | loc | units[:6] | tags[:4] | idle. No transcript, no call list. That may be too thin for the model to justify a link, so it defaults to "new".
decisions_agree() / the consensus gate in upload.py:_correlate_with_consensus — 40/88 going to tiebreak suggests the cheap tier and rules disagree far more than expected; check whether the cheap LLM is being fed enough to agree with an obvious link.
Interaction with #96: corr_debug is per-call last-write-wins, so for multi-scene calls these fields are a splice. The single-scene majority makes the aggregate directionally sound but a per-scene record (#96) is needed before tuning against exact numbers.

Geocode coverage is near-zero (location_proximity fired 5× in 154 calls), so the location-fit path can't back up a link — "Ave" vs "Avenue" splits. Worth a normalisation pass (St/Street, Ave/Avenue, Rd/Road) either as its own issue or folded here.
#103 (unit-overlap veto dark on None embedding), #86 (0.82 veto literal) — same bounding problem from the rules side.
#5 (scene over-split prompt) — measured this window, no effect; the lever was here all along.

From the 2026-09-07 correlation measurement window (`CORRELATION_REVIEW_0907.md`, `corr_dump_9-7_0437am.json`). This — not scene detection (#5) — is the dominant over-split driver. ## Data (88 calls in a 5h, 2-node window) `corr_consensus` × cheap-LLM `corr_llm_action` × resulting `corr_path`: | consensus | cheap LLM said | count | → `new` | |---|---|---|---| | agreed | — | 27 | 20 | | rules_only | — | 19 | 3 | | **tiebreak** | **link** | 10 | **8** | | tiebreak | new | 5 | 0 | | **tiebreak** | **orphan** | 25 | **24** | - `tiebreak` ran on **40 / 88 calls (45%)** — nearly every call hits the expensive smart model. Large Gemini spend on its own. - Cheap LLM wanted **link** → tiebreaker chose "new" **8/10**. - Cheap LLM wanted **orphan** → tiebreaker opened a **new incident 24/25**. - **32 of 55 window "new" incidents** came out of a tiebreak where the cheap LLM wanted to link or orphan. The tiebreaker's own `_RULES` (shared in `llm_correlator.py`) say *"orphan when in doubt — conservative is always correct"* and *"new only if the call has a clear incident_type AND describes a distinct, identifiable scene."* Its behaviour is the opposite. ## Concrete cases - `f53119a5` — same TG-383 event ("female backed up the ramp and left", 05:12–05:16) landed in **two** incidents (`daa58459` "Vehicle Leaving Scene", `e55de831` "Vehicle Check at ramp from Midland"). `corr_consensus: tiebreak`, `corr_llm_action: link` → `corr_path: new` + double-linked. - Mohegan Park Ave (TG 9560): `4e4a7864` "Nuisance Alarm" (07:54:57) + `0a95dc40` "Noise Complaint" (07:55:52) — same car-alarms call 55s apart. 2nd call `corr_llm_action: link` → `corr_path: new`. ## Where to look - `llm_correlator.tiebreak()` + `_build_tiebreak_prompt()` — is the prompt biased toward "new"? Does it actually instruct the model that resolving a link/new disagreement toward *link* is the conservative choice for an over-splitting system? - `_inc_summary()` — the active-incident block the tiebreaker sees is `id | type | loc | units[:6] | tags[:4] | idle`. No transcript, no call list. That may be too thin for the model to justify a link, so it defaults to "new". - `decisions_agree()` / the consensus gate in `upload.py:_correlate_with_consensus` — 40/88 going to tiebreak suggests the cheap tier and rules disagree far more than expected; check whether the cheap LLM is being fed enough to agree with an obvious link. - Interaction with #96: `corr_debug` is per-call last-write-wins, so for multi-scene calls these fields are a splice. The single-scene majority makes the aggregate directionally sound but a per-scene record (#96) is needed before tuning against exact numbers. ## Related - Geocode coverage is near-zero (`location_proximity` fired 5× in 154 calls), so the location-fit path can't back up a link — "Ave" vs "Avenue" splits. Worth a normalisation pass (St/Street, Ave/Avenue, Rd/Road) either as its own issue or folded here. - #103 (unit-overlap veto dark on `None` embedding), #86 (0.82 veto literal) — same bounding problem from the rules side. - #5 (scene over-split prompt) — measured this window, no effect; the lever was here all along.

logan referenced this issue

2026-09-07 16:53:22 -04:00

correlator: give the LLM tier what it needs to link, stop it defaulting to "new" (#115) #116

logan commented

2026-09-07 21:19:55 -04:00

Measurement window #2 (code `bccb3e0` = #116) — result: no measurable improvement

Dump corr_dump_9-7_pm.json, ~4h, 2 nodes. Full call-by-call write-up in Version 5C/CORRELATION_REVIEW_0907b.md.

metric	window #1 (pre-#116)	window #2 (#116)
new-incident rate	62%	57% (99/173)
tiebreak escalation	45%	43% (75/173)
single-call incidents	~50%	62% (62/100)
`llm=orphan` overridden to `new` by rules→tiebreak	24/25	~21/21

The reframed _RULES text changed nothing the tiebreaker outputs.

The call-by-call read reframes the bug

It is not "one conversation shattered into 5–10 incidents." Reading transcripts, the 1-call incidents are almost all different transmissions, and ~35–40 of the 80 window incidents are non-events that should never have been incidents: unit check-ins ("114 Paul, Metro Central. 10-4"), clearings ("6-8 show me 98"), roll call ("stand by for roll call"), call-number admin. The genuinely distinct events (property retrieval, subject check, Pelham Station, trespassing MP 13.7, MVA, fall injury) are correctly separate.

Mechanism, every time: corr_llm_action=orphan · corr_rules_action=new · corr_consensus=tiebreak → new. The cheap LLM reads the transcript and correctly says "not an incident"; rules says new only because there's no link candidate; the tiebreaker sides with rules ~21/21.

Recommendation — stop iterating the prompt, change the consensus logic

When corr_llm_action=="orphan" and corr_rules_action=="new" (rules has only "no candidate", not a positive event signal) → resolve to orphan, do not escalate. The LLM read the transcript; the rules engine didn't. Removes ~21/80 of this window's incidents. Small change in _correlate_with_consensus, no prompt. Guard: still escalate when rules has a real positive (unit-continuity, location+unit-overlap, fast/disambig).
Tighten corr_path=location — 3–4 clear over-merges this window (8 Hill St + Ossining restroom; two different churches an hour apart; a tg383 harassment + a tg9048 call). Needs unit overlap OR tight geocoded proximity, not a shared road token. Ties to #86 / #103.
Re-measure whether the 43% tiebreak tier earns its cost once (1) lands (#35, #96).

## Measurement window #2 (code bccb3e0 = #116) — result: no measurable improvement Dump `corr_dump_9-7_pm.json`, ~4h, 2 nodes. Full call-by-call write-up in `Version 5C/CORRELATION_REVIEW_0907b.md`. | metric | window #1 (pre-#116) | window #2 (#116) | |---|---|---| | new-incident rate | 62% | 57% (99/173) | | tiebreak escalation | 45% | 43% (75/173) | | single-call incidents | ~50% | 62% (62/100) | | `llm=orphan` overridden to `new` by rules→tiebreak | 24/25 | ~21/21 | The reframed `_RULES` text changed nothing the tiebreaker outputs. ### The call-by-call read reframes the bug It is **not** "one conversation shattered into 5–10 incidents." Reading transcripts, the 1-call incidents are almost all *different* transmissions, and **~35–40 of the 80 window incidents are non-events that should never have been incidents**: unit check-ins ("114 Paul, Metro Central. 10-4"), clearings ("6-8 show me 98"), roll call ("stand by for roll call"), call-number admin. The genuinely distinct events (property retrieval, subject check, Pelham Station, trespassing MP 13.7, MVA, fall injury) are *correctly* separate. Mechanism, every time: `corr_llm_action=orphan` · `corr_rules_action=new` · `corr_consensus=tiebreak` → **new**. The cheap LLM reads the transcript and correctly says "not an incident"; rules says `new` only because there's no link candidate; the tiebreaker sides with rules ~21/21. ### Recommendation — stop iterating the prompt, change the consensus logic 1. **When `corr_llm_action=="orphan"` and `corr_rules_action=="new"` (rules has only "no candidate", not a positive event signal) → resolve to `orphan`, do not escalate.** The LLM read the transcript; the rules engine didn't. Removes ~21/80 of this window's incidents. Small change in `_correlate_with_consensus`, no prompt. Guard: still escalate when rules has a real positive (`unit-continuity`, `location`+unit-overlap, `fast/disambig`). 2. **Tighten `corr_path=location`** — 3–4 clear over-merges this window (8 Hill St + Ossining restroom; two different churches an hour apart; a tg383 harassment + a tg9048 call). Needs unit overlap OR tight geocoded proximity, not a shared road token. Ties to #86 / #103. 3. Re-measure whether the 43% tiebreak tier earns its cost once (1) lands (#35, #96).

logan referenced this issue from a commit

2026-09-07 23:33:02 -04:00

correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)

logan referenced this issue

2026-09-07 23:33:51 -04:00

correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115) #125

logan commented

2026-09-07 23:34:06 -04:00

The consensus gate is implemented in PR #125 (branch fix/115-consensus-orphan-gate), pending review.

Fix 1 (primary) _correlate_with_consensus in routers/upload.py: when the cheap LLM says orphan and rules says new with no positive event signal, resolve to orphan and skip the tiebreak call. Guard:
llm==orphan and rules==new and not _rules_has_positive_event_signal(rules_decision), where a positive signal is corr_fit_signal in {unit_overlap, location_proximity} or corr_path in {unit-continuity, location, fast/disambig, fast/single}. A bare new (empty corr_debug / fit signal in {None, none, thin_recency}) is gated; a real signal still escalates to the tiebreaker. The gated outcome records corr_consensus="llm_orphan_gate" and keeps corr_llm_reasoning / corr_rules_action / corr_llm_action.

Fix 2 (secondary) corr_path=location in incident_correlator.py: a sub-0.5km distance alone no longer links. Now requires unit overlap with the candidate OR distance under _LOCATION_TIGHT_PROXIMITY_KM = 0.2km; pursuit incidents keep their wide movement-validated radius. A surviving location link now also stamps corr_fit_signal. Proximity half done for real (no TODO).

Tests: new tests/test_consensus_gate.py (13 cases, tiebreak spied). Full c2-core suite 309 -> 322 passing, green.

The consensus gate is implemented in **PR #125** (branch `fix/115-consensus-orphan-gate`), pending review. **Fix 1 (primary)** `_correlate_with_consensus` in `routers/upload.py`: when the cheap LLM says `orphan` and rules says `new` with no positive event signal, resolve to `orphan` and skip the `tiebreak` call. Guard: `llm==orphan and rules==new and not _rules_has_positive_event_signal(rules_decision)`, where a positive signal is `corr_fit_signal in {unit_overlap, location_proximity}` or `corr_path in {unit-continuity, location, fast/disambig, fast/single}`. A bare `new` (empty corr_debug / fit signal in {None, none, thin_recency}) is gated; a real signal still escalates to the tiebreaker. The gated outcome records `corr_consensus="llm_orphan_gate"` and keeps `corr_llm_reasoning` / `corr_rules_action` / `corr_llm_action`. **Fix 2 (secondary)** `corr_path=location` in `incident_correlator.py`: a sub-0.5km distance alone no longer links. Now requires unit overlap with the candidate OR distance under `_LOCATION_TIGHT_PROXIMITY_KM = 0.2km`; pursuit incidents keep their wide movement-validated radius. A surviving `location` link now also stamps `corr_fit_signal`. Proximity half done for real (no TODO). Tests: new `tests/test_consensus_gate.py` (13 cases, tiebreak spied). Full c2-core suite 309 -> 322 passing, green.

logan referenced this issue from a commit

2026-09-07 23:50:51 -04:00

correlator: fix consensus orphan-gate to test call substance, not empty corr_debug (#115)

logan commented

2026-09-07 23:51:54 -04:00

PR #125 revised after review (commit dd42657).

MUST-FIX 1 — dead safety net. Confirmed: _rules_has_positive_event_signal read rules_decision["corr_debug"], which is empty at preview time for action=="new" (corr_path/fit-signal written at apply time). The gate was firing on every llm=orphan/rules=new disagreement. Replaced with _call_is_substanceless(ctx) — gates to orphan ONLY when all of: severity not moderate/major, no ctx["call_vehicles"|"coords"|"tags"], and no incident in ctx["recent"] on the same system+talkgroup. Otherwise escalates to tiebreak as before. The vehicle/geocode/tag test is incident_correlator.has_event_substance, factored out of the creation gate (~line 1372) and shared so the two can't drift.

MUST-FIX 2 — permanent gating. Decision: longer link-only sweep budget for gated orphans (GATED_ORPHAN_SWEEP_ATTEMPTS=10 vs 3), NOT creation-eligibility. Making the sweep create_if_new=True for these would be inert — the rules creation gate resolves no incident_type for a substanceless call and returns orphan regardless — and minting an incident from a content-free call is the bug that gate prevents. The cascade is broken instead by the new same-talkgroup check: once any substantive call opens the incident, later thin calls on that TG escalate rather than gate.

Fix-2 nits. Location path now scores every in-radius candidate and links the nearest qualifying one (was break on first in unsorted recent). Unit-overlap location links now write corr_fit_signal="location_unit_overlap" so they stop merging into the fast-path unit_overlap bucket in the admin.py histogram (#35).

Tests. test_consensus_gate.py 13 -> 16 cases; corr_debug-signal cases replaced with ctx-substance cases (severity/coords/tags/vehicles/same-tg); nearest-wins location test added; the location guard tests now assert they reach the new guard (caplog on the skip line, plus the distinct fit_signal). Full drb-c2-core suite 322 -> 325, green (sandboxed Linux pytest).

PR #125 revised after review (commit dd42657). **MUST-FIX 1 — dead safety net.** Confirmed: `_rules_has_positive_event_signal` read `rules_decision["corr_debug"]`, which is empty at preview time for `action=="new"` (corr_path/fit-signal written at apply time). The gate was firing on every llm=orphan/rules=new disagreement. Replaced with `_call_is_substanceless(ctx)` — gates to orphan ONLY when all of: severity not moderate/major, no `ctx["call_vehicles"|"coords"|"tags"]`, and no incident in `ctx["recent"]` on the same system+talkgroup. Otherwise escalates to tiebreak as before. The vehicle/geocode/tag test is `incident_correlator.has_event_substance`, factored out of the creation gate (~line 1372) and shared so the two can't drift. **MUST-FIX 2 — permanent gating.** Decision: longer link-only sweep budget for gated orphans (`GATED_ORPHAN_SWEEP_ATTEMPTS=10` vs 3), NOT creation-eligibility. Making the sweep `create_if_new=True` for these would be inert — the rules creation gate resolves no incident_type for a substanceless call and returns orphan regardless — and minting an incident from a content-free call is the bug that gate prevents. The cascade is broken instead by the new same-talkgroup check: once any substantive call opens the incident, later thin calls on that TG escalate rather than gate. **Fix-2 nits.** Location path now scores every in-radius candidate and links the nearest qualifying one (was `break` on first in unsorted `recent`). Unit-overlap location links now write `corr_fit_signal="location_unit_overlap"` so they stop merging into the fast-path `unit_overlap` bucket in the admin.py histogram (#35). **Tests.** test_consensus_gate.py 13 -> 16 cases; corr_debug-signal cases replaced with ctx-substance cases (severity/coords/tags/vehicles/same-tg); nearest-wins location test added; the location guard tests now assert they reach the new guard (caplog on the skip line, plus the distinct fit_signal). Full drb-c2-core suite 322 -> 325, green (sandboxed Linux pytest).

logan referenced this issue from a commit

2026-09-07 23:57:22 -04:00

correlator: keep the tiebreak for typed / reassignment calls in the orphan gate (#115)

logan commented

2026-09-07 23:57:42 -04:00

PR #125 — ready for merge (commit `15a9d10`, 3 commits, sandboxed pytest 327 green)

Two drb-correlation-review passes, both must-fixes applied:

Fix 1 — the gate. llm=orphan + rules=new resolves to orphan without a tiebreak only when _call_is_substanceless(ctx):

severity not moderate/major
no coords, tags, or vehicles (incident_correlator.has_event_substance, extracted from the creation gate so the two can't drift)
no recent incident on the same system+talkgroup (the "unit dispatched, thin ack 10-30s later" case — keeps the tiebreak)
no resolved incident_type and not reassignment (round-2 fix: a routine-severity fire/medical call, or a unit pulled to a new job, would otherwise be dropped where rules would open an incident)

Any one false → unchanged escalation to tiebreak.

Fix 2 — corr_path=location. A non-pursuit location link now needs unit overlap OR ≤ 0.2 km (was: anywhere in the 0.5 km radius). Unit-overlap links write corr_fit_signal="location_unit_overlap" (distinct bucket for the #35 tally); distance-only keep location_proximity. Candidates are scored and the nearest qualifying one is linked (was: break on first in an unsorted list).

Permanence. Gated orphans get GATED_ORPHAN_SWEEP_ATTEMPTS=10 (vs 3) in the re-correlation sweep, link-only — so a substantive call landing on the TG within ~20 min still pulls the earlier thin calls in. Not made creation-eligible (a substanceless call resolves no incident_type, so it'd be inert, and minting an incident from a content-free call is the failure this gate prevents).

Residual (not blocking, follow-ups):

_LOCATION_TIGHT_PROXIMITY_KM = 0.2 is a hardcoded literal beside the configurable location_proximity_km — same shape as #86. Should move to config.
On a busy dispatch backbone a recent same-TG incident is nearly always live, so the gate mostly fires on quiet channels — recovers less of the window-#2 21/21 than the raw count suggests. Measure the llm_orphan_gate bucket in a window #3 (it appears in orphaned_calls, not the linked histogram).
The upstream fix (dispatch-vs-chatter classifier so "10-4"/"98"/roll-call never create a scene) is still the real lever — this PR is the stopgap.

## PR #125 — ready for merge (commit 15a9d10, 3 commits, sandboxed pytest 327 green) Two `drb-correlation-review` passes, both must-fixes applied: **Fix 1 — the gate.** `llm=orphan` + `rules=new` resolves to **orphan without a tiebreak** only when `_call_is_substanceless(ctx)`: - severity not moderate/major - no coords, tags, or vehicles (`incident_correlator.has_event_substance`, extracted from the creation gate so the two can't drift) - no recent incident on the same system+talkgroup (the "unit dispatched, thin ack 10-30s later" case — keeps the tiebreak) - **no resolved `incident_type` and not `reassignment`** (round-2 fix: a routine-severity fire/medical call, or a unit pulled to a new job, would otherwise be dropped where rules would open an incident) Any one false → unchanged escalation to `tiebreak`. **Fix 2 — `corr_path=location`.** A non-pursuit location link now needs unit overlap OR ≤ 0.2 km (was: anywhere in the 0.5 km radius). Unit-overlap links write `corr_fit_signal="location_unit_overlap"` (distinct bucket for the #35 tally); distance-only keep `location_proximity`. Candidates are scored and the nearest qualifying one is linked (was: `break` on first in an unsorted list). **Permanence.** Gated orphans get `GATED_ORPHAN_SWEEP_ATTEMPTS=10` (vs 3) in the re-correlation sweep, link-only — so a substantive call landing on the TG within ~20 min still pulls the earlier thin calls in. Not made creation-eligible (a substanceless call resolves no `incident_type`, so it'd be inert, and minting an incident from a content-free call is the failure this gate prevents). **Residual (not blocking, follow-ups):** - `_LOCATION_TIGHT_PROXIMITY_KM = 0.2` is a hardcoded literal beside the configurable `location_proximity_km` — same shape as #86. Should move to config. - On a busy dispatch backbone a recent same-TG incident is nearly always live, so the gate mostly fires on quiet channels — recovers less of the window-#2 21/21 than the raw count suggests. **Measure the `llm_orphan_gate` bucket in a window #3** (it appears in `orphaned_calls`, not the linked histogram). - The upstream fix (dispatch-vs-chatter classifier so "10-4"/"98"/roll-call never create a scene) is still the real lever — this PR is the stopgap.

logan referenced this issue from a commit

2026-09-11 23:18:14 -04:00

Merge pull request 'correlator: gate LLM-orphan against rules-new instead of escalating to tiebreak (#115)' (#125) from fix/115-consensus-orphan-gate into main

logan commented

2026-09-12 04:19:11 -04:00

Window #3 result (code `07ff9ba193` = #125): the gate fired ZERO times

100 incidents, 91 orphans, 4h live. Full write-up Version 5C/CORRELATION_REVIEW_0912.md.

new-rate unchanged: 57% (was 57% in window #2)
corr_consensus=="llm_orphan_gate" count: 0 in the entire dump
24 linked calls this window have the exact rules=new, llm=orphan shape #125 targets — all 24 still escalated to tiebreak → new, same as window #2's 21/21

Root cause (confirmed 22/24): the gate's "skip if a recent incident exists on the same talkgroup" escape hatch uses a 2-hour lookback. The three busiest channels produce 3-13 other incidents per 2h, so that escape hatch is satisfied almost unconditionally — it disables the gate on exactly the channels it exists for. 2/24 have zero explainable recent-same-tg candidates and still weren't gated — needs an engineer to read the actual ctx["recent"] construction, not inferred from the dump.

Decision: don't ship a third consensus-layer tweak. Shrink the escape-hatch window from 2h to a few minutes (matching what "thin ack right after dispatch" actually means — fast/thin already handles true immediate follow-ups upstream of this branch), investigate the 2 unexplained misses, then re-measure. The durable fix is still the unscoped upstream chatter classifier — two iterations at the consensus layer (#116, #125) have now both measured as no-ops on the top-line rate.

## Window #3 result (code 07ff9ba193 = #125): the gate fired ZERO times 100 incidents, 91 orphans, 4h live. Full write-up `Version 5C/CORRELATION_REVIEW_0912.md`. - new-rate unchanged: 57% (was 57% in window #2) - `corr_consensus=="llm_orphan_gate"` count: **0** in the entire dump - 24 linked calls this window have the exact `rules=new, llm=orphan` shape #125 targets — all 24 still escalated to `tiebreak` → `new`, same as window #2's 21/21 **Root cause (confirmed 22/24):** the gate's "skip if a recent incident exists on the same talkgroup" escape hatch uses a 2-hour lookback. The three busiest channels produce 3-13 other incidents per 2h, so that escape hatch is satisfied almost unconditionally — it disables the gate on exactly the channels it exists for. 2/24 have zero explainable recent-same-tg candidates and still weren't gated — needs an engineer to read the actual `ctx["recent"]` construction, not inferred from the dump. **Decision:** don't ship a third consensus-layer tweak. Shrink the escape-hatch window from 2h to a few minutes (matching what "thin ack right after dispatch" actually means — `fast/thin` already handles true immediate follow-ups upstream of this branch), investigate the 2 unexplained misses, then re-measure. The durable fix is still the unscoped upstream chatter classifier — two iterations at the consensus layer (#116, #125) have now both measured as no-ops on the top-line rate.

logan referenced this issue from a commit

2026-09-12 04:28:16 -04:00

correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)

logan referenced this issue

2026-09-12 04:31:07 -04:00

correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115) #126

logan commented

2026-09-12 04:31:19 -04:00

Opened PR #126: shrinks _recent_incident_on_same_talkgroup's lookback from 2h (correlation_window_hours) to settings.tg_dispatch_thin_idle_minutes (5 min) — the same recency bound the fast/thin path already uses for "dispatch, thin ack" — which is what the 2h version was letting through unconditionally on busy channels (window #3 measured 0/24 gate fires against the exact target shape).

Re: the 2 unexplained misses — confirmed ctx["recent"] is status=="active" incidents with over-capacity ones dropped (_build_context/_drop_capped), not a full collection scan, so a resolved/capped same-tg incident is invisible to the check even though it's chronologically recent. That's a real, documented gap (docstring + TODO(server-26#115) left in place), but it doesn't explain the review's specific 2 misses — those already had zero same-tg candidates by a full-collection scan, active or not, so some other _call_is_substanceless condition must have been true for those two. Not reproducible without the live dump, so left as-is rather than guessed at.

Sandboxed pytest: 327 -> 329 passed, including a regression test that fails on main and passes on the branch.

Opened PR #126: shrinks `_recent_incident_on_same_talkgroup`'s lookback from 2h (`correlation_window_hours`) to `settings.tg_dispatch_thin_idle_minutes` (5 min) — the same recency bound the fast/thin path already uses for "dispatch, thin ack" — which is what the 2h version was letting through unconditionally on busy channels (window #3 measured 0/24 gate fires against the exact target shape). Re: the 2 unexplained misses — confirmed `ctx["recent"]` is `status=="active"` incidents with over-capacity ones dropped (`_build_context`/`_drop_capped`), not a full collection scan, so a resolved/capped same-tg incident is invisible to the check even though it's chronologically recent. That's a real, documented gap (docstring + `TODO(server-26#115)` left in place), but it doesn't explain the review's specific 2 misses — those already had zero same-tg candidates by a full-collection scan, active or not, so some other `_call_is_substanceless` condition must have been true for those two. Not reproducible without the live dump, so left as-is rather than guessed at. Sandboxed pytest: 327 -> 329 passed, including a regression test that fails on `main` and passes on the branch.

logan referenced this issue from a commit

2026-09-12 04:42:17 -04:00

correlator: mirror the dispatch/tactical idle split in the escape hatch, record the gate-veto reason (#115)

logan referenced this issue from a commit

2026-09-12 04:47:18 -04:00

admin: surface corr_gate_veto on the correlation-debug endpoint (#115)

logan commented

2026-09-12 04:47:41 -04:00

PR #126 ready — fixes why #125 fired 0/24 times in window #3. Root cause: the "recent same-talkgroup incident" escape hatch used a 2h lookback, satisfied almost unconditionally on busy dispatch channels. Now 5 min (dispatch) / 15 min (tactical), mirroring the fast/thin path's own bound-selection. Added corr_gate_veto to corr_debug + the admin endpoint so window #4 measures the escape rate directly instead of reconstructing it from the raw dump. 2 review rounds, 332 tests green. Details on PR #126.

Expectation: the hatch still lacks a single-candidate requirement (the fast/thin path it mirrors has one), so expect partial recovery of the 24 target calls next window, not all of them — that's the next tuning decision, not a defect in this PR.

PR #126 ready — fixes why #125 fired 0/24 times in window #3. Root cause: the "recent same-talkgroup incident" escape hatch used a 2h lookback, satisfied almost unconditionally on busy dispatch channels. Now 5 min (dispatch) / 15 min (tactical), mirroring the fast/thin path's own bound-selection. Added `corr_gate_veto` to corr_debug + the admin endpoint so window #4 measures the escape rate directly instead of reconstructing it from the raw dump. 2 review rounds, 332 tests green. Details on PR #126. Expectation: the hatch still lacks a single-candidate requirement (the fast/thin path it mirrors has one), so expect partial recovery of the 24 target calls next window, not all of them — that's the next tuning decision, not a defect in this PR.

logan referenced this issue from a commit

2026-09-12 04:47:51 -04:00

Merge pull request 'correlator: shrink the same-talkgroup escape hatch from 2h to a few minutes (#115)' (#126) from fix/115-escape-hatch-window into main

logan referenced this issue

2026-09-12 23:58:06 -04:00

Upstream dispatch-vs-chatter classifier — shadow mode #127

logan referenced this issue

2026-09-13 00:00:33 -04:00

intelligence: shadow-mode upstream dispatch-vs-chatter classifier (#127) #128

logan referenced this issue from a commit

2026-09-13 14:07:04 -04:00

correlator: drop the dispatch/tactical name-guess from the escape hatch, use one window (#115)

logan referenced this issue

2026-09-13 14:07:28 -04:00

correlator: drop the dispatch/tactical name-guess from the escape hatch (#115) #133

logan referenced this issue from a commit

2026-09-13 14:07:37 -04:00

Merge pull request 'correlator: drop the dispatch/tactical name-guess from the escape hatch (#115)' (#133) from fix/115-drop-dispatch-tactical-split into main

logan referenced this issue

2026-09-13 14:08:03 -04:00

incident_correlator fast/thin path has the same dispatch/tactical name-guess as the #115 escape hatch, but it changes link decisions #134

logan commented

2026-09-13 17:21:12 -04:00

Window #4 result — 0/22, but #126 was never actually tested.

Full writeup: CORRELATION_REVIEW_0913.md.

The debug panel's "Incidents (limit)=100" is NOT time-boxed — 79/287 (28%) of the raw dump's scene-decisions were from 2026-09-12 or earlier, contaminating the first pass. After filtering to calls actually started inside the window (16:34-20:26 UTC), the true orphan-vs-new population is 22 calls, 0 gate fires.

0/22 hit the escape hatch #126 fixed. The 9 unexplained gate_veto=None records in the raw dump were all out-of-window contamination, not evidence against #126. This window gives no signal on whether #126 works — real traffic never exercised it.

18/22 (82%) vetoed on "type" — and every one of those 18 transcripts is visibly pure radio housekeeping (roll calls, 10-4 acks, plate readbacks) with the call's own final incident_type field == None. ctx["incident_type"] (read at correlation time, in-memory from the scene's own extraction) was truthy anyway. Either extraction is over-typing trivial chatter, or a sibling scene's type is leaking into this scene's ctx before its own extraction overwrites the shared top-level field — can't tell apart from this dump because #96/#114's per-scene scenes map doesn't capture incident_type per scene.

This is the actual reason the gate hasn't fired in either window — not the escape hatch. Fragmentation shape is unchanged: 46/62 (74%) of incidents started in-window are single-call, same as every prior window.

Per CORRELATION_REVIEW_0912.md's ceiling math this doesn't reopen #115 — filing two follow-ups instead (linked below) and standing by the existing call to deprioritize further gate tuning in favor of #127.

Closing scope on this issue's measurement question: #126 is unverified-but-not-refuted, and the real live blocker is a different, newly-identified mechanism.

**Window #4 result — 0/22, but #126 was never actually tested.** Full writeup: `CORRELATION_REVIEW_0913.md`. The debug panel's "Incidents (limit)=100" is NOT time-boxed — 79/287 (28%) of the raw dump's scene-decisions were from 2026-09-12 or earlier, contaminating the first pass. After filtering to calls actually started inside the window (16:34-20:26 UTC), the true orphan-vs-new population is **22 calls, 0 gate fires**. **0/22 hit the escape hatch #126 fixed.** The 9 unexplained gate_veto=None records in the raw dump were all out-of-window contamination, not evidence against #126. This window gives no signal on whether #126 works — real traffic never exercised it. **18/22 (82%) vetoed on `"type"`** — and every one of those 18 transcripts is visibly pure radio housekeeping (roll calls, 10-4 acks, plate readbacks) with the call's own final `incident_type` field == None. `ctx["incident_type"]` (read at correlation time, in-memory from the scene's own extraction) was truthy anyway. Either extraction is over-typing trivial chatter, or a sibling scene's type is leaking into this scene's ctx before its own extraction overwrites the shared top-level field — can't tell apart from this dump because #96/#114's per-scene `scenes` map doesn't capture `incident_type` per scene. **This is the actual reason the gate hasn't fired in either window — not the escape hatch.** Fragmentation shape is unchanged: 46/62 (74%) of incidents started in-window are single-call, same as every prior window. Per CORRELATION_REVIEW_0912.md's ceiling math this doesn't reopen #115 — filing two follow-ups instead (linked below) and standing by the existing call to deprioritize further gate tuning in favor of #127. Closing scope on this issue's measurement question: #126 is unverified-but-not-refuted, and the real live blocker is a different, newly-identified mechanism.

logan referenced this issue

2026-09-13 18:49:36 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — FINAL MINUTES #146

logan referenced this issue

2026-09-14 04:54:53 -04:00

LLM-orphan gate's "type" veto fires on visibly content-free chatter (18/22 this window) #138

logan added the area/correlation