Same call_id linked into 2 incident records with identical extracted data (~2% of linked calls) #131

New Issue

2026-09-13T13:19:39-04:00

logan commented

2026-09-13 13:19:39 -04:00

Investigation of #5 (scene over-splitting) found this instead, while ruling out #5's own stated mechanism. Full evidence in the #5 comment thread.

The bug: the same single-scene call (call_id) ends up attached to TWO separate incident records with byte-identical extracted data (same tags, location, units, severity) — not two different scenes describing two different things, the exact same extracted content, duplicated across incidents.

Scan across 3 live dumps (corr_dump_9-7_0437am.json, corr_dump_9-7_pm.json, corr_dump_9-12.json): 10 hits total (4/2/4), out of 173/173/181 linked calls per window (~2%). Every hit has identical tags/location/units/severity between its two incident appearances — ruling out genuine multi-scene extraction (which would produce different content per scene).

Example (corr_dump_9-7_0437am), transcript: "6-David, that's going to be track 4, intox male in his 40s, track 4, Mt. Vernon East. Affirmative. 10-4." — appears in incident 3254fd80 ("Intoxicated Subject at track 4, Mt. Vernon East") AND fe04dd3a ("Police — MTA PD Districts 6/7/11 - Police Dispatch"), both carrying tags=['intoxicated-subject'], location='track 4, Mt. Vernon East', units=['6-David']. This is the same call DEFERRED.md cites from 2026-09-06 (then 3 incidents) — still occurring 2026-09-07 (now 2 incidents).

Likely mechanism: corr_path/corr_consensus on the two appearances are usually identical (both new/tiebreak, or both fast/disambig/agreed) — consistent with the same call being run through correlation twice (initial pass + a re-correlation sweep pass, or two overlapping upload/reprocess paths), with each pass landing on or creating a different incident target without removing the call from the other. Needs an engineer reading incident_correlator.py's sweep/re-correlation call_ids-append path — does anything guard against a call already carrying an incident_id being re-appended to a second, different incident without being detached from the first?

Test gap: nothing currently covers "the same call_id must not end up in two different incidents' call_ids" — worth an invariant test regardless of root cause once found.

Relates to: #5 (originally suspected as the mechanism; ruled out — closing #5, this issue replaces it), #27 (sweep retry budget), #96 (per-scene call-doc storage, unrelated mechanism but same investigation pass).

Investigation of #5 (scene over-splitting) found this instead, while ruling out #5's own stated mechanism. Full evidence in the #5 comment thread. **The bug:** the same single-scene call (`call_id`) ends up attached to TWO separate incident records with byte-identical extracted data (same tags, location, units, severity) — not two different scenes describing two different things, the exact same extracted content, duplicated across incidents. **Scan across 3 live dumps** (`corr_dump_9-7_0437am.json`, `corr_dump_9-7_pm.json`, `corr_dump_9-12.json`): 10 hits total (4/2/4), out of 173/173/181 linked calls per window (~2%). Every hit has identical `tags`/`location`/`units`/`severity` between its two incident appearances — ruling out genuine multi-scene extraction (which would produce different content per scene). **Example** (`corr_dump_9-7_0437am`), transcript: *"6-David, that's going to be track 4, intox male in his 40s, track 4, Mt. Vernon East. Affirmative. 10-4."* — appears in incident `3254fd80` ("Intoxicated Subject at track 4, Mt. Vernon East") AND `fe04dd3a` ("Police — MTA PD Districts 6/7/11 - Police Dispatch"), both carrying `tags=['intoxicated-subject']`, `location='track 4, Mt. Vernon East'`, `units=['6-David']`. This is the same call DEFERRED.md cites from 2026-09-06 (then 3 incidents) — still occurring 2026-09-07 (now 2 incidents). **Likely mechanism:** `corr_path`/`corr_consensus` on the two appearances are usually identical (both `new`/`tiebreak`, or both `fast/disambig`/`agreed`) — consistent with the same call being run through correlation **twice** (initial pass + a re-correlation sweep pass, or two overlapping upload/reprocess paths), with each pass landing on or creating a different incident target without removing the call from the other. Needs an engineer reading `incident_correlator.py`'s sweep/re-correlation `call_ids`-append path — does anything guard against a call already carrying an `incident_id` being re-appended to a second, different incident without being detached from the first? **Test gap:** nothing currently covers "the same call_id must not end up in two different incidents' `call_ids`" — worth an invariant test regardless of root cause once found. Relates to: #5 (originally suspected as the mechanism; ruled out — closing #5, this issue replaces it), #27 (sweep retry budget), #96 (per-scene call-doc storage, unrelated mechanism but same investigation pass).

logan referenced this issue

2026-09-13 13:19:50 -04:00

Scene over-splitting: one radio conversation becomes two incidents #5

logan referenced this issue

2026-09-13 13:34:18 -04:00

corr_debug is overwritten per scene on the shared call doc, so #35's "LLM tier decides 3.5% of links" is a per-call blend and cannot answer whether the tier works #96

logan referenced this issue

2026-09-13 13:34:18 -04:00

summarizer reads the whole-call transcript per incident — next leg of the #80/#95/#102 scene-context leak #114

logan referenced this issue

2026-09-13 17:33:31 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CTO draft #141

logan referenced this issue

2026-09-13 18:49:36 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — FINAL MINUTES #146

logan referenced this issue from a commit

2026-09-14 00:19:16 -04:00

correlator: stop the re-correlation sweep racing an in-flight upload (#131)

logan referenced this issue

2026-09-14 00:25:33 -04:00

OpenAI/Gemini clients have no request timeout -- sweep hold-back constants are guessed, not derived #153

logan referenced this issue from a commit

2026-09-14 00:30:37 -04:00

correlator: stop the re-correlation sweep racing an in-flight upload (#131)

logan referenced a pull request that will close this issue

2026-09-14 00:30:54 -04:00

correlator: stop the re-correlation sweep racing an in-flight upload (#131) #154

logan closed this issue

2026-09-14 00:31:15 -04:00

logan commented

2026-09-14 00:31:16 -04:00

Fixed by PR #154, merged to main. Root cause was a race between the re-correlation sweep and the real-time upload pipeline, not a scene-extraction issue -- confirmed with drb-correlation-review. Follow-up filed at #153 (missing client timeouts, the actual reason the 5-min hold-back was raised to 15).

logan referenced this issue from a commit

2026-09-14 00:31:16 -04:00

Merge pull request 'correlator: stop the re-correlation sweep racing an in-flight upload (#131)' (#154) from fix/131-sweep-race-duplicate-link into main

logan commented

2026-09-14 04:54:53 -04:00

Fix confirmed working — window #5 measurement (CORRELATION_REVIEW_0914.md).

Zero duplicate-links among calls that started after this window began (i.e., processed entirely under the fixed code, deployed as a919770). The 3 duplicates visible in the raw (non-time-boxed) dump all started 2026-09-13, hours before the fix deployed -- pre-existing contamination, not a new occurrence.

Not addressed: those 3 stale duplicates are still bad data in the database (each linked to two incidents). Cosmetic at this volume -- not filing a cleanup issue unless it recurs at scale.

**Fix confirmed working — window #5 measurement (`CORRELATION_REVIEW_0914.md`).** Zero duplicate-links among calls that started after this window began (i.e., processed entirely under the fixed code, deployed as `a919770`). The 3 duplicates visible in the raw (non-time-boxed) dump all started 2026-09-13, hours before the fix deployed -- pre-existing contamination, not a new occurrence. Not addressed: those 3 stale duplicates are still bad data in the database (each linked to two incidents). Cosmetic at this volume -- not filing a cleanup issue unless it recurs at scale.

logan added the area/correlation

labels 2026-10-05 02:04:48 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#131