correlator: stop the re-correlation sweep racing an in-flight upload (#131) #154

2026-09-14T00:30:54-04:00

logan commented

2026-09-14 00:30:54 -04:00

Closes #131.

Same call_id ends up in TWO incidents' call_ids, byte-identical extracted data, ~2% of linked calls across 3 live dumps. Root cause: the re-correlation sweep's orphan filter can't tell "never processed" apart from "real-time pipeline is still mid-flight" -- a call whose STT/scene-extraction/correlation chain hasn't finished has no incident_id/corr_path set yet, so the sweep picks it up and correlates it independently, sometimes onto a different incident than the real-time path lands on moments later. Confirmed in review: the incident-write layer appends call_id unconditionally with no cross-incident dedup guard, so preventing the second correlation attempt is the only lever available.

Fix: _run_intelligence_pipeline marks intelligence_started_at before any slow step; the sweep holds back any call whose marker is under 15 minutes old. Also covers /calls/{id}/reprocess (same pipeline function). 15 min (not the original 5) because neither the OpenAI nor Gemini client sets a request timeout (filed #153) -- raising the hold-back is free on the recovery side since a completed call is already excluded by the existing corr_path filter regardless of this constant.

recorrelation_sweep.py had zero test coverage before this -- new file added.

Reviewed twice (drb-correlation-review): first pass confirmed the race is real and the fix targets the right layer, flagged the 5-min constant as an unmeasured guess against an unbounded client-timeout tail (addressed, now 15 min + #153 filed) and that DEFERRED.md's stale ~6min sweep-timing figure needed a note (addressed). Note: this PR was rebuilt once after review -- the first draft was accidentally branched off a stale local main that predated #139's merge; rebuilt clean off current main with identical content, same review findings apply.

Sandboxed pytest: 381 -> 387.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

Closes #131. Same call_id ends up in TWO incidents' `call_ids`, byte-identical extracted data, ~2% of linked calls across 3 live dumps. Root cause: the re-correlation sweep's orphan filter can't tell "never processed" apart from "real-time pipeline is still mid-flight" -- a call whose STT/scene-extraction/correlation chain hasn't finished has no `incident_id`/`corr_path` set yet, so the sweep picks it up and correlates it independently, sometimes onto a different incident than the real-time path lands on moments later. Confirmed in review: the incident-write layer appends `call_id` unconditionally with no cross-incident dedup guard, so preventing the second correlation attempt is the only lever available. Fix: `_run_intelligence_pipeline` marks `intelligence_started_at` before any slow step; the sweep holds back any call whose marker is under 15 minutes old. Also covers `/calls/{id}/reprocess` (same pipeline function). 15 min (not the original 5) because neither the OpenAI nor Gemini client sets a request timeout (filed #153) -- raising the hold-back is free on the recovery side since a completed call is already excluded by the existing `corr_path` filter regardless of this constant. `recorrelation_sweep.py` had zero test coverage before this -- new file added. Reviewed twice (`drb-correlation-review`): first pass confirmed the race is real and the fix targets the right layer, flagged the 5-min constant as an unmeasured guess against an unbounded client-timeout tail (addressed, now 15 min + #153 filed) and that DEFERRED.md's stale ~6min sweep-timing figure needed a note (addressed). Note: this PR was rebuilt once after review -- the first draft was accidentally branched off a stale local `main` that predated #139's merge; rebuilt clean off current `main` with identical content, same review findings apply. Sandboxed pytest: 381 -> 387. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

logan added 1 commit 2026-09-14 00:30:54 -04:00

correlator: stop the re-correlation sweep racing an in-flight upload (#131 ) 8dd636af8f

server-26#131: same call_id ends up in TWO incidents' call_ids, byte-identical
extracted data, ~2% of linked calls across 3 live dumps. Root cause: the
sweep's orphan filter (incident_id/incident_ids/corr_path all absent) can't
tell 'never processed' apart from 'real-time pipeline is still mid-flight' --
a call whose STT/scene-extraction/correlation chain (routers/upload.py
_run_intelligence_pipeline) hasn't finished yet has none of those fields set,
so the sweep picks it up and correlates it independently, sometimes onto a
different incident than the real-time path lands on. Confirmed in review:
_update_incident/_create_incident append call_id to an incident's call_ids
unconditionally, with no cross-incident dedup guard -- preventing the second
correlation attempt is the only lever available at this layer.

Fix: _run_intelligence_pipeline marks intelligence_started_at on the call doc
before any slow step; the sweep holds back any call whose marker is under 15
minutes old (raised from an initial 5 -- see below), regardless of how
orphaned it otherwise looks. No marker at all (pre-#131 call doc, or the
marker write itself failed) is not held back -- absence isn't evidence of an
in-flight pipeline, and that's #131's own pre-existing population. Also
covers the /calls/{id}/reprocess path, which calls the same
_run_intelligence_pipeline.

15 min, not 5: neither the OpenAI Whisper client nor the Gemini call in
llm_correlator.py sets a request timeout (filed server-26#153), so 5 min was
a guess against an unbounded tail -- drb-correlation-review flagged this.
Raising it is free on the recovery side: a call that finished processing
(linked or genuinely orphaned) always has corr_path set and is already
excluded by the sweep's other filter, so this constant only ever delays
calls that are still actually running. DEFERRED.md row 52 (outside this
repo, Version 5C root) updated to flag its ~6 min timing figure as stale.

recorrelation_sweep.py had zero test coverage before this. New file covers
the guard function's boundary (age < threshold vs exactly-at vs old vs
missing vs unparseable) and one integration-shaped test proving a racing call
never reaches correlate_call while a genuinely-orphaned call still does.

Sandboxed pytest: 381 -> 387.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

logan merged commit a9197709f8 into main

2026-09-14 00:31:15 -04:00

logan deleted branch fix/131-sweep-race-duplicate-link

2026-09-14 00:31:15 -04:00

logan referenced this pull request

2026-09-14 00:31:16 -04:00

Same call_id linked into 2 incident records with identical extracted data (~2% of linked calls) #131

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#154