Sweep retry budget can expire before the incident the orphan belongs to exists #27

Open
opened 2026-08-20 03:36:20 -04:00 by logan · 1 comment
Owner

recorrelation_sweep.py allows MAX_SWEEP_ATTEMPTS = 3 link-only retries, summary_interval_minutes (2) apart, then writes corr_path: "unlinked" as a permanent tombstone. That is a ~6 minute budget from when the call ends, and the sweep always runs create_if_new=False - so a call can be tombstoned before the incident it belongs to has been created.

Observed 2026-08-20: orphan f321b047 ("Fire 1 ... courtesy drop-off", 02:46:51) exhausted its three attempts around 02:53, while incident 9d376ffe - the completion of that same drop-off by that same unit - was not created until 02:50. The orphan carries units ["Fire 1", "22-47"], overlapping the incident's units ["Fire 1", "5-5", "5-5-2"], so the unit-continuity path would have caught it given one more pass.

Note this is an UNDER-linking bug, and every fix for it pushes toward more merging - the opposite direction from server-26#22, where the entire observed failure mode was over-merging. It was left alone deliberately in that pass and should be re-validated against post-fix data before being worked.

Fix direction: budget in wall-clock time rather than attempt count, or do not tombstone until the recorrelation_scan_minutes window itself has passed.

Source: CORRELATION_REVIEW_0820.md section 3, ranked #7.

`recorrelation_sweep.py` allows `MAX_SWEEP_ATTEMPTS = 3` link-only retries, `summary_interval_minutes` (2) apart, then writes `corr_path: "unlinked"` as a permanent tombstone. That is a ~6 minute budget from when the call ends, and the sweep always runs `create_if_new=False` - so a call can be tombstoned before the incident it belongs to has been created. Observed 2026-08-20: orphan `f321b047` ("Fire 1 ... courtesy drop-off", 02:46:51) exhausted its three attempts around 02:53, while incident `9d376ffe` - the completion of that same drop-off by that same unit - was not created until 02:50. The orphan carries units ["Fire 1", "22-47"], overlapping the incident's units ["Fire 1", "5-5", "5-5-2"], so the unit-continuity path would have caught it given one more pass. Note this is an UNDER-linking bug, and every fix for it pushes toward more merging - the opposite direction from server-26#22, where the entire observed failure mode was over-merging. It was left alone deliberately in that pass and should be re-validated against post-fix data before being worked. Fix direction: budget in wall-clock time rather than attempt count, or do not tombstone until the `recorrelation_scan_minutes` window itself has passed. Source: CORRELATION_REVIEW_0820.md section 3, ranked #7.
Author
Owner

Board minutes #62 (2026-08-24), section 9. Recording so this does not get quietly marked covered.

#57 recurring correlation dump does not cover this issue and cannot be made to. #57 counts orphan totals and retry-exhaustion state. #27 is a tombstone-timing race and needs the call-end to incident-created timestamp delta, which #57 never collects. Running #57 every cycle and treating #27 as instrumented is ceremony that reads as diligence.

This stays a standalone fix: retry budget in wall-clock time, not attempt count (CTO draft #60 rec. 3). Not folded into #57.

**Board minutes #62 (2026-08-24), section 9.** Recording so this does not get quietly marked covered. #57 recurring correlation dump **does not** cover this issue and cannot be made to. #57 counts orphan totals and retry-exhaustion state. #27 is a tombstone-timing race and needs the **call-end to incident-created timestamp delta**, which #57 never collects. Running #57 every cycle and treating #27 as instrumented is ceremony that reads as diligence. This stays a standalone fix: retry budget in **wall-clock time**, not attempt count (CTO draft #60 rec. 3). Not folded into #57.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#27