corr_debug is overwritten per scene on the shared call doc, so #35's "LLM tier decides 3.5% of links" is a per-call blend and cannot answer whether the tier works #96

New Issue

2026-09-01T02:43:34-04:00

logan commented

2026-09-01 02:43:34 -04:00

Found by the standing P0 correlation static pass, unattended run 2026-09-01, at b722223. New since the CORRELATION_REVIEW_0820.md baseline — the baseline's §7 complaint was that these fields were not exposed at all, and that part is now fixed at admin.py:134-136. This is the next layer of the same problem: they are exposed, but they are not trustworthy.

The defect

Every scene of a multi-scene call writes corr_debug onto the same call doc (incident_correlator.py:1321), with partial keys, last-write-wins. A call carrying three scenes ends up with corr_path from whichever scene wrote last and corr_consensus / corr_llm_action / corr_llm_reason from a different one. The record does not describe any single correlation decision — it is a splice of several.

admin.py:292-295 then tallies these per call.

Why this matters more than a cosmetic debug bug

It invalidates the measurement #35 is built on. #35 states the LLM correlation tier decides only 3.5% of links and that the fast/thin path dominates. That number is computed off a per-call blend, so it is not a per-link rate and cannot be converted into one after the fact. The tier could be deciding materially more or materially less; the instrument cannot tell us.

Consequences that follow:

#35 cannot be actioned on its current evidence. Any tuning aimed at raising the LLM tier's share would be steering by a broken gauge.
The standing evaluation procedure (#57) and the correlation dumps saved per run compare corr_path distribution across runs. For multi-scene calls that trend line is noise, and multi-scene calls are exactly the population where over-split (#5) and over-merge live.
It hides which scene produced which decision, which is the single fact needed to debug an over-split.

Fix shape

corr_debug should be per scene, keyed by scene index, not a single map on the call doc — e.g. a list or a map keyed by scene id, each entry complete in itself. admin.py then tallies per scene, which is the unit correlation actually operates on. Once that lands, #35's 3.5% figure must be re-measured before anything is concluded from it, and this issue should be referenced when it is.

Cheap interim option if the schema change is too large for one pass: write the scene index into each corr_debug entry and have the tally count only single-scene calls, so at least the reported rate is honest about its population.

Test gap

Nothing covers the multi-scene corr_debug write. drb-c2-core is 290 pass / 0 fail at HEAD; this path is outside the tested gate and dedup paths, like the other two findings from this pass.

Related: #35 (its metric is the thing invalidated), #57, #80, #5. Comment left on #35 so its number is not read at face value in the meantime.

Found by the standing P0 correlation static pass, unattended run 2026-09-01, at `b722223`. **New since the `CORRELATION_REVIEW_0820.md` baseline** — the baseline's §7 complaint was that these fields were not exposed at all, and that part is now fixed at `admin.py:134-136`. This is the next layer of the same problem: they are exposed, but they are not trustworthy. ## The defect Every scene of a multi-scene call writes `corr_debug` onto **the same call doc** (`incident_correlator.py:1321`), with partial keys, last-write-wins. A call carrying three scenes ends up with `corr_path` from whichever scene wrote last and `corr_consensus` / `corr_llm_action` / `corr_llm_reason` from a different one. The record does not describe any single correlation decision — it is a splice of several. `admin.py:292-295` then tallies these **per call**. ## Why this matters more than a cosmetic debug bug **It invalidates the measurement #35 is built on.** #35 states the LLM correlation tier decides only 3.5% of links and that the fast/thin path dominates. That number is computed off a per-call blend, so it is not a per-link rate and cannot be converted into one after the fact. The tier could be deciding materially more or materially less; the instrument cannot tell us. Consequences that follow: - **#35 cannot be actioned on its current evidence.** Any tuning aimed at raising the LLM tier's share would be steering by a broken gauge. - The standing evaluation procedure (#57) and the correlation dumps saved per run compare `corr_path` distribution across runs. For multi-scene calls that trend line is noise, and multi-scene calls are exactly the population where over-split (#5) and over-merge live. - It hides which scene produced which decision, which is the single fact needed to debug an over-split. ## Fix shape `corr_debug` should be **per scene, keyed by scene index**, not a single map on the call doc — e.g. a list or a map keyed by scene id, each entry complete in itself. `admin.py` then tallies per scene, which is the unit correlation actually operates on. Once that lands, **#35's 3.5% figure must be re-measured before anything is concluded from it**, and this issue should be referenced when it is. Cheap interim option if the schema change is too large for one pass: write the scene index into each `corr_debug` entry and have the tally count only single-scene calls, so at least the reported rate is honest about its population. ## Test gap Nothing covers the multi-scene `corr_debug` write. `drb-c2-core` is 290 pass / 0 fail at HEAD; this path is outside the tested gate and dedup paths, like the other two findings from this pass. Related: #35 (its metric is the thing invalidated), #57, #80, #5. Comment left on #35 so its number is not read at face value in the meantime.