Correlation over-merge veto uses a bare 0.82 literal, looser than the 0.93 match bar it sits beside #86

New Issue

2026-08-29T02:43:30-04:00

logan commented

2026-08-29 02:43:30 -04:00

Found during the standing correlation static pass, unattended run 2026-08-29 (live SHA 3df427f, which equalled origin/main, so this is what is running). Searched open and closed first: #80 is scene-level vs call-level judgment, #74 (closed) was the signed-idle recency gate, #37/#48 are elsewhere. No existing issue covers this threshold.

What is wrong

The content-divergence veto added to correlation mechanism B gates on a hardcoded literal 0.82 at drb-c2-core/app/internal/incident_correlator.py:1637 and :1650.

The system's own definition of a confirmed embedding match is settings.embedding_similarity_threshold = 0.93 (app/config.py:75). So the bar the veto uses to decide two calls have diverged is materially looser than the bar the slow path uses to decide they match. There is a band between 0.82 and 0.93 where a pair is simultaneously "not a match" and "not divergent enough to veto".

Why it matters

The veto exists specifically to break the unit-overlap positive-feedback loop documented in DEFERRED.md — the failure mode where an incident that has already absorbed many units keeps absorbing more because unit overlap alone keeps clearing the bar.

Two calls about genuinely different jobs on the same dispatch channel share a great deal of surface language: ten-codes, unit numbers, the local address format, the dispatcher's own phrasing. That shared register plausibly scores above 0.82 on a routine basis. If it does, the veto rarely fires, and the fix does not actually deliver the over-merge protection it was added for.

Failure scenario. An incident has accumulated many units. A later, unrelated call shares exactly one unit with it, and the incident has no coordinates, so the embedding check is what stands between them. The pair scores 0.86 — below the 0.93 match bar, above the 0.82 veto bar. The veto does not fire and the call merges. The incident grows, its unit set grows, and the next unrelated call is now more likely to overlap. That is the loop the veto was meant to cut.

What is not known

This is unverified against real similarity scores. Only the gate and dedup paths in incident_correlator.py have test coverage; this branch has none, and no live scores were sampled. The 0.82 may well have been chosen deliberately and be correct. The defect being reported is that it is a bare literal with no recorded reasoning and no relationship to the configured threshold it sits next to — so nobody can tell whether it is tuned or accidental, and it cannot be adjusted without a code change and a deploy.

What done looks like

The value moves into config.py next to embedding_similarity_threshold, with a comment stating what it is for and why it differs.
Either a stated rationale for it being looser than the match bar, or it is raised to match.
A test covering the band between the two thresholds, so the intended behaviour is pinned rather than inferred.
Ideally, the actual distribution of divergence scores is sampled during the next live AI window before the number is changed — tuning it blind trades one guess for another.

Refs #5, #22, #35, DEFERRED.md (unit-overlap positive feedback).

Found during the standing correlation static pass, unattended run 2026-08-29 (live SHA `3df427f`, which equalled `origin/main`, so this is what is running). Searched open and closed first: #80 is scene-level vs call-level *judgment*, #74 (closed) was the signed-idle recency gate, #37/#48 are elsewhere. No existing issue covers this threshold. ## What is wrong The content-divergence veto added to correlation mechanism B gates on a **hardcoded literal `0.82`** at `drb-c2-core/app/internal/incident_correlator.py:1637` and `:1650`. The system's own definition of a confirmed embedding match is `settings.embedding_similarity_threshold = 0.93` (`app/config.py:75`). So the bar the veto uses to decide two calls have *diverged* is materially looser than the bar the slow path uses to decide they *match*. There is a band between 0.82 and 0.93 where a pair is simultaneously "not a match" and "not divergent enough to veto". ## Why it matters The veto exists specifically to break the unit-overlap positive-feedback loop documented in `DEFERRED.md` — the failure mode where an incident that has already absorbed many units keeps absorbing more because unit overlap alone keeps clearing the bar. Two calls about genuinely different jobs on the same dispatch channel share a great deal of surface language: ten-codes, unit numbers, the local address format, the dispatcher's own phrasing. That shared register plausibly scores above 0.82 on a routine basis. If it does, the veto rarely fires, and the fix does not actually deliver the over-merge protection it was added for. **Failure scenario.** An incident has accumulated many units. A later, unrelated call shares exactly one unit with it, and the incident has no coordinates, so the embedding check is what stands between them. The pair scores 0.86 — below the 0.93 match bar, above the 0.82 veto bar. The veto does not fire and the call merges. The incident grows, its unit set grows, and the next unrelated call is now more likely to overlap. That is the loop the veto was meant to cut. ## What is not known **This is unverified against real similarity scores.** Only the gate and dedup paths in `incident_correlator.py` have test coverage; this branch has none, and no live scores were sampled. The 0.82 may well have been chosen deliberately and be correct. The defect being reported is that **it is a bare literal with no recorded reasoning and no relationship to the configured threshold it sits next to** — so nobody can tell whether it is tuned or accidental, and it cannot be adjusted without a code change and a deploy. ## What done looks like - The value moves into `config.py` next to `embedding_similarity_threshold`, with a comment stating what it is for and why it differs. - Either a stated rationale for it being looser than the match bar, or it is raised to match. - A test covering the band between the two thresholds, so the intended behaviour is pinned rather than inferred. - Ideally, the actual distribution of divergence scores is sampled during the next live AI window before the number is changed — tuning it blind trades one guess for another. Refs #5, #22, #35, `DEFERRED.md` (unit-overlap positive feedback).

logan referenced this issue

2026-08-31 02:45:53 -04:00

Scene correlation is judged on the call-level (primary-scene) embedding and severity, not the scene it is correlating #80

logan referenced this issue

2026-08-31 02:45:55 -04:00

LLM correlation tier sees the whole-call transcript for every scene, so it cannot separate scenes and pays N times for one prompt #91

logan referenced this issue

2026-09-06 15:11:07 -04:00

content_divergence veto goes dark when the scene embedding is legitimately None (post-#80/#95) #103

logan referenced this issue

2026-09-07 04:42:16 -04:00

Consensus tiebreaker manufactures incidents — resolves ~all disagreements as "new", overriding the LLM's "link" #115