Scene correlation is judged on the call-level (primary-scene) embedding and severity, not the scene it is correlating #80

New Issue

2026-08-26T09:31:11-04:00

logan commented

2026-08-26 09:31:11 -04:00

Found during the standing correlation static pass, 2026-08-26. Not a duplicate of the DEFERRED.md multi-scene entry: that entry is about link fan-out, this is about the wrong content reaching the one guard that bounds over-merging.

What happens

intelligence.py writes only the primary scene embedding to the call document. _build_context in incident_correlator.py then reads call_embedding / call_severity back off that document, and _correlate_with_consensus never passes a per-scene embedding or severity.

So on any multi-scene call, every scene after the first is judged on scene 1 content by:

both content-divergence vetoes,
the embedding scoring in the slow path,
the thin-path test and the incident creation gate (severity).

Why it compounds

The incident that scene 2 links to (or creates) then stores scene 1 embedding. That poisons the divergence check for every later call tested against that incident. The error does not stay local to one call.

Direction

Over-merging. The veto that exists to break unit-overlap feedback is precisely the thing being fed the wrong vector.

Fix shape

Pass scene["embedding"] and scene["severity"] through preview_correlation into ctx, falling back to the call document when a scene does not carry them. No threshold change, so it does not trade against the over-splitting currently measured on the main path (see #5).

Not verified

No live window this run (morning slot, AI off for cost). Static read only, and neither upload.py nor the fit function has test coverage.

Refs #5, #35, #74.

Found during the standing correlation static pass, 2026-08-26. Not a duplicate of the DEFERRED.md multi-scene entry: that entry is about link fan-out, this is about the wrong content reaching the one guard that bounds over-merging. ## What happens `intelligence.py` writes only the **primary** scene embedding to the call document. `_build_context` in `incident_correlator.py` then reads `call_embedding` / `call_severity` back off that document, and `_correlate_with_consensus` never passes a per-scene embedding or severity. So on any multi-scene call, **every scene after the first is judged on scene 1 content** by: - both content-divergence vetoes, - the embedding scoring in the slow path, - the thin-path test and the incident creation gate (severity). ## Why it compounds The incident that scene 2 links to (or creates) then **stores scene 1 embedding**. That poisons the divergence check for every later call tested against that incident. The error does not stay local to one call. ## Direction **Over-merging.** The veto that exists to break unit-overlap feedback is precisely the thing being fed the wrong vector. ## Fix shape Pass `scene["embedding"]` and `scene["severity"]` through `preview_correlation` into `ctx`, falling back to the call document when a scene does not carry them. No threshold change, so it does **not** trade against the over-splitting currently measured on the main path (see #5). ## Not verified No live window this run (morning slot, AI off for cost). Static read only, and neither `upload.py` nor the fit function has test coverage. Refs #5, #35, #74.

logan referenced this issue

2026-08-26 09:31:44 -04:00

Scene over-splitting: one radio conversation becomes two incidents #5

logan commented

2026-08-27 02:49:23 -04:00

Confirmed with a concrete mechanism by the standing correlation pass, unattended run 2026-08-27.

intelligence.py:356 writes only the primary scene's severity to the call doc, and :368 only the primary scene's embedding. But _build_context (incident_correlator.py:784) re-reads call_doc["severity"] for every scene's _is_thin_call check.

Consequence: a "10-4" second scene on a call whose first scene was major is never judged thin. It falls through to the fit path, returns no_signal on a dispatch channel, and opens its own incident.

Direction: over-splitting — and it is a new mechanism created by the _is_thin_call tightening, not the multi-scene case already described in DEFERRED.md.

Same line is the concrete evidence for this issue's premise: per-scene embeddings are computed at intelligence.py:329 and then discarded, so the content-divergence veto judges scene N against scene 1's vector.

Not fixed this run. Fixing it means writing per-scene severity and embedding through to whatever _build_context reads, which is a correlation-behaviour change that wants its own evaluation window rather than riding along with unrelated work.

**Confirmed with a concrete mechanism by the standing correlation pass, unattended run 2026-08-27.** `intelligence.py:356` writes only the **primary** scene's `severity` to the call doc, and `:368` only the primary scene's `embedding`. But `_build_context` (`incident_correlator.py:784`) re-reads `call_doc["severity"]` for **every** scene's `_is_thin_call` check. Consequence: a "10-4" second scene on a call whose first scene was `major` is never judged thin. It falls through to the fit path, returns `no_signal` on a dispatch channel, and opens its own incident. **Direction: over-splitting** — and it is a *new* mechanism created by the `_is_thin_call` tightening, not the multi-scene case already described in `DEFERRED.md`. Same line is the concrete evidence for this issue's premise: per-scene embeddings are computed at `intelligence.py:329` and then discarded, so the content-divergence veto judges scene N against scene 1's vector. Not fixed this run. Fixing it means writing per-scene severity and embedding through to whatever `_build_context` reads, which is a correlation-behaviour change that wants its own evaluation window rather than riding along with unrelated work.

logan commented

2026-08-28 02:44:28 -04:00

Extends this issue — same root cause, same fix site, two more fields. Static pass, unattended run 2026-08-28, at d18e4f0.

This issue names the embedding and severity as the fields that bleed from the primary scene. There are two more, and one of them is worse than either, because it does not merely mis-score a guard — it manufactures a positive fit signal out of nothing.

location_coords bleeds too. _build_context takes location, tags and units per scene, but falls back to the call document for the rest:

coords = location_coords or call_doc.get("location_coords") (incident_correlator.py, in _build_context)
call_severity = call_doc.get("severity") (same function)

and intelligence.py writes only primary["location_coords"] and primary["severity"] onto the call doc. So scene 2 — which usually has no geocode of its own — is fit-tested against scene 1's pin.

Two distinct consequences, neither of which the embedding/severity story covers:

It can score location_proximity on coordinates belonging to a different event. That is not a weakened veto; it is a fabricated positive.
With coords truthy, scene 2 is never thin, so it takes the slow path and skips the thin-call tiering entirely.

Both push the same direction this issue already names — over-merge — and both amplify the fan-out.

Fix shape is unchanged, just wider: pass scene["location_coords"] and scene["severity"] through preview_correlation into ctx alongside the embedding, falling back to the call document only when the scene genuinely does not carry them. One change, four fields.

Cost note, same file. intelligence.py computes and pays for a per-scene embedding, but only the primary's is ever written to the call doc and therefore only the primary's ever reaches the correlator. Every secondary scene's embedding is bought and discarded. That is measurable spend against Gate B5 (#45) with no consumer.

Not verified: no live window this run (02:37 slot, AI off for cost). Static read only. The multi-scene path still has zero test coverage, so the coords bleed is read from source and unexercised — worth a test alongside the fix rather than after it.

Refs #5, #35, #45, #74.

**Extends this issue — same root cause, same fix site, two more fields. Static pass, unattended run 2026-08-28, at `d18e4f0`.** This issue names the **embedding** and **severity** as the fields that bleed from the primary scene. There are two more, and one of them is worse than either, because it does not merely mis-score a guard — it manufactures a positive fit signal out of nothing. **`location_coords` bleeds too.** `_build_context` takes `location`, `tags` and `units` per scene, but falls back to the call document for the rest: - `coords = location_coords or call_doc.get("location_coords")` (`incident_correlator.py`, in `_build_context`) - `call_severity = call_doc.get("severity")` (same function) and `intelligence.py` writes only `primary["location_coords"]` and `primary["severity"]` onto the call doc. So scene 2 — which usually has no geocode of its own — is fit-tested against **scene 1's pin**. Two distinct consequences, neither of which the embedding/severity story covers: 1. It can score `location_proximity` on coordinates belonging to a **different event**. That is not a weakened veto; it is a fabricated positive. 2. With `coords` truthy, scene 2 is **never thin**, so it takes the slow path and skips the thin-call tiering entirely. Both push the same direction this issue already names — **over-merge** — and both amplify the fan-out. **Fix shape is unchanged**, just wider: pass `scene["location_coords"]` and `scene["severity"]` through `preview_correlation` into `ctx` alongside the embedding, falling back to the call document only when the scene genuinely does not carry them. One change, four fields. **Cost note, same file.** `intelligence.py` computes and pays for a per-scene embedding, but only the primary's is ever written to the call doc and therefore only the primary's ever reaches the correlator. Every secondary scene's embedding is bought and discarded. That is measurable spend against Gate B5 (#45) with no consumer. **Not verified:** no live window this run (02:37 slot, AI off for cost). Static read only. The multi-scene path still has zero test coverage, so the coords bleed is read from source and unexercised — worth a test alongside the fix rather than after it. Refs #5, #35, #45, #74.

logan referenced this issue from a commit

2026-08-28 02:51:11 -04:00

Stop a back-dated call from silently disabling every recency gate (server-26#74)

logan referenced this issue

2026-08-28 03:10:09 -04:00

_call_fits_incident still uses signed idle, so the sweep path skips both over-merge vetoes on back-dated calls #74

logan referenced this issue

2026-08-29 02:43:30 -04:00

Correlation over-merge veto uses a bare 0.82 literal, looser than the 0.93 match bar it sits beside #86

logan referenced this issue

2026-08-30 02:44:06 -04:00

The primary scene's geocode leaks into every other scene's correlation, fabricating a location match #87

logan referenced this issue

2026-08-30 02:44:06 -04:00

`resolved` and `reassignment` are prompt-only invariants with no validator, so a string "false" resolves an incident #88

logan commented

2026-08-31 02:45:53 -04:00

Confirmed and sharpened at 29c2fb1, static correlation pass, unattended run 2026-08-31. This issue is correct, and it is a stronger over-merging force than server-26#87 - which was fixed tonight on the geocode channel only. Same leak, different channel.

The per-scene embedding is computed and then thrown away:

intelligence.py:329 embeds each scene.
intelligence.py:369 writes only primary["embedding"] to the call doc.
_build_context reads call_doc.get("embedding") (incident_correlator.py:780), because _correlate_with_consensus has no embedding parameter at all (upload.py:210-224).

So a non-primary scene is correlated on the primary scene's content. That propagates into three places:

the slow path (incident_correlator.py:1212),
the cross-talkgroup path (:1174),
worst - the content-divergence veto (:1636, :1649), which compares scene 1's embedding against an incident built from scene 1.

That third consequence is the one that matters. The veto is structurally incapable of firing for a non-primary scene: it is asked whether a thing diverges from itself. So the over-merge veto that server-26#86 proposes to tune is, for every non-primary scene, not merely mis-thresholded but inoperative.

Also: the discarded per-scene embeddings are paid for. Every extra scene on a call buys an embedding call whose result is written nowhere.

The fix is not a threshold change. It is threading the scene's own embedding through _correlate_with_consensus into _build_context so each scene is judged on its own content. That is a signature change across upload.py and incident_correlator.py, which is why it was not taken in the same run as #87.

Sequencing note. #86 must not be tuned before this lands. Tuning a veto that cannot fire for the affected population would fit the threshold to the wrong data.

**Confirmed and sharpened at `29c2fb1`, static correlation pass, unattended run 2026-08-31.** This issue is correct, and it is a *stronger* over-merging force than server-26#87 - which was fixed tonight on the geocode channel only. Same leak, different channel. The per-scene embedding is computed and then thrown away: - `intelligence.py:329` embeds **each** scene. - `intelligence.py:369` writes only `primary["embedding"]` to the call doc. - `_build_context` reads `call_doc.get("embedding")` (`incident_correlator.py:780`), because `_correlate_with_consensus` has no embedding parameter at all (`upload.py:210-224`). So a non-primary scene is correlated on the **primary scene's** content. That propagates into three places: 1. the slow path (`incident_correlator.py:1212`), 2. the cross-talkgroup path (`:1174`), 3. worst - the content-divergence veto (`:1636`, `:1649`), which compares scene 1's embedding against an incident **built from scene 1**. That third consequence is the one that matters. The veto is structurally incapable of firing for a non-primary scene: it is asked whether a thing diverges from itself. So the over-merge veto that server-26#86 proposes to tune is, for every non-primary scene, not merely mis-thresholded but inoperative. Also: the discarded per-scene embeddings are paid for. Every extra scene on a call buys an embedding call whose result is written nowhere. **The fix is not a threshold change.** It is threading the scene's own embedding through `_correlate_with_consensus` into `_build_context` so each scene is judged on its own content. That is a signature change across `upload.py` and `incident_correlator.py`, which is why it was not taken in the same run as #87. **Sequencing note.** #86 must not be tuned before this lands. Tuning a veto that cannot fire for the affected population would fit the threshold to the wrong data.

logan referenced this issue

2026-08-31 02:45:55 -04:00

LLM correlation tier sees the whole-call transcript for every scene, so it cannot separate scenes and pays N times for one prompt #91

logan referenced this issue

2026-09-01 02:43:34 -04:00

Non-primary scenes inherit the primary scene's severity from the call doc, so bdb57ae closed one of three leaks and the other two now carry the weight alone #95

logan referenced this issue

2026-09-01 02:43:34 -04:00

corr_debug is overwritten per scene on the shared call doc, so #35's "LLM tier decides 3.5% of links" is a per-call blend and cannot answer whether the tier works #96

logan referenced this issue

2026-09-01 02:44:04 -04:00

The primary scene's geocode leaks into every other scene's correlation, fabricating a location match #87

logan commented

2026-09-01 02:44:06 -04:00

Now higher-priority than when filed, and it has a sibling. Unattended run 2026-09-01.

bdb57ae (#87, live) removed the third channel of the same leak — the call doc's location_coords fallback in _build_context. The primary-scene leak had three channels; coords was the one that got fixed. This issue (embedding, incident_correlator.py:780) and the newly filed #95 (severity, :784) are the two that remain, and they now carry the full weight of the primary scene's identity alone.

Both have the same root: intelligence.py unions tags, units and vehicles across scenes but writes severity (:356) and embedding (:368) from processed[0] only.

Fix #95 and this one in a single pass. They are one defect with two field names, and doing them separately repeats exactly the partial-fix pattern that bdb57ae left behind. The fix shape is the one bdb57ae already established for coords: make the value per-scene in the extraction output and read the scene's own value with no call-doc fallback.

No test covers _build_context's embedding or severity sourcing — the fix needs a multi-scene fixture where scene 1 is major and scene 2 is content-free.

**Now higher-priority than when filed, and it has a sibling. Unattended run 2026-09-01.** `bdb57ae` (#87, live) removed the third channel of the same leak — the call doc's `location_coords` fallback in `_build_context`. The primary-scene leak had three channels; coords was the one that got fixed. This issue (`embedding`, `incident_correlator.py:780`) and the newly filed **#95** (`severity`, `:784`) are the two that remain, and they now carry the full weight of the primary scene's identity alone. Both have the same root: `intelligence.py` unions `tags`, `units` and `vehicles` across scenes but writes `severity` (`:356`) and `embedding` (`:368`) from `processed[0]` only. **Fix #95 and this one in a single pass.** They are one defect with two field names, and doing them separately repeats exactly the partial-fix pattern that `bdb57ae` left behind. The fix shape is the one `bdb57ae` already established for coords: make the value per-scene in the extraction output and read the scene's own value with no call-doc fallback. No test covers `_build_context`'s embedding or severity sourcing — the fix needs a multi-scene fixture where scene 1 is `major` and scene 2 is content-free.

logan commented

2026-09-06 15:11:33 -04:00

Fixed in a9d1d24 (branch fix/scene-context-leak-80-95, not yet merged/deployed).

_build_context no longer reads embedding / severity off the call doc — each scene threads its own value through preview_correlation / correlate_call, exactly like the #87 coords fix. recorrelation_sweep passes the call doc value explicitly (whole-call re-link, link-only). +2 regression tests in test_incident_identity.py; full c2-core suite green (295 passed).

Remaining leg of this defect class split to #102 (llm_correlator._call_block still reads the whole-call transcript per scene). Review-flagged side effects: #103 (divergence veto on None embedding), #104 (reprocess stale severity).

Fixed in `a9d1d24` (branch `fix/scene-context-leak-80-95`, not yet merged/deployed). `_build_context` no longer reads `embedding` / `severity` off the call doc — each scene threads its own value through `preview_correlation` / `correlate_call`, exactly like the #87 coords fix. `recorrelation_sweep` passes the call doc value explicitly (whole-call re-link, link-only). +2 regression tests in `test_incident_identity.py`; full c2-core suite green (295 passed). Remaining leg of this defect class split to #102 (`llm_correlator._call_block` still reads the whole-call transcript per scene). Review-flagged side effects: #103 (divergence veto on None embedding), #104 (reprocess stale severity).

logan closed this issue

2026-09-06 15:11:33 -04:00

logan referenced a pull request that will close this issue

2026-09-06 17:49:08 -04:00

correlator: judge each scene on its own embedding + severity (#80, #95) #105

logan referenced this issue from a commit

2026-09-06 17:49:25 -04:00

Merge pull request 'correlator: judge each scene on its own embedding + severity (#80, #95)' (#105) from fix/scene-context-leak-80-95 into main

logan referenced this issue from a commit

2026-09-07 00:12:59 -04:00

correlator: address #102 review — 0-based segment labels, never-empty slice

logan added the area/correlation