correlator+summarizer: per-scene call-doc storage, fixes #96 and #114's real fix

Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).

Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
  - the existing flat corr_* fields, unchanged (last-scene-wins, the safe
    backward-compatible default for any reader that doesn't know about
    `scenes` yet)
  - a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
    corr_debug}, via doc_set(..., merge=True). Firestore's
    DocumentReference.set(data, merge=True) recursively merges nested map
    fields by key (documented SDK behaviour, not assumed) — a write to
    scenes.1 merges alongside an existing scenes.0 instead of replacing the
    whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.

admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.

summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.

Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.

Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.

Fixes server-26#96, server-26#114

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

drb-c2-core/app/internal/incident_correlator.py

@@ -709,6 +709,7 @@ async def correlate_call(
     embedding: Optional[list] = None,
     severity: Optional[str] = None,
     transcript: Optional[str] = None,
+    scene_index: int = 0,
 ) -> Optional[str]:
     """
     Link call_id to an existing incident or create a new one.
@@ -718,6 +719,11 @@ async def correlate_call(
     Callers that re-correlate a whole call rather than a scene — the
     recorrelation sweep — pass the call doc's stored values explicitly; they are
     no longer read from the doc inside _build_context.
+
+    ``scene_index`` (server-26#96) identifies which scene of the call this
+    decision belongs to for the per-scene ``scenes`` map written by
+    _apply_and_log. Defaults to 0 — correct for every caller here, since this
+    entry point always re-correlates a call as a single unit, not a scene loop.
     """
     ctx = await _build_context(
         call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units,
@@ -726,6 +732,7 @@ async def correlate_call(
         tags=tags, incident_type=incident_type, location=location,
         reassignment=reassignment, create_if_new=create_if_new,
         embedding=embedding, severity=severity, transcript=transcript,
+        scene_index=scene_index,
     )
     decision = _run_decision(ctx)
     return await _apply_and_log(decision, ctx)
@@ -750,6 +757,7 @@ async def preview_correlation(
     embedding: Optional[list] = None,
     severity: Optional[str] = None,
     transcript: Optional[str] = None,
+    scene_index: int = 0,
 ) -> dict:
     """
     Run the rules engine and return the decision WITHOUT committing to Firestore.
@@ -763,6 +771,15 @@ async def preview_correlation(
       matched_incident  the candidate incident doc  (action == "link")
       incident_type     resolved type after tag inference  (action == "new")
       corr_debug        fields to persist on the call doc
+
+    ``scene_index`` (server-26#96) — which scene of the call (upload.py's
+    ``for scene_index, scene in enumerate(scenes):`` loop) this call is. It
+    rides through ctx to _apply_and_log, which uses it as the key under the
+    call doc's ``scenes`` map so each scene's corr_debug/transcript lands in
+    its own map entry instead of colliding on the shared flat fields. Defaults
+    to 0 for callers with no scene concept (a single-scene call, or the
+    no-scenes-extracted correlation attempt) — equivalent to today's
+    behaviour for those calls.
     """
     ctx = await _build_context(
         call_id=call_id, units=units, vehicles=vehicles, cleared_units=cleared_units,
@@ -771,6 +788,7 @@ async def preview_correlation(
         tags=tags, incident_type=incident_type, location=location,
         reassignment=reassignment, create_if_new=create_if_new,
         embedding=embedding, severity=severity, transcript=transcript,
+        scene_index=scene_index,
     )
     decision = _run_decision(ctx)
     return {"decision": decision, "ctx": ctx}
@@ -806,6 +824,7 @@ async def _build_context(
     embedding: Optional[list] = None,
     severity: Optional[str] = None,
     transcript: Optional[str] = None,
+    scene_index: int = 0,
 ) -> dict:
     now    = reference_time or datetime.now(timezone.utc)
     window = timedelta(hours=settings.correlation_window_hours)
@@ -883,6 +902,9 @@ async def _build_context(
         "incident_type": incident_type, "location": location,
         "location_coords": location_coords, "reassignment": reassignment,
         "create_if_new": create_if_new,
+        # server-26#96 — which scene of the call this decision is for. Carried
+        # through so _apply_and_log can key the per-scene write correctly.
+        "scene_index": scene_index,
     }
 
 
@@ -1419,12 +1441,46 @@ def _run_decision(ctx: dict) -> dict:
 # ─────────────────────────────────────────────────────────────────────────────
 
 async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
-    """Commit a rules decision and persist the corr_debug fields to the call doc."""
+    """
+    Commit a rules decision and persist the corr_debug fields to the call doc.
+
+    server-26#96: every scene of a multi-scene call reaches this function
+    independently (upload.py's ``for scene_index, scene in enumerate(scenes):``
+    loop → _correlate_with_consensus → preview_correlation/apply_correlation),
+    and every scene writes to the SAME call doc. Writing corr_debug only at
+    the flat top level meant scene 2's write silently clobbered scene 1's
+    corr_path/corr_consensus/etc — the call doc ended up describing a splice
+    of decisions, not any one of them.
+
+    Fix: write the flat fields exactly as before (kept for any reader that
+    doesn't know about `scenes` yet — last-scene-wins, same as pre-#96
+    behaviour, a safe backward-compatible default) AND additionally nest the
+    same corr_debug — plus this scene's own transcript and the incident_id it
+    resolved to — under scenes.<scene_index>. Firestore's
+    `DocumentReference.set(data, merge=True)` recursively merges nested map
+    fields by key (confirmed against the documented set-with-merge semantics,
+    not assumed): a write of {"scenes": {"1": {...}}} merges into an existing
+    {"scenes": {"0": {...}}} to produce {"scenes": {"0": {...}, "1": {...}}}
+    rather than replacing the whole `scenes` map, so scene 0's and scene 1's
+    entries land side by side instead of colliding like the flat fields do.
+    `scene_index` defaults to 0 (see preview_correlation/correlate_call), so a
+    plain single-scene call still gets a `scenes` map — just with one entry,
+    equivalent to reading the flat fields today.
+    """
     incident_id = await _apply_decision(decision, ctx)
     corr_debug = decision.get("corr_debug") or {}
     if corr_debug:
+        scene_index = ctx.get("scene_index", 0)
+        updates = dict(corr_debug)
+        updates["scenes"] = {
+            str(scene_index): {
+                "transcript":   ctx.get("scene_transcript"),
+                "incident_id":  incident_id,
+                "corr_debug":   corr_debug,
+            }
+        }
         try:
-            await fstore.doc_set("calls", ctx["call_id"], corr_debug)
+            await fstore.doc_set("calls", ctx["call_id"], updates)
         except Exception as e:
             logger.warning(f"Could not write corr_debug for call {ctx['call_id']}: {e}")
     return incident_id

drb-c2-core/app/internal/summarizer.py

@@ -15,6 +15,39 @@ from app.internal import firestore as fstore
 from app.config import settings
 
 
+def _scene_sort_key(scene_index: str):
+    """Numeric-first sort so a >=10-scene call's entries still read in order."""
+    return (0, int(scene_index)) if scene_index.isdigit() else (1, scene_index)
+
+
+def _scene_text_for_incident(doc: dict, incident_id: str) -> Optional[str]:
+    """
+    The text of `doc` (a call doc) that actually belongs to `incident_id`.
+
+    server-26#96 records, per scene, which incident_id that scene's
+    correlation decision resolved to (incident_correlator._apply_and_log's
+    `scenes.<index>.incident_id`). Use that to pick only the scene(s) of this
+    call that are genuinely part of this incident, joining more than one if
+    several scenes happened to link into the same incident.
+
+    Falls back to transcript_corrected-or-transcript when the call doc has no
+    `scenes` field (predates server-26#96) or — defensively — when it has one
+    but nothing in it names this incident_id (should not happen for a call_id
+    that's actually in this incident's call_ids, but silently dropping a
+    call's contribution to its own summary would be a worse failure mode than
+    falling back to the whole-call text).
+    """
+    scenes = doc.get("scenes") or {}
+    matched = [
+        scene.get("transcript")
+        for _, scene in sorted(scenes.items(), key=lambda kv: _scene_sort_key(kv[0]))
+        if scene.get("incident_id") == incident_id and scene.get("transcript")
+    ]
+    if matched:
+        return "\n".join(matched)
+    return doc.get("transcript_corrected") or doc.get("transcript")
+
+
 async def summarizer_loop() -> None:
     from app.internal.feature_flags import get_flags
     interval = settings.summary_interval_minutes * 60
@@ -63,12 +96,30 @@ async def _summarize_incident(inc: dict) -> None:
     if not call_ids:
         return
 
-    # Fetch transcripts for all calls in this incident
+    # Fetch transcripts for all calls in this incident.
+    #
+    # server-26#114: a call links into an incident one SCENE at a time (see
+    # incident_correlator._apply_decision / server-26#96's `scenes` map on the
+    # call doc), and the same call_id can appear in more than one incident's
+    # call_ids — once per scene, each scene possibly landing in a different
+    # incident. Reading doc["transcript"] (the whole call, raw) meant an
+    # incident's summary was built partly on text from a DIFFERENT scene of
+    # that call that this incident has nothing to do with, and ignored
+    # transcript_corrected entirely.
+    #
+    # _scene_text_for_incident reads the specific scene(s) whose corr_debug
+    # recorded a link into THIS incident_id. For a call doc that predates
+    # this fix (no `scenes` field) it falls back to
+    # transcript_corrected-or-transcript — the one-liner half of #114, worth
+    # doing even for old-schema docs since it stops raw-transcript summaries.
     transcripts: list[str] = []
     for cid in call_ids:
         doc = await fstore.doc_get("calls", cid)
-        if doc and doc.get("transcript"):
-            transcripts.append(doc["transcript"])
+        if not doc:
+            continue
+        text = _scene_text_for_incident(doc, incident_id)
+        if text:
+            transcripts.append(text)
 
     if not transcripts:
         # No transcripts yet — clear stale flag and wait for next pass