correlator: delete stale scenes on re-extraction instead of leaving them to rot (#96/#114)

Review of #132 found a blocker: PATCH /calls/{id}/transcript wipes tags/severity/location/units/embedding before re-extraction, but not the new scenes map, and doc_set(merge=True) can only add/overwrite nested map keys, never remove one. A call corrected from 3 scenes to 1 kept scenes.1/scenes.2 with pre-correction transcripts and incident_ids forever -- corrupting the exact per-scene tally #96 exists to make trustworthy, and able to re-feed stale text into #114's summarizer fix if a stale scene's incident_id still names a real incident.

Fix: fstore.doc_update(...,{"scenes": fstore.DELETE_FIELD}) -- a real delete, not a merge over an empty map. Added fstore.DELETE_FIELD (re-exports the real firebase_admin sentinel) and stubbed it in the sandboxed test conftest, which didn't have it. Also softened an overclaiming docstring: the Firestore nested-merge behavior is verified against the doc_set wrapper's pass-through, not against live Firestore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix

drb-c2-core/app/internal/firestore.py

@@ -7,6 +7,15 @@ from google.cloud.firestore_v1.base_query import FieldFilter
 from app.config import settings
 from app.internal.logger import logger
 
+# Re-exported so callers never need their own `firebase_admin.firestore` import
+# just to delete a field. server-26#96/#114 review: `doc_set(..., merge=True)`
+# merges nested maps by key but can never REMOVE one — writing `{"scenes": {}}`
+# to clear a map is a no-op, not a delete. Use `doc_update(coll, id, {"field":
+# fstore.DELETE_FIELD})` (or doc_set + merge, DELETE_FIELD works under both)
+# whenever a re-extraction/reprocess path needs a stale nested field gone
+# rather than merged over.
+DELETE_FIELD = fs.DELETE_FIELD
+
 # ---------------------------------------------------------------------------
 # In-memory TTL cache for rarely-changing documents (systems, nodes config)
 # ---------------------------------------------------------------------------

drb-c2-core/app/internal/incident_correlator.py

@@ -1458,8 +1458,13 @@ async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
     same corr_debug — plus this scene's own transcript and the incident_id it
     resolved to — under scenes.<scene_index>. Firestore's
     `DocumentReference.set(data, merge=True)` recursively merges nested map
-    fields by key (confirmed against the documented set-with-merge semantics,
-    not assumed): a write of {"scenes": {"1": {...}}} merges into an existing
+    fields by key. Verified against `internal/firestore.py`'s `doc_set`
+    wrapper (a straight `ref.set(data, merge=merge)` pass-through — no
+    `update()`, no read-modify-write, nothing that would change this) and
+    against the documented set-with-merge semantics; NOT exercised against a
+    live Firestore instance (no SDK available in the sandboxes this landed
+    from — review flagged this distinction explicitly). A write of
+    {"scenes": {"1": {...}}} merges into an existing
     {"scenes": {"0": {...}}} to produce {"scenes": {"0": {...}, "1": {...}}}
     rather than replacing the whole `scenes` map, so scene 0's and scene 1's
     entries land side by side instead of colliding like the flat fields do.

drb-c2-core/app/routers/calls.py

@@ -260,6 +260,15 @@ async def patch_transcript(
         "vehicles": [],
         "embedding": None,
     })
+    # server-26#96/#114 review: doc_set(merge=True) can only ADD/overwrite keys
+    # in a nested map, never remove one, so the fields above get cleared but a
+    # prior `scenes` map would survive re-extraction forever. A call corrected
+    # from 3 scenes down to 1 would keep scenes.1/scenes.2 with pre-correction
+    # transcripts and incident_ids -- corrupting the exact per-scene tally #96
+    # exists to make trustworthy, and re-feeding stale text into #114's
+    # summarizer fix if a stale scene's incident_id still names a real
+    # incident. Must be a real delete, not a merge over an empty map.
+    await fstore.doc_update("calls", call_id, {"scenes": fstore.DELETE_FIELD})
 
     # Unlink from ALL current incidents so re-correlation starts clean.
     # Handles both old single incident_id and new incident_ids list.