Every scene of a multi-scene call correlates independently in upload.py's
scene loop, but every scene's corr_debug was written flat onto the same
shared call doc — scene 2's write silently clobbered scene 1's
corr_path/corr_consensus/etc (#96), and summarizer.py read the whole call's
raw transcript per linked call, mixing text from scenes the incident had
nothing to do with, while ignoring transcript_corrected entirely (#114).
Fix: thread a scene_index from both `for scene in scenes:` loops in
upload.py down through _correlate_with_consensus ->
incident_correlator.preview_correlation/correlate_call -> _build_context ->
ctx["scene_index"]. incident_correlator._apply_and_log now writes, in the
same Firestore call:
- the existing flat corr_* fields, unchanged (last-scene-wins, the safe
backward-compatible default for any reader that doesn't know about
`scenes` yet)
- a new nested `scenes.<scene_index>` entry with {transcript, incident_id,
corr_debug}, via doc_set(..., merge=True). Firestore's
DocumentReference.set(data, merge=True) recursively merges nested map
fields by key (documented SDK behaviour, not assumed) — a write to
scenes.1 merges alongside an existing scenes.0 instead of replacing the
whole `scenes` map.
scene_index defaults to 0 for every caller with no scene concept (the
recorrelation sweep, the no-scenes-extracted orphan-check path), so a plain
single-scene call still gets a one-entry `scenes` map equivalent to reading
its flat fields today.
admin.py's _call_summary exposes the new `scenes` list per call (each entry
carrying the same corr_* field names as the flat fields, so the two shapes
are interchangeable to the tally); the summary tally now iterates each
call's scenes-if-present, else its own flat fields, so a 2-scene call with
two different corr_path values counts as two data points instead of one
blend. New `scene_decision_count` sits next to `linked_call_count` to make
that distinction visible.
summarizer.py's _scene_text_for_incident reads a linked call's `scenes` map
to find the scene(s) whose corr_debug recorded a link into the specific
incident being summarized, joining more than one if several scenes landed
in the same incident. Falls back to transcript_corrected-or-transcript for a
call doc with no `scenes` field (predates this change) — the one-liner half
of #114, worth doing regardless since it stops raw-transcript summaries even
for old-schema docs.
Does not touch #80/#95/#102's existing ctx-threading fixes
(embedding/severity/coords/LLM-prompt-transcript) — correct as-is, out of
scope here.
Tests: 14 new (test_per_scene_call_doc.py, test_summarizer_scene_transcript.py,
additions to test_admin_debug_correlation.py) covering the merge shape,
last-scene-wins flat-field backward compat, the admin tally's per-scene vs
per-call counting (including old-schema fallback), and the summarizer's
scene-specific text selection (including old-schema fallback). Full sandboxed
suite: 364 -> 378 passed, all green.
Fixes server-26#96, server-26#114
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix
config/ai_features was not the switch it was documented to be. Three paths
spent money with it off, and one path read it wrong, so per-system opt-outs
did not opt anything out.
- Correlation in the ingest pipeline tested the raw global flag instead of the
per-system resolution. With a system opted out, extraction was skipped but
the no-scenes fallback still correlated the call with empty tags, taking the
thin/recency path and attaching it to whatever incident was most recent on
that system. The opt-out did not disable correlation, it disabled good
correlation and left the worst kind running. (#75)
- Transcript correction ran on every transcribed call gated only by an env var,
spending Gemini tokens and a Places lookup per proposed location. An
"STT-only" window was never STT-only and its cost could not be attributed.
Now behind transcript_correction_enabled. (#76)
- _run_extraction_pipeline and the vocabulary learner, both reachable from
PATCH /calls/{id}/transcript, checked no flags at all. (#76, #81)
The flag resolver now lives in feature_flags.resolve_flags() rather than as a
local helper in upload.py. Three copies of that logic is how #75 happened.
PATCH /calls/{id}/transcript now refuses with 409 when correlation is off.
That route wipes tags, severity, location, units, embedding and unlinks the
call from every incident before queueing re-extraction. Gating extraction
alone would have made it destructive-only in the standing flags-off
configuration: the call left blank and orphaned forever, with the route still
answering 200. The wipe and the rebuild are one transaction in intent, so it
refuses before the first write.
Also: the summarizer's stale-incident sweep is no longer behind
summaries_enabled. It is pure Firestore with no model call in it, and gating
it meant nothing auto-resolved while AI was off - so every incident stayed
active forever and the candidate set every correlation reads kept growing.
transcript_correction_enabled is documented as NOT a pure cost lever. The
corrector is also the noise gate that sets not_speech; with it off, recogniser
noise reaches extraction as a real transcript, comes back thin, and
auto-attaches. Never open an evaluation window with correction off and
correlation on.
14 tests added covering flag precedence, both pipeline paths, the 409, the
correction gate and the summarizer no-op. Suite: 264 passed.
Refs #75, #76, #81, #45.
#17: severity was written once at _create_incident and never touched again,
so an incident that opened routine and escalated to a working fire stayed
routine forever. _update_incident now merges call_severity into the incident
via _max_severity() on every link.
Severity is monotonic: it only ever rises, never falls. An incident briefly
assessed "major" genuinely was major at that moment; a later, calmer-sounding
call is evidence the situation is winding down, not that the earlier read was
wrong. status/resolved_at exist to retire an incident — severity should stay
as the high-water mark so the worst-first rail, "Major only" filter, and map
colouring never bury a call that was genuinely major. See _max_severity's
docstring in incident_correlator.py for the full argument.
#18: none of the resolution sites wrote resolved_at, so an incident's
lifespan couldn't be reconstructed for the history-scrub feature. Added
resolved_at alongside status="resolved" at all six sites that flip it:
- incident_correlator.py _update_incident (signal-based: units all cleared)
- incident_correlator.py maybe_resolve_parent (master auto-resolve)
- summarizer.py _stale_sweep (90-minute auto-resolve)
- upload.py, both scene-resolution loops (single- and multi-scene)
- calls.py reprocess/correction path
(_update_incident's signal-resolve and maybe_resolve_parent's master-resolve
weren't named in the issue's four call sites, but they set status the same
way and were missing resolved_at too.)
No backfill: existing resolved incidents keep resolved_at = null, which
means "resolved before this field existed," not "never resolved." Backfilling
from updated_at would be a guess dressed up as data.
Tests: added to tests/test_correlator_gate.py, which needs no Firestore for
the pure _max_severity cases and patches fstore for the _update_incident/
maybe_resolve_parent writes. Covers the escalation case (routine -> major),
the no-downgrade case, and resolved_at on both the signal-resolve and
master-resolve paths. 52/52 passing in that file; 83 passed / 10
pre-existing failures for drb-c2-core overall (baseline was 69/10 — the
+14 is exactly the new tests, no regressions).
Fixes#17, #18.
Correlator
- Raise fast-path idle gate 30 → 90 min (tg_fast_path_idle_minutes)
- Fix disambiguate always-commits bug: run _call_fits_incident on winner
before committing; fall through to new-incident creation if it fails
- Add unit-continuity path (path 1.5): matches all_active by shared unit
IDs with a reassignment guard, bridges calls past the idle gate
- Add tag-based incident_type inference (_TAG_TYPE_HINTS) as GPT fallback,
rescuing tagged calls that would have been dropped (616 observed orphans)
- Add master/child incident model: _create_master_incident, _demote_to_child,
_add_child_to_master; new incidents stamped incident_type="master"
- Add cross-system parent detection (_find_cross_system_parent): two-signal
scoring (road overlap=0.4, embedding≥0.78=0.3, proximity=0.3, threshold=0.5)
wired into create-if-new path; creates master shell on first cross-system match
- Add maybe_resolve_parent: auto-resolves master when all children close;
called from upload pipeline (LLM closure) and summarizer stale sweep
- Add signal-based auto-resolve via units_active/units_cleared tracking:
GPT now extracts cleared_units per scene; _update_incident moves units
between active/cleared lists and resolves the incident when active empties;
stored on call doc for re-correlation sweep reuse
- Add _create_incident initialization of units_active/units_cleared fields
Re-correlation sweep
- Add corr_sweep_count + MAX_SWEEP_ATTEMPTS=3: orphans get 3 attempts
then are tombstoned as corr_path="unlinked", ending the re-sweep loop
(previously hammering each orphan 29-31 times per shift)
Intelligence extraction
- Add cleared_units to GPT prompt schema and rules
- Extract and propagate cleared_units per scene; merge across scenes;
store on call doc for re-correlation sweep
Token management
- Fix token release bug: remove release_token call on discord_connected=False
in MQTT checkin (transient Discord drops were orphaning bots mid-shift)
- Add PUT /tokens/{id}/prefer/{system_id} endpoint: lock a bot token to a
system; pass _none as system_id to clear; stored bidirectionally on both
token and system documents
- discord_join handler resolves preferred_token_id from system doc and passes
system_name in MQTT payload
incident_correlator.py — full rewrite: always runs on every call, fetches all active incidents cross-type, fast path collects all talkgroup matches and disambiguates by unit/vehicle overlap → location proximity → embedding, new location proximity path, slow path requires location corroboration, "Auto:" stripped from titles, "auto-generated" tag added, units/vehicles now accumulated on update
intelligence.py — resolved field in GPT schema, returned as 5th value
upload.py — both pipelines unpack 5-tuple, always call correlate, auto-resolve on resolved=True
summarizer.py — stale sweep runs each tick, resolves incidents idle for 90+ minutes
config.py — correlation_window_hours=2, embedding_similarity_threshold=0.93, location_proximity_km=0.5, incident_auto_resolve_minutes=90
Logan CusanoandClaude Sonnet 5