logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

correlation: rules decide, one cheap LLM call only when ambiguous; drop the consensus tiebreak (server-26#299)

Browse Source 
The rules engine now commits every confident decision alone. _run_decision
marks a decision ambiguous (new_with_candidates, unfit_tg_candidate,
unverified_unit_link) and only then does upload._correlate make ONE
llm_correlator.decide call; a usable answer is committed, otherwise the rules
decision stands. Removed: _correlate_with_consensus, llm_correlator.tiebreak /
decisions_agree / tiebreak prompt, corr_smart_model, the correlation_smart
health tier, the LLM-orphan gate (_call_is_substanceless, effectively dead:
it could only fire on minor-severity untyped calls) and its sweep budget.
corr_consensus is now rules_only | llm | rules_fallback | rules_guard, with
corr_band naming why; admin debug and replay metrics read the new values.

Section fixes in the same pass:
- #115: no tiebreaker left to manufacture incidents; an LLM orphan of a
  rules "new" is honoured only for untyped, substanceless, routine/minor calls.
- #86: divergence veto bar moved to config (embedding_divergence_threshold);
  the band between it and the match bar is ambiguous, not a silent merge.
- #103: a unit link with no embedding is "unit_overlap_unverified" ->
  ambiguous, instead of skipping the veto and linking as verified.
- #189: unit-only and LLM links refused once an incident is open 90 min;
  unit links on incidents open >30 min are ambiguous; LLM links distance-
  guarded (incident_max_spread_km) and only to shown candidates.
- #190: severity steps down on an explicit stand-down phrase; new
  severity_peak keeps the monotonic max.
- #88: parse_llm_bool fails closed on "false"; scene closures honoured only
  on confident links or the incident the scene opened.
- #27: sweep budget is the scan window (wall clock), not 3 attempts.
- LLM prompt: "Reserve new" bullet no longer contradicts the #189 EXCEPT.
- updated_at never moves backward on a back-dated link.

Unmeasured: no live AI or replay (credits unfunded); band frequency, F1 and
thresholds are unverified. Must be re-scored on both answer keys in Replay
before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-27 20:26:38 -04:00
co-authored by Claude Opus 5.5
parent f4d78aba77
commit 3246fc97b0
18 changed files with 1207 additions and 1062 deletions
Download Patch File Download Diff File Expand all files Collapse all files
drb-c2-core/app/routers/upload.py
+175 -241
View File
@@ -112,99 +112,56 @@ async def upload_call_audio(
return {"url": gcs_uri}
# server-26#115 — the consensus LLM-orphan gate only fires when the call is
# genuinely substanceless. The earlier version tested `rules_decision["corr_debug"]`
# for a "positive signal", but corr_debug is EMPTY at preview time for
# action=="new" (corr_path:"new" is written at APPLY time), so that test was
# always False and the gate dropped real events — a major "extinguishing fire",
# geocoded calls, pursuit updates. The substance test now runs against `ctx`,
# which is fully populated at preview time.
# New-event signals: an LLM "orphan" may not drop a call carrying one when the
# rules would open an incident for it (server-26#299). A missed real event is
# worse than one extra incident, and the sweep is link-only, so a dropped event
# never comes back. These are the old #115 gate's vetoes, which existed because
# a looser gate dropped typed and geocoded fire calls.
_LLM_ORPHAN_VETO_SEVERITIES = ("moderate", "major")
def _recent_incident_on_same_talkgroup(ctx: dict) -> bool:
def _llm_orphan_vetoed(ctx: dict) -> bool:
from app.internal.incident_correlator import has_event_substance
return bool(
ctx.get("reassignment") or ctx.get("incident_type")
or ctx.get("call_severity") in _LLM_ORPHAN_VETO_SEVERITIES
or has_event_substance(ctx)
)
def _merge_llm_decision(rules: dict, llm: dict, ctx: dict) -> tuple[dict, str]:
"""
True when a recent incident is running on this call's own system +
talkgroup, within `settings.tg_dispatch_thin_idle_minutes` (5 min) —
applied uniformly regardless of the talkgroup's name (server-26#134).
Covers "unit dispatched, thin ack 10-30s later": the ack has no
substance of its own but plainly belongs to the job just opened.
Reads ctx["recent"] (the rules engine's own candidate list — no extra
Firestore read). That list is status=="active" incidents only, so an
already-resolved or capacity-capped same-talkgroup incident won't be
seen here even if chronologically recent (server-26#115, unresolved —
would need a dedicated non-status-filtered query).
Whether this limitation explains the 2/24 unexplained gate misses in the
window #3 measurement is UNANSWERED, not confirmed either way — a prior
pass here claimed a "confirmed explanation" for both that turned out to
be self-contradictory. Read `corr_gate_veto` (written to corr_debug on
every escalation of this exact disagreement shape — see the caller) in
the next measurement window instead of guessing from the raw dump again.
# TODO(server-26#115): add a talkgroup-scoped incident lookup (any
# status, no capacity filter) if a future measurement window pins a real
# gate miss on a resolved/capped same-talkgroup incident.
The decision to commit for an ambiguous scene, and the corr_consensus
value that says who made it: "llm", or "rules_guard" when the LLM's
orphan was refused for a call with a strong new-event signal.
"""
from app.internal.incident_correlator import _idle_gate_minutes
tg_id = ctx.get("talkgroup_id")
system_id = ctx.get("system_id")
if tg_id is None or not system_id:
return False
tg_str = str(tg_id)
now = ctx.get("now") or clock.now()
idle_limit = settings.tg_dispatch_thin_idle_minutes
for inc in ctx.get("recent") or []:
if system_id not in (inc.get("system_ids") or []):
continue
if tg_str not in (inc.get("talkgroup_ids") or []):
continue
if _idle_gate_minutes(inc, now) <= idle_limit:
return True
return False
if llm["action"] == "orphan" and rules["action"] == "new" and _llm_orphan_vetoed(ctx):
return rules, "rules_guard"
if (
llm["action"] == rules["action"] == "link"
and llm["matched_incident"]["incident_id"] == rules["matched_incident"]["incident_id"]
):
# The LLM confirmed the rules' own link: keep the rules' path/fit
# signal so corr_path histograms stay comparable across runs.
return rules, "llm"
final = {
"action": llm["action"],
"matched_incident": llm["matched_incident"],
"incident_type": llm["incident_type"],
"corr_debug": dict(llm.get("corr_debug") or {}),
}
if llm["action"] == "link":
# The rules' corr_path/fit signal described a different judgement
# (possibly a different incident); keep it under corr_rules_* only.
final["corr_debug"]["corr_path"] = "llm"
elif llm["action"] == "orphan" and rules["action"] == "link":
# Tombstone: the rules-only sweep would otherwise re-commit the very
# unverified unit link the LLM just refused.
final["corr_debug"]["corr_path"] = "llm_orphan"
return final, "llm"
def _call_is_substanceless(ctx: dict) -> tuple[bool, Optional[str]]:
"""
True when the call carries nothing that marks it as a real event:
• no resolved incident_type and not a reassignment, AND
• severity is not moderate/major, AND
• no vehicle, geocode or tag (incident_correlator.has_event_substance —
the same predicate the incident-creation gate uses), AND
• no recent incident already running on the same talkgroup.
Only then may the LLM-orphan gate drop the call without a tiebreak.
Returns (substanceless, veto_reason). veto_reason names whichever
condition kept the tiebreak alive ("type" | "reassignment" | "severity" |
"substance" | "recent_tg"), or None when the call is substanceless. The
caller writes this into corr_debug on the escalation path so a live
measurement window can see *why* each llm=orphan/rules=new call escaped
the gate instead of inferring it after the fact from the raw dump —
exactly the guesswork that produced a wrong "confirmed explanation" for
2 window-#3 misses on the first pass of this fix.
"""
from app.internal import incident_correlator
# The incident-creation gate skips the has_event_substance check entirely
# when a type resolved (incident_correlator._run_decision ~:1397), so a
# typed call — fire/medical/etc. — opens an incident on substance we do not
# re-check here. reassignment=True is dispatch pulling a unit onto a NEW
# job (units are blanked at :296 for exactly that reason): the strongest
# new-incident signal in the pipeline. Either one means "keep the tiebreak".
if ctx.get("incident_type"):
return False, "type"
if ctx.get("reassignment"):
return False, "reassignment"
if (ctx.get("call_severity") or "routine") in ("moderate", "major"):
return False, "severity"
if incident_correlator.has_event_substance(ctx):
return False, "substance"
if _recent_incident_on_same_talkgroup(ctx):
return False, "recent_tg"
return True, None
async def _correlate_with_consensus(
async def _correlate(
call_id: str,
node_id: str,
system_id: Optional[str],
@@ -222,19 +179,22 @@ async def _correlate_with_consensus(
severity: Optional[str] = None,
transcript: Optional[str] = None,
scene_index: int = 0,
) -> Optional[str]:
) -> tuple[Optional[str], dict]:
"""
Consensus correlator: runs the rules engine and the cheap LLM in sequence.
If they agree the rules decision is committed directly.
If they disagree a smarter tiebreaker LLM makes the final call.
Correlate one scene (server-26#299): the rules engine decides; only when
it marks its decision ambiguous is ONE cheap LLM call made, and a usable
LLM answer is committed instead. Returns (incident_id, committed decision).
Falls back to rules-only when GEMINI_API_KEY is absent, the call is
content-free (thin), or any LLM call fails.
corr_consensus on the call doc says who decided:
"rules_only" — confident rules decision, no LLM call
"llm" — ambiguous, the LLM's answer was committed
"rules_fallback" — ambiguous, LLM unavailable/failed/unusable → rules
"rules_guard" — ambiguous, LLM said orphan on a call with a strong
new-event signal → rules "new" kept
and corr_band names why it was ambiguous.
``scene_index`` (server-26#96) — which scene of the call this is, from the
caller's ``enumerate(scenes)`` loop. Threaded through so the call doc's
per-scene ``scenes`` map records this scene's own corr_debug/transcript
instead of colliding with every other scene's write on the flat fields.
``scene_index`` (server-26#96) — which scene of the call this is, so the
call doc's per-scene ``scenes`` map records this scene's own decision.
"""
from app.internal import incident_correlator, llm_correlator
@@ -247,68 +207,119 @@ async def _correlate_with_consensus(
embedding=embedding, severity=severity, transcript=transcript,
scene_index=scene_index,
)
ctx = preview["ctx"]
rules_decision = preview["decision"]
ctx = preview["ctx"]
rules = preview["decision"]
band = rules.get("ambiguous")
llm_decision = await llm_correlator.decide(call_id, ctx)
if not band:
rules["corr_debug"]["corr_consensus"] = "rules_only"
return await incident_correlator.apply_correlation(preview), rules
if llm_decision is None:
# LLM unavailable, skipped (thin call), or errored — rules wins.
rules_decision["corr_debug"]["corr_consensus"] = "rules_only"
return await incident_correlator.apply_correlation(preview)
if llm_correlator.decisions_agree(rules_decision, llm_decision):
rules_decision["corr_debug"]["corr_consensus"] = "agreed"
rules_decision["corr_debug"]["corr_llm_reasoning"] = llm_decision.get("reasoning", "")
return await incident_correlator.apply_correlation(preview)
# server-26#115 — LLM-orphan gate.
# When the cheap LLM says `orphan`, the rules engine says `new`, and the call
# is genuinely substanceless (routine severity, no vehicle/geocode/tag, and
# no incident already running on this talkgroup), resolve to `orphan` and DO
# NOT pay for the smart tiebreaker. A bare rules `new` there means only
# "nothing to link to" — trivially true for radio housekeeping (check-ins,
# roll call, 10-8/10-98) — and the tiebreaker rubber-stamped it ~21/21 of the
# time on exactly this disagreement (CORRELATION_REVIEW_0907b.md). Any real
# signal (severity, coords, tags, a live same-talkgroup incident) still
# escalates, so an event the LLM misreads as orphan is not lost.
is_orphan_vs_new = llm_decision["action"] == "orphan" and rules_decision["action"] == "new"
substanceless, gate_veto_reason = _call_is_substanceless(ctx) if is_orphan_vs_new else (False, None)
if is_orphan_vs_new and substanceless:
logger.info(
f"Consensus gate for call {call_id}: llm=orphan vs rules=new and call "
f"is substanceless — resolving orphan, skipping tiebreak"
)
gated = {
"action": "orphan",
"matched_incident": None,
"incident_type": None,
"corr_debug": dict(rules_decision.get("corr_debug") or {}),
}
gated["corr_debug"].update({
"corr_consensus": "llm_orphan_gate",
"corr_rules_action": rules_decision["action"],
"corr_llm_action": llm_decision["action"],
"corr_llm_reasoning": llm_decision.get("reasoning", ""),
})
return await incident_correlator.apply_correlation({"decision": gated, "ctx": ctx})
# Disagree — escalate to the smarter tiebreaker.
llm = await llm_correlator.decide(call_id, ctx)
if llm is None:
final, consensus = rules, "rules_fallback"
else:
final, consensus = _merge_llm_decision(rules, llm, ctx)
final["corr_debug"].update({
"corr_consensus": consensus,
"corr_band": band,
"corr_rules_action": rules["action"],
})
if llm is not None:
final["corr_debug"]["corr_llm_action"] = llm["action"]
final["corr_debug"]["corr_llm_reasoning"] = llm.get("reasoning", "")
logger.info(
f"Consensus disagreement for call {call_id}: "
f"rules={rules_decision['action']} vs llm={llm_decision['action']} — tiebreak"
f"Correlation for call {call_id} scene {scene_index}: ambiguous ({band}) — "
f"rules={rules['action']} llm={llm['action'] if llm else 'none'} → {consensus}"
)
final = await llm_correlator.tiebreak(rules_decision, llm_decision, ctx)
final["corr_debug"]["corr_consensus"] = "tiebreak"
final["corr_debug"]["corr_rules_action"] = rules_decision["action"]
final["corr_debug"]["corr_llm_action"] = llm_decision["action"]
if is_orphan_vs_new:
# server-26#115 — record *why* the llm=orphan/rules=new gate stood
# down instead of leaving a future measurement window to guess it
# from the raw dump (which produced a wrong "confirmed explanation"
# for 2/24 misses the first time around).
final["corr_debug"]["corr_gate_veto"] = gate_veto_reason
return await incident_correlator.apply_correlation({"decision": final, "ctx": ctx})
return await incident_correlator.apply_correlation({"decision": final, "ctx": ctx}), final
def _may_resolve(decision: dict) -> bool:
"""
server-26#88: honour a scene's `resolved` flag only on a link the
correlator was confident about, or on the incident this very scene opened.
A scene that attached by thin recency, or whose placement was ambiguous
(an LLM pick, or a rules fallback in the ambiguous band), has no business
closing an incident it may not belong to — that closed live incidents
while the event was still on the air.
"""
if decision.get("action") == "new":
return True
debug = decision.get("corr_debug") or {}
return (
decision.get("action") == "link"
and debug.get("corr_consensus") == "rules_only"
and debug.get("corr_path") != "fast/thin"
)
async def _correlate_scenes(
call_id: str,
node_id: str,
system_id: Optional[str],
talkgroup_id: Optional[int],
talkgroup_name: Optional[str],
scenes: list[dict],
) -> tuple[list[str], list[str]]:
"""
Correlate each scene of one call independently — a single recording can
carry several events on a busy channel — and apply each scene's LLM
closure. Returns (incident_ids, merged tags). Shared by the live pipeline,
replay (via _extract_and_correlate) and the transcript-PATCH path.
"""
from app.internal import incident_correlator
incident_ids: list[str] = []
all_tags: list[str] = []
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
# When dispatch is pulling a unit to a NEW call (reassignment), suppress
# unit overlap so the new scene doesn't chain into the unit's previous
# incident. parse_llm_bool (#88): a quoted "false" is not True.
is_reassignment = incident_correlator.parse_llm_bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id, decision = await _correlate(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if incident_id and incident_correlator.parse_llm_bool(scene.get("resolved")):
if not _may_resolve(decision):
logger.info(
f"Ignoring closure from call {call_id} scene {scene_index}: link into "
f"{incident_id} was not confident (server-26#88)"
)
continue
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
# One transmission read as "it's over" ("transport complete")
# closed the whole 09-22 bridge MVA at 14:44 and its next 56
# calls opened a second incident (server-26#170). Inferred from
# a single call, so provisional, like a timer close.
"reopenable": True,
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
return incident_ids, all_tags
async def _resolve_flags(system_id: Optional[str]):
@@ -335,7 +346,7 @@ async def _run_extraction_pipeline(
preserve_transcript_correction: bool = False,
) -> None:
"""Run steps 2-4 of the intelligence pipeline using an existing transcript."""
from app.internal import intelligence, incident_correlator, alerter
from app.internal import intelligence, alerter
flags, _flag = await _resolve_flags(system_id)
@@ -353,45 +364,9 @@ async def _run_extraction_pipeline(
)
# Step 3: Correlate each scene to an incident independently.
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
# When dispatch is pulling a unit to a NEW call (reassignment), suppress unit
# overlap so the new scene doesn't chain into the unit's previous incident.
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
"reopenable": True, # provisional, see _extract_and_correlate
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
incident_ids, all_tags = await _correlate_scenes(
call_id, node_id, system_id, talkgroup_id, talkgroup_name, scenes,
)
else:
scope = "globally" if not flags["correlation_enabled"] else f"system {system_id}"
logger.info(f"Correlation disabled ({scope}) — skipping scene extraction and correlation for call {call_id} (reprocess)")
@@ -430,7 +405,7 @@ async def _extract_and_correlate(
copy of it that can drift. Caller owns the correlation feature-flag check
and alerting.
"""
from app.internal import intelligence, incident_correlator
from app.internal import intelligence
# Step 2: Scene detection + intelligence extraction
if scenes is None:
@@ -443,50 +418,9 @@ async def _extract_and_correlate(
)
# Step 3: Correlate each scene independently.
# A single recording can produce multiple incidents on a busy channel.
incident_ids: list[str] = []
all_tags: list[str] = []
# server-26#96: scene_index is threaded through so each scene's
# corr_debug/transcript lands in its own entry of the call doc's
# `scenes` map instead of clobbering every other scene's write.
for scene_index, scene in enumerate(scenes):
all_tags.extend(scene["tags"])
is_reassignment = bool(scene.get("reassignment"))
corr_units = [] if is_reassignment else scene.get("units")
incident_id = await _correlate_with_consensus(
call_id=call_id,
node_id=node_id,
system_id=system_id,
talkgroup_id=talkgroup_id,
talkgroup_name=talkgroup_name,
tags=scene["tags"],
incident_type=scene["incident_type"],
location=scene["location"],
location_coords=scene["location_coords"],
units=corr_units,
vehicles=scene.get("vehicles"),
cleared_units=scene.get("cleared_units"),
reassignment=is_reassignment,
embedding=scene.get("embedding"),
severity=scene.get("severity"),
transcript=scene.get("transcript"),
scene_index=scene_index,
)
if incident_id and incident_id not in incident_ids:
incident_ids.append(incident_id)
if scene["resolved"] and incident_id:
await fstore.doc_set("incidents", incident_id, {
"status": "resolved",
"resolved_at": clock.now().isoformat(),
"resolved_via": "llm_closure",
# One transmission read as "it's over" ("transport complete")
# closed the whole 09-22 bridge MVA at 14:44 and its next 56
# calls opened a second incident (server-26#170). Inferred from
# a single call, so provisional, like a timer close.
"reopenable": True,
})
await incident_correlator.maybe_resolve_parent(incident_id)
logger.info(f"Auto-resolved incident {incident_id} (LLM closure detection)")
incident_ids, all_tags = await _correlate_scenes(
call_id, node_id, system_id, talkgroup_id, talkgroup_name, scenes,
)
# Correlator also runs for calls with no scenes (unclassified) to attempt
# talkgroup-based linking even when no transcript could be produced.
@@ -502,7 +436,7 @@ async def _extract_and_correlate(
_call_doc = await fstore.doc_get("calls", call_id)
skip_reason = (_call_doc or {}).get("skip_reason")
if not skip_reason or skip_reason == "transcript_too_short":
incident_id = await _correlate_with_consensus(
incident_id, _ = await _correlate(
call_id=call_id,
node_id=node_id,
system_id=system_id,