correlation: rules decide, one cheap LLM call only when ambiguous; drop the consensus tiebreak (server-26#299)
The rules engine now commits every confident decision alone. _run_decision marks a decision ambiguous (new_with_candidates, unfit_tg_candidate, unverified_unit_link) and only then does upload._correlate make ONE llm_correlator.decide call; a usable answer is committed, otherwise the rules decision stands. Removed: _correlate_with_consensus, llm_correlator.tiebreak / decisions_agree / tiebreak prompt, corr_smart_model, the correlation_smart health tier, the LLM-orphan gate (_call_is_substanceless, effectively dead: it could only fire on minor-severity untyped calls) and its sweep budget. corr_consensus is now rules_only | llm | rules_fallback | rules_guard, with corr_band naming why; admin debug and replay metrics read the new values. Section fixes in the same pass: - #115: no tiebreaker left to manufacture incidents; an LLM orphan of a rules "new" is honoured only for untyped, substanceless, routine/minor calls. - #86: divergence veto bar moved to config (embedding_divergence_threshold); the band between it and the match bar is ambiguous, not a silent merge. - #103: a unit link with no embedding is "unit_overlap_unverified" -> ambiguous, instead of skipping the veto and linking as verified. - #189: unit-only and LLM links refused once an incident is open 90 min; unit links on incidents open >30 min are ambiguous; LLM links distance- guarded (incident_max_spread_km) and only to shown candidates. - #190: severity steps down on an explicit stand-down phrase; new severity_peak keeps the monotonic max. - #88: parse_llm_bool fails closed on "false"; scene closures honoured only on confident links or the incident the scene opened. - #27: sweep budget is the scan window (wall clock), not 3 attempts. - LLM prompt: "Reserve new" bullet no longer contradicts the #189 EXCEPT. - updated_at never moves backward on a back-dated link. Unmeasured: no live AI or replay (credits unfunded); band frequency, F1 and thresholds are unverified. Must be re-scored on both answer keys in Replay before merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
f4d78aba77
commit
3246fc97b0
@@ -32,7 +32,7 @@ from typing import Optional
|
||||
from app.internal.logger import logger
|
||||
from app.config import settings
|
||||
|
||||
TIERS = ("transcription", "correlation_cheap", "correlation_smart", "extraction")
|
||||
TIERS = ("transcription", "correlation_cheap", "extraction")
|
||||
|
||||
# Consecutive failures a TRANSIENT condition must reach before it alerts.
|
||||
# Permanent conditions skip this entirely and alert on failure #1.
|
||||
|
||||
@@ -18,8 +18,8 @@ SHADOW MODE ONLY. As of this module's introduction, nothing skips scene
|
||||
extraction based on this verdict. `intelligence.extract_scenes` calls
|
||||
`classify_chatter` purely to record the verdict on the call doc
|
||||
(`chatter_classifier_verdict` / `chatter_classifier_reason`) so it becomes
|
||||
observable in the next `/admin` correlation-debug dump, exactly like
|
||||
`corr_gate_veto` (server-26#115 / PR #126). See the TODO at that call site for
|
||||
observable in the next `/admin` correlation-debug dump (as `corr_gate_veto`
|
||||
was, before server-26#299 removed that gate). See the TODO at that call site for
|
||||
what has to be true before this flips live.
|
||||
|
||||
Precision over recall, deliberately. A false positive here — flagging a REAL
|
||||
|
||||
@@ -43,6 +43,13 @@ Matching priority (in order):
|
||||
Calls with no incident_type skip new-incident creation but still run paths 1–3,
|
||||
so unclassified calls (short transport end, "en route", etc.) can link to an
|
||||
existing incident via talkgroup match.
|
||||
|
||||
Decision flow (server-26#299): the rules engine decides every confident case on
|
||||
its own. When it flags its decision as ambiguous (`_ambiguity`: opening an
|
||||
incident while live candidates exist, orphaning an unfit same-talkgroup
|
||||
follow-up, or linking on unverified shared units), routers/upload.py makes ONE
|
||||
cheap LLM call (llm_correlator.decide) and commits its answer instead. There is
|
||||
no second LLM tier and no consensus vote.
|
||||
"""
|
||||
import math
|
||||
import re
|
||||
@@ -65,26 +72,65 @@ _SEVERITY_RANK = {"routine": 0, "minor": 1, "moderate": 2, "major": 3}
|
||||
|
||||
def _max_severity(current: Optional[str], new: Optional[str]) -> str:
|
||||
"""
|
||||
Highest of two severities on the four-level ladder — the merge rule used
|
||||
every time a call attaches to an existing incident.
|
||||
|
||||
Severity is monotonic by design: it only ever rises, never falls, as more
|
||||
calls link. An incident briefly assessed "major" genuinely was major at
|
||||
that moment; a later call that sounds calmer ("units clear", dispatcher
|
||||
moving on) is evidence the SITUATION is winding down, not that the earlier
|
||||
read was wrong. That's what `status`/`resolved_at` are for — resolution
|
||||
retires an incident, it doesn't retroactively erase how serious it was.
|
||||
Every severity-driven surface (worst-first incident rail, "Major only"
|
||||
filter, map colour) exists to make sure a major event is never missed;
|
||||
downgrading severity mid-incident would silently defeat that on the exact
|
||||
incidents it exists to protect. A malformed/unrecognized value from either
|
||||
side ranks as "routine" so it can never suppress a real escalation.
|
||||
Highest of two severities on the four-level ladder. A malformed or
|
||||
unrecognized value from either side ranks as "routine" so it can never
|
||||
suppress a real escalation.
|
||||
"""
|
||||
current = current if current in _SEVERITY_RANK else "routine"
|
||||
new = new if new in _SEVERITY_RANK else "routine"
|
||||
return current if _SEVERITY_RANK[current] >= _SEVERITY_RANK[new] else new
|
||||
|
||||
|
||||
# server-26#190 — explicit stand-down / unfounded dispositions. Deliberately a
|
||||
# short list of phrases that only ever mean "there is less here than was
|
||||
# reported"; a calmer-SOUNDING follow-up ("Engine 5 on scene", "copy") is not
|
||||
# one, because extraction judges each transmission on its own words and a
|
||||
# working fire's routine status traffic would otherwise talk it down to
|
||||
# routine. Left out: "disregard"/"cancel" ("disregard my last" is a radio
|
||||
# correction), "all clear" ("primary search all clear" on a working fire) and
|
||||
# "code 4" (scene secure, not "nothing happened").
|
||||
_STAND_DOWN_RE = re.compile(
|
||||
r"\b(?:unfounded|false alarm|no (?:fire|flames|smoke showing)|nothing (?:showing|found|evident)"
|
||||
r"|gone on arrival|g\.?o\.?a\.?|stand(?:ing)? down|stood down"
|
||||
r"|fire(?:'s| is)? out)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _is_stand_down(transcript: Optional[str]) -> bool:
|
||||
return bool(transcript and _STAND_DOWN_RE.search(transcript))
|
||||
|
||||
|
||||
def _severity_after_link(
|
||||
inc: dict, call_severity: Optional[str], transcript: Optional[str],
|
||||
) -> dict:
|
||||
"""
|
||||
The incident's severity fields after one more call links (server-26#190).
|
||||
|
||||
`severity` is the CURRENT read and drives the live surfaces (worst-first
|
||||
rail, map colour, severity-scaled quiet timer). It rises immediately on
|
||||
any worse call, and falls only on an explicit stand-down disposition
|
||||
("they lit a fireplace... only smoke showing, no flames") — then to that
|
||||
call's own severity. It used to only ever ratchet up, which left a false
|
||||
Major on the wall for the full 90-minute Major quiet timer after units
|
||||
reported nothing there.
|
||||
|
||||
`severity_peak` keeps the old monotonic meaning (the worst this incident
|
||||
was ever judged), so history and any "was this ever major" question still
|
||||
have an answer that later chatter cannot erase.
|
||||
"""
|
||||
current = inc.get("severity") if inc.get("severity") in _SEVERITY_RANK else "routine"
|
||||
peak = _max_severity(inc.get("severity_peak"), current)
|
||||
new = call_severity if call_severity in _SEVERITY_RANK else "routine"
|
||||
out = {"severity_peak": _max_severity(peak, new)}
|
||||
if _SEVERITY_RANK[new] < _SEVERITY_RANK[current] and _is_stand_down(transcript):
|
||||
out["severity"] = new
|
||||
out["severity_deescalated_from"] = current
|
||||
else:
|
||||
out["severity"] = _max_severity(current, new)
|
||||
return out
|
||||
|
||||
|
||||
# Maximum plausible ground speed for a moving incident (pursuit/transport).
|
||||
# ~3 miles/min ≈ 180 mph — well above real pursuit speeds, but generous enough
|
||||
# to tolerate GPS drift and call-timing jitter. Anything faster is a bad geocode
|
||||
@@ -278,16 +324,88 @@ def has_event_substance(ctx: dict) -> bool:
|
||||
a vehicle, a geocode, or a tag.
|
||||
|
||||
This is the substance half of the incident-creation gate (see
|
||||
`_run_decision`, "Severity, not type, decides..."), factored out so the
|
||||
consensus LLM-orphan gate in routers/upload.py mirrors it exactly and can
|
||||
never drop a call the creation gate would have opened. `call_units` and
|
||||
`location` are deliberately excluded — radio protocol puts a unit ID and a
|
||||
place name in almost every transmission, so counting them as substance
|
||||
makes the check trivially true.
|
||||
`resolve_new_incident_type`). `call_units` and `location` are deliberately
|
||||
excluded — radio protocol puts a unit ID and a place name in almost every
|
||||
transmission, so counting them as substance makes the check trivially true.
|
||||
"""
|
||||
return bool(ctx.get("call_vehicles") or ctx.get("coords") or ctx.get("tags"))
|
||||
|
||||
|
||||
def resolve_new_incident_type(ctx: dict) -> Optional[str]:
|
||||
"""
|
||||
The type a NEW incident opened by this call gets, or None when the call is
|
||||
not incident-worthy and must orphan instead. The one creation gate, shared
|
||||
by the rules engine and the LLM tier (llm_correlator._parse_response), so
|
||||
an LLM "new" can neither open what the rules gate refuses nor be demoted
|
||||
to orphan for lacking a type the rules gate would have defaulted.
|
||||
|
||||
Severity, not type, decides whether a call is incident-worthy. Requiring a
|
||||
concrete incident_type meant a channel whose traffic never classifies —
|
||||
transit/rail administration, records lookups, prisoner transports — could
|
||||
never open a SECOND incident: every later call funnelled into whichever
|
||||
incident was created first (2026-08-16: TG 9048, 28 calls / 49 min)
|
||||
alongside 30+ permanent orphans. So anything the extractor judged a real
|
||||
event (minor+), or that carries substance (`has_event_substance`), opens
|
||||
an incident under the neutral "other" type. `call_units` and `location`
|
||||
are NOT substance — the first version of this gate counted them, turned
|
||||
"11-Victor, 72 at Holland Station" into its own incident, and left 37 of
|
||||
50 incidents open, one call each.
|
||||
"""
|
||||
resolved = ctx.get("incident_type") or _infer_type_from_tags(ctx.get("tags") or [])
|
||||
if resolved:
|
||||
return resolved
|
||||
if ctx.get("call_severity") in ("minor", "moderate", "major") or has_event_substance(ctx):
|
||||
return "other"
|
||||
return None
|
||||
|
||||
|
||||
def is_clearance_only(ctx: dict) -> bool:
|
||||
"""
|
||||
"45-9, I'm clear." carries one fact: which unit is done. Only the rules
|
||||
engine's unit match can say which incident that is; an LLM link here would
|
||||
apply the clear to whatever incident it picked, so such a call is never
|
||||
sent to the LLM tier.
|
||||
"""
|
||||
units = ctx.get("call_units") or []
|
||||
cleared = ctx.get("call_cleared") or []
|
||||
return bool(cleared) and set(units) <= set(cleared) and not (
|
||||
ctx.get("tags") or ctx.get("location") or ctx.get("call_vehicles") or ctx.get("incident_type")
|
||||
)
|
||||
|
||||
|
||||
_LLM_TRUE = frozenset({"true", "yes", "1"})
|
||||
_LLM_FALSE = frozenset({"false", "no", "0", "", "none", "null"})
|
||||
|
||||
|
||||
def parse_llm_bool(value) -> bool:
|
||||
"""
|
||||
A yes/no field from an LLM, failing CLOSED (server-26#88).
|
||||
|
||||
`bool(value)` reads the JSON *string* "false" as True, so a model that
|
||||
quoted its booleans would resolve whatever incident the scene linked into.
|
||||
Only a real bool, or an exact true-ish token, counts as True; anything
|
||||
else — "false", "maybe", a number, a dict — is False. For `resolved` and
|
||||
`reassignment` a missed True costs a stale open incident or one missed
|
||||
unit release; a false True closes a live incident or blanks a call's units.
|
||||
|
||||
Public because the extraction boundary (intelligence.py) is where it
|
||||
belongs; routers/upload.py applies it too, so a value that reaches the
|
||||
scene loop un-coerced is still validated there.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
token = value.strip().lower()
|
||||
if token in _LLM_TRUE:
|
||||
return True
|
||||
if token not in _LLM_FALSE:
|
||||
logger.warning(f"parse_llm_bool: unrecognised value {value!r} read as False")
|
||||
return False
|
||||
if value is not None:
|
||||
logger.warning(f"parse_llm_bool: non-boolean {type(value).__name__} {value!r} read as False")
|
||||
return False
|
||||
|
||||
|
||||
def _infer_type_from_tags(tags: list[str]) -> Optional[str]:
|
||||
"""Return an incident type inferred from tags, or None if ambiguous."""
|
||||
for tag in tags:
|
||||
@@ -483,7 +601,7 @@ def _resolve_incident_title(
|
||||
of strictly higher severity. Rationale in the commit message; in short, an
|
||||
incident's identity is the event that opened it, and the one situation
|
||||
where the header must change is the one where things got worse. This
|
||||
mirrors `_max_severity`: monotonic, never walked back by later chatter.
|
||||
mirrors `severity_peak`: monotonic, never walked back by later chatter.
|
||||
|
||||
Two non-renames are still allowed, because neither replaces an event name:
|
||||
• filling in a placeholder title on an incident that opened on a call
|
||||
@@ -656,6 +774,43 @@ def _incident_at_capacity(inc: dict, now: datetime) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
def _weak_link_capped(inc: dict, now: datetime) -> bool:
|
||||
"""server-26#189 — past this span an incident takes only strong evidence."""
|
||||
return _incident_span_minutes(inc, now) >= settings.incident_weak_link_max_span_minutes
|
||||
|
||||
|
||||
def llm_candidates(ctx: dict) -> list[dict]:
|
||||
"""
|
||||
The incidents the LLM tier may see and link into: the rules engine's
|
||||
`recent` pool minus anything at the hard caps or past the weak-link span
|
||||
cap (server-26#189). Every LLM link is weak evidence — a model's reading
|
||||
of two transcripts — so an incident that has been open long enough to
|
||||
have become a catch-all is not offered to it at all.
|
||||
"""
|
||||
now = ctx["now"]
|
||||
return [
|
||||
inc for inc in ctx.get("recent") or []
|
||||
if not _incident_at_capacity(inc, now) and not _weak_link_capped(inc, now)
|
||||
]
|
||||
|
||||
|
||||
def llm_link_allowed(inc: dict, ctx: dict) -> bool:
|
||||
"""
|
||||
Distance guard on an LLM-chosen link (server-26#189). When both the call
|
||||
and a non-pursuit incident are geocoded and farther apart than
|
||||
`incident_max_spread_km`, the model's link is refused: every rules path
|
||||
already has a location check of its own, and the LLM tier had none.
|
||||
"""
|
||||
coords = ctx.get("coords")
|
||||
pin = _verified_pin(inc)
|
||||
if not coords or not pin:
|
||||
return True
|
||||
if set(inc.get("tags") or []) & _PURSUIT_TAGS:
|
||||
return True
|
||||
dist = _haversine_km(coords["lat"], coords["lng"], pin["lat"], pin["lng"])
|
||||
return dist <= settings.incident_max_spread_km
|
||||
|
||||
|
||||
def _drop_capped(incidents: list[dict], now: datetime) -> list[dict]:
|
||||
"""Remove over-cap incidents from a candidate pool (see _incident_at_capacity)."""
|
||||
kept: list[dict] = []
|
||||
@@ -979,7 +1134,7 @@ def _run_decision(ctx: dict) -> dict:
|
||||
# Hard caps, enforced regardless of which path would have matched. Normally
|
||||
# a no-op because _build_context already filtered these out; re-applied here
|
||||
# so the guarantee holds for any caller that assembles a ctx directly
|
||||
# (tests, the LLM tiebreak path) rather than through _build_context.
|
||||
# (tests) rather than through _build_context.
|
||||
_capped_ids = {
|
||||
inc.get("incident_id")
|
||||
for inc in (list(ctx["all_active"]) + list(ctx["recent"]))
|
||||
@@ -1013,6 +1168,10 @@ def _run_decision(ctx: dict) -> dict:
|
||||
|
||||
matched_incident: Optional[dict] = None
|
||||
corr_debug: dict = {}
|
||||
# Why the fast path refused a same-talkgroup candidate, when it did. A
|
||||
# "no_signal" refusal is the one the rules cannot settle: a substantive
|
||||
# call on a channel with a live incident and no evidence either way.
|
||||
tg_reject: Optional[str] = None
|
||||
|
||||
# ── 1. Fast path: talkgroup match with recency gate ──────────────────────
|
||||
#
|
||||
@@ -1168,13 +1327,14 @@ def _run_decision(ctx: dict) -> dict:
|
||||
"corr_incident_idle_min": round(_incident_idle_minutes(candidate, now), 1),
|
||||
"corr_fit_signal": fit_signal,
|
||||
}
|
||||
if fit_signal == "unit_overlap" and call_units:
|
||||
if fit_signal.startswith("unit_") and call_units:
|
||||
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
|
||||
logger.info(
|
||||
f"Correlator fast-path: call {call_id} → {candidate['incident_id']} "
|
||||
f"(signal={fit_signal})"
|
||||
)
|
||||
else:
|
||||
tg_reject = fit_signal
|
||||
logger.info(
|
||||
f"Correlator fast-path skipped: call {call_id} — {fit_signal} "
|
||||
f"from {candidate['incident_id']}; will attempt new incident"
|
||||
@@ -1207,13 +1367,14 @@ def _run_decision(ctx: dict) -> dict:
|
||||
"corr_candidates": len(tg_recent),
|
||||
"corr_fit_signal": fit_signal,
|
||||
}
|
||||
if fit_signal == "unit_overlap" and call_units:
|
||||
if fit_signal.startswith("unit_") and call_units:
|
||||
corr_debug["corr_matched_units"] = _matching_units(call_units, candidate.get("units"))
|
||||
logger.info(
|
||||
f"Correlator fast-path (disambig {len(tg_recent)} candidates): "
|
||||
f"call {call_id} → {candidate['incident_id']} (signal={fit_signal})"
|
||||
)
|
||||
else:
|
||||
tg_reject = fit_signal
|
||||
logger.info(
|
||||
f"Correlator fast-path disambig: no candidate fits call {call_id} "
|
||||
f"across {len(tg_recent)} incidents ({fit_signal}) — will attempt new incident"
|
||||
@@ -1249,28 +1410,27 @@ def _run_decision(ctx: dict) -> dict:
|
||||
and inc.get("updated_at", "") > best_unit_inc.get("updated_at", "")
|
||||
for inc in all_active
|
||||
)
|
||||
# Location conflict check: same unit at a clearly different location
|
||||
# means the officer has moved to a new scene (e.g. dispatch sent
|
||||
# them to a second call without an explicit reassignment signal).
|
||||
location_conflict = False
|
||||
if coords and not reassigned_away:
|
||||
inc_coords_uc = best_unit_inc.get("location_coords")
|
||||
if inc_coords_uc:
|
||||
uc_dist = _haversine_km(
|
||||
coords["lat"], coords["lng"],
|
||||
inc_coords_uc["lat"], inc_coords_uc["lng"],
|
||||
)
|
||||
if uc_dist > settings.location_proximity_km:
|
||||
location_conflict = True
|
||||
logger.info(
|
||||
f"Correlator unit-continuity skipped: call {call_id} — "
|
||||
f"unit match but location conflict ({uc_dist:.2f}km) with "
|
||||
f"{best_unit_inc['incident_id']}"
|
||||
)
|
||||
if not reassigned_away and not location_conflict:
|
||||
# Same unit-overlap judgement as the fast path (_unit_fit): a
|
||||
# location conflict means the officer has moved to a new scene, a
|
||||
# sink past the weak-link span cap takes no unit-only links
|
||||
# (server-26#189), and an old incident with no location or content
|
||||
# agreement is linked but flagged unverified, i.e. ambiguous.
|
||||
uc_fit, uc_signal = (False, "reassigned_away")
|
||||
if not reassigned_away:
|
||||
uc_fit, uc_signal = _unit_fit(
|
||||
best_unit_inc, coords, call_embedding, now,
|
||||
settings.location_proximity_km,
|
||||
)
|
||||
if not uc_fit:
|
||||
logger.info(
|
||||
f"Correlator unit-continuity skipped: call {call_id} vs "
|
||||
f"{best_unit_inc['incident_id']} — {uc_signal}"
|
||||
)
|
||||
else:
|
||||
matched_incident = best_unit_inc
|
||||
corr_debug = {
|
||||
"corr_path": "unit-continuity",
|
||||
"corr_fit_signal": uc_signal,
|
||||
"corr_incident_idle_min": round(_incident_idle_minutes(best_unit_inc, now), 1),
|
||||
# Unlike this file's other two paths (fast/single, fast/disambig),
|
||||
# a match here is unit-driven by construction (call_unit_set &
|
||||
@@ -1445,63 +1605,66 @@ def _run_decision(ctx: dict) -> dict:
|
||||
|
||||
# ── Decision output ───────────────────────────────────────────────────────
|
||||
if matched_incident:
|
||||
return {
|
||||
"action": "link",
|
||||
"matched_incident": matched_incident,
|
||||
"incident_type": incident_type,
|
||||
"corr_debug": corr_debug,
|
||||
}
|
||||
|
||||
if not create_if_new:
|
||||
return {"action": "orphan", "matched_incident": None, "incident_type": None, "corr_debug": corr_debug}
|
||||
|
||||
# Attempt type inference from tags before giving up on creation
|
||||
resolved_type = incident_type
|
||||
if not resolved_type and tags:
|
||||
resolved_type = _infer_type_from_tags(tags)
|
||||
if resolved_type:
|
||||
action, resolved_type = "link", incident_type
|
||||
elif not create_if_new:
|
||||
action, resolved_type = "orphan", None
|
||||
else:
|
||||
resolved_type = resolve_new_incident_type(ctx)
|
||||
action = "new" if resolved_type else "orphan"
|
||||
if resolved_type and resolved_type != incident_type:
|
||||
logger.info(
|
||||
f"Correlator: inferred incident_type={resolved_type!r} from tags {tags} for call {call_id}"
|
||||
f"Correlator: call {call_id} has no incident_type — opening a "
|
||||
f"{resolved_type!r} incident (severity={call_severity}, tags={tags})"
|
||||
)
|
||||
|
||||
# Severity, not type, decides whether a call is incident-worthy.
|
||||
#
|
||||
# Requiring a concrete incident_type here meant a channel whose traffic never
|
||||
# classifies — transit/rail administration, records lookups, prisoner
|
||||
# transports — could never open a SECOND incident. Every later call on that
|
||||
# talkgroup then funnelled into whichever incident happened to be created
|
||||
# first, producing hour-long incidents made of unrelated transmissions
|
||||
# (2026-08-16: TG 9048, 28 calls / 49 min) alongside 30+ permanent orphans.
|
||||
#
|
||||
# Anything the extractor judged a real event, or that carries any concrete
|
||||
# content, now opens an incident under the neutral "other" type. Only
|
||||
# content-free routine traffic is still left for the thin path to attach.
|
||||
#
|
||||
# `call_units` and `location` are NOT substance. Radio protocol puts a unit
|
||||
# ID in essentially every transmission and a place name in most of them, so
|
||||
# including them made has_substance true almost always and the severity check
|
||||
# dead code — the first version of this gate turned "11-Victor, 72 at Holland
|
||||
# Station" into its own incident and left 37 of 50 incidents open, one call
|
||||
# each. A vehicle, a geocode, or a tag means the extractor found something
|
||||
# beyond who was speaking and where they stood.
|
||||
if not resolved_type:
|
||||
has_substance = has_event_substance(ctx)
|
||||
if call_severity in ("minor", "moderate", "major") or has_substance:
|
||||
resolved_type = "other"
|
||||
logger.info(
|
||||
f"Correlator: call {call_id} has no incident_type — opening an 'other' "
|
||||
f"incident (severity={call_severity}, substance={has_substance})"
|
||||
)
|
||||
|
||||
if not resolved_type:
|
||||
return {"action": "orphan", "matched_incident": None, "incident_type": None, "corr_debug": corr_debug}
|
||||
|
||||
return {
|
||||
"action": "new",
|
||||
"matched_incident": None,
|
||||
decision = {
|
||||
"action": action,
|
||||
"matched_incident": matched_incident,
|
||||
"incident_type": resolved_type,
|
||||
"corr_debug": corr_debug,
|
||||
}
|
||||
decision["ambiguous"] = _ambiguity(decision, tg_reject, ctx)
|
||||
return decision
|
||||
|
||||
|
||||
def _ambiguity(decision: dict, tg_reject: Optional[str], ctx: dict) -> Optional[str]:
|
||||
"""
|
||||
Why the rules decision is NOT confident enough to commit alone, or None
|
||||
when it is (server-26#299). Only an ambiguous decision is sent to the LLM
|
||||
tier, once; every other decision is the rules engine's alone.
|
||||
|
||||
Ambiguous:
|
||||
• "new_with_candidates" — the rules would OPEN an incident while live,
|
||||
LLM-eligible incidents exist. Opening is the expensive mistake: this
|
||||
system over-splits, and rules-only correlation opened ~1.8x the
|
||||
incidents per hour of LLM-assisted (#170, 2026-10-04 control window)
|
||||
— duplicates of a running job and typed radio housekeeping.
|
||||
• "unfit_tg_candidate" — the rules would ORPHAN a substantive call that
|
||||
has a live incident on its own talkgroup but no evidence for or
|
||||
against it (fit test "no_signal"): an unfit follow-up.
|
||||
• "unverified_unit_link" — the rules would LINK on shared units alone
|
||||
into an incident that is no longer fresh, with no location or content
|
||||
agreement (mechanism B; #103: the content veto has nothing to measure
|
||||
when the scene has no embedding). Between the divergence and match
|
||||
embedding bars (#86) a link lands here too.
|
||||
|
||||
Never ambiguous: thin calls (nothing to reason about), clearance-only
|
||||
calls (only the unit match can place a 10-8), link-only sweeps
|
||||
(create_if_new=False — rules-only by design), or when the LLM would have
|
||||
no candidates to choose from.
|
||||
"""
|
||||
if not ctx.get("create_if_new", True) or ctx.get("is_thin_call"):
|
||||
return None
|
||||
if is_clearance_only(ctx) or not llm_candidates(ctx):
|
||||
return None
|
||||
action = decision["action"]
|
||||
if action == "new":
|
||||
return "new_with_candidates"
|
||||
if action == "orphan" and tg_reject == "no_signal":
|
||||
return "unfit_tg_candidate"
|
||||
if action == "link" and (decision["corr_debug"] or {}).get("corr_fit_signal") == "unit_overlap_unverified":
|
||||
return "unverified_unit_link"
|
||||
return None
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
@@ -1514,7 +1677,7 @@ async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
|
||||
|
||||
server-26#96: every scene of a multi-scene call reaches this function
|
||||
independently (upload.py's ``for scene_index, scene in enumerate(scenes):``
|
||||
loop → _correlate_with_consensus → preview_correlation/apply_correlation),
|
||||
loop → _correlate → preview_correlation/apply_correlation),
|
||||
and every scene writes to the SAME call doc. Writing corr_debug only at
|
||||
the flat top level meant scene 2's write silently clobbered scene 1's
|
||||
corr_path/corr_consensus/etc — the call doc ended up describing a splice
|
||||
@@ -1553,18 +1716,14 @@ async def _apply_and_log(decision: dict, ctx: dict) -> Optional[str]:
|
||||
"incident_id": incident_id,
|
||||
"corr_debug": corr_debug,
|
||||
# server-26#139: this scene's OWN extracted incident_type/
|
||||
# severity, as read by _call_is_substanceless's ctx at
|
||||
# decision time — not the call doc's flat top-level field,
|
||||
# which is last-scene-wins (server-26#96) and was the reason
|
||||
# #138's "type" veto couldn't be told apart from cross-scene
|
||||
# contamination without re-guessing from a live dump.
|
||||
# severity, as read from ctx at decision time — not the call
|
||||
# doc's flat top-level field, which is last-scene-wins
|
||||
# (server-26#96).
|
||||
# NOTE: unlike incident_type, call_severity is already
|
||||
# coerced to "routine" when extraction emitted nothing
|
||||
# (ctx build: `severity or "routine"`) — a scene reading
|
||||
# "routine" here doesn't distinguish "extraction said
|
||||
# routine" from "extraction said nothing". Don't split a
|
||||
# severity veto the way #138 splits the type veto without
|
||||
# accounting for that.
|
||||
# routine" from "extraction said nothing".
|
||||
"incident_type": ctx.get("incident_type"),
|
||||
"severity": ctx.get("call_severity"),
|
||||
}
|
||||
@@ -1613,6 +1772,7 @@ async def _apply_decision(decision: dict, ctx: dict) -> Optional[str]:
|
||||
talkgroup_name=talkgroup_name, incident_type=incident_type,
|
||||
cleared_units=call_cleared, refresh_activity=not thin_link,
|
||||
call_severity=call_severity, call_srcaddr=call_srcaddr,
|
||||
call_transcript=ctx.get("scene_transcript"),
|
||||
)
|
||||
return matched_incident["incident_id"]
|
||||
|
||||
@@ -1798,6 +1958,65 @@ def _disambiguate(
|
||||
return best
|
||||
|
||||
|
||||
def _unit_fit(
|
||||
inc: dict,
|
||||
call_coords: Optional[dict],
|
||||
call_embedding: Optional[list],
|
||||
now: Optional[datetime],
|
||||
proximity_km: float,
|
||||
) -> tuple[bool, str]:
|
||||
"""
|
||||
Judge a link whose evidence is a shared unit. Same officer usually means
|
||||
same call — but on a shared dispatch channel a patrol unit works job after
|
||||
job, so shared units alone grow chains (mechanism B,
|
||||
CORRELATION_REVIEW_0820.md). In order:
|
||||
|
||||
• call and incident both geocoded: far apart → (False,
|
||||
"unit_loc_conflict"), the officer moved to a new scene; close →
|
||||
(True, "unit_location_match").
|
||||
• embeddings agree at the match bar → (True, "unit_content_match").
|
||||
• incident past the weak-link span cap (#189) → (False,
|
||||
"weak_link_span_cap"): a long-running incident stops taking
|
||||
unit-only evidence.
|
||||
• incident fresh (idle < unit_link_fresh_idle_minutes) and young (open
|
||||
< unit_link_confident_max_span_minutes) → (True, "unit_overlap"). No
|
||||
content veto here: an update on the same scene often shares little
|
||||
vocabulary with the dispatch and does not restate the address.
|
||||
• not fresh, embeddings below the DIVERGENCE bar (#86) → (False,
|
||||
"content_divergence").
|
||||
• anything else → (True, "unit_overlap_unverified"): the rules keep the
|
||||
link but cannot vouch for it, so the decision is ambiguous and goes
|
||||
to the LLM tier. This is where a scene with no embedding lands
|
||||
(#103) — it used to skip the veto and link as if verified, which
|
||||
removed the only brake on the unit loop for ~93% of calls.
|
||||
"""
|
||||
inc_coords = inc.get("location_coords")
|
||||
if call_coords and inc_coords:
|
||||
dist_km = _haversine_km(
|
||||
call_coords["lat"], call_coords["lng"],
|
||||
inc_coords["lat"], inc_coords["lng"],
|
||||
)
|
||||
return (False, "unit_loc_conflict") if dist_km > proximity_km else (True, "unit_location_match")
|
||||
|
||||
sim: Optional[float] = None
|
||||
inc_emb = inc.get("embedding")
|
||||
if call_embedding and inc_emb:
|
||||
sim = _cosine_similarity(call_embedding, inc_emb)
|
||||
if sim >= settings.embedding_similarity_threshold:
|
||||
return True, "unit_content_match"
|
||||
|
||||
if now is None:
|
||||
return True, "unit_overlap_unverified"
|
||||
if _weak_link_capped(inc, now):
|
||||
return False, "weak_link_span_cap"
|
||||
fresh = _idle_gate_minutes(inc, now) < settings.unit_link_fresh_idle_minutes
|
||||
if fresh and _incident_span_minutes(inc, now) < settings.unit_link_confident_max_span_minutes:
|
||||
return True, "unit_overlap"
|
||||
if not fresh and sim is not None and sim < settings.embedding_divergence_threshold:
|
||||
return False, "content_divergence"
|
||||
return True, "unit_overlap_unverified"
|
||||
|
||||
|
||||
def _call_fits_incident(
|
||||
inc: dict,
|
||||
call_units: list[str],
|
||||
@@ -1812,18 +2031,14 @@ def _call_fits_incident(
|
||||
Return (fits, signal) — fits is True when this call plausibly belongs to
|
||||
the incident; signal names the specific evidence that drove the decision.
|
||||
|
||||
fits=True signals: "unit_overlap" | "vehicle_overlap" | "location_proximity"
|
||||
fits=True signals: see `_unit_fit` for the unit_* family | "vehicle_overlap"
|
||||
| "location_proximity"
|
||||
fits=False signals: "unit_loc_conflict" | "content_divergence"
|
||||
| "location_conflict" | "no_signal"
|
||||
| "weak_link_span_cap" | "location_conflict" | "no_signal"
|
||||
|
||||
Evaluation order:
|
||||
1. Unit overlap. Same officer = same call. Also runs a location-conflict
|
||||
guard: geocoded on both sides and clearly different → the officer has
|
||||
moved to a new scene, false positive. No geocode on the call (or on
|
||||
the incident, with a call geocode) AND the incident is old (≥ 15 min)
|
||||
→ content divergence as a location proxy (embedding similarity < 0.82
|
||||
→ different scene). Skipped for recent incidents (< 15 min) — an
|
||||
update without re-stating the address is normal.
|
||||
1. Unit overlap — judged by `_unit_fit`, which also decides whether the
|
||||
link is confident or only "unit_overlap_unverified" (ambiguous).
|
||||
2. Vehicle overlap → True.
|
||||
3. Location proximity. Both geocoded and close → True; far apart with no
|
||||
other positive signal → False.
|
||||
@@ -1850,44 +2065,9 @@ def _call_fits_incident(
|
||||
inc_units = _unit_keys(inc.get("units"))
|
||||
matched_units = _matching_units(call_units, inc.get("units"))
|
||||
if matched_units:
|
||||
if call_coords:
|
||||
# Hard location conflict: geocoded on both sides and clearly different.
|
||||
inc_coords_u = inc.get("location_coords")
|
||||
if inc_coords_u:
|
||||
dist_km = _haversine_km(
|
||||
call_coords["lat"], call_coords["lng"],
|
||||
inc_coords_u["lat"], inc_coords_u["lng"],
|
||||
)
|
||||
if dist_km > proximity_km:
|
||||
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but location_conflict dist={dist_km:.2f}km → unit_loc_conflict")
|
||||
return False, "unit_loc_conflict"
|
||||
elif call_embedding and idle_min >= 15:
|
||||
# Call has geocode but incident doesn't — fall back to content
|
||||
# divergence as a location proxy. Without this, stale incidents
|
||||
# that never geocoded absorb unrelated calls purely on unit
|
||||
# overlap (e.g. a patrol officer working a second scene 70 min
|
||||
# after the original call).
|
||||
inc_emb_u = inc.get("embedding")
|
||||
if inc_emb_u:
|
||||
sim = _cosine_similarity(call_embedding, inc_emb_u)
|
||||
if sim < 0.82:
|
||||
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence (has_call_coords/no_inc_coords) sim={sim:.3f} → content_divergence")
|
||||
return False, "content_divergence"
|
||||
elif call_embedding and idle_min >= 15:
|
||||
# No geocode available AND old incident: use content divergence as a
|
||||
# location-proxy veto. After 15+ minutes an officer at a completely
|
||||
# different scene will be discussing clearly different content.
|
||||
# Skip this for recent incidents — an officer updating on the same
|
||||
# scene without re-stating the address is normal and their update
|
||||
# won't share much vocabulary with the original dispatch.
|
||||
inc_emb_u = inc.get("embedding")
|
||||
if inc_emb_u:
|
||||
sim = _cosine_similarity(call_embedding, inc_emb_u)
|
||||
if sim < 0.82:
|
||||
logger.info(f" fits[{inc_id}]: unit_overlap({matched_units}) but content_divergence sim={sim:.3f} → content_divergence")
|
||||
return False, "content_divergence"
|
||||
logger.info(f" fits[{inc_id}]: unit_overlap matched={matched_units} → unit_overlap")
|
||||
return True, "unit_overlap"
|
||||
fit, signal = _unit_fit(inc, call_coords, call_embedding, now, proximity_km)
|
||||
logger.info(f" fits[{inc_id}]: unit_overlap matched={matched_units} → {signal}")
|
||||
return fit, signal
|
||||
|
||||
# ── 2. Vehicle overlap ────────────────────────────────────────────────────
|
||||
inc_vehicles = set(inc.get("vehicles") or [])
|
||||
@@ -2047,6 +2227,7 @@ async def _update_incident(
|
||||
refresh_activity: bool = True,
|
||||
call_severity: Optional[str] = None,
|
||||
call_srcaddr: Optional[str] = None,
|
||||
call_transcript: Optional[str] = None,
|
||||
) -> None:
|
||||
incident_id = inc["incident_id"]
|
||||
|
||||
@@ -2109,7 +2290,9 @@ async def _update_incident(
|
||||
"units_cleared": units_cleared,
|
||||
"location_mentions": location_mentions,
|
||||
"summary_stale": True,
|
||||
"severity": _max_severity(inc.get("severity"), call_severity),
|
||||
# Current severity can step DOWN on an explicit stand-down; the peak
|
||||
# never does (server-26#190, _severity_after_link).
|
||||
**_severity_after_link(inc, call_severity, call_transcript),
|
||||
# Always all three, always together — writing one without the others is
|
||||
# what let the label and the pin drift apart (server-26#23).
|
||||
**location_fields,
|
||||
@@ -2123,7 +2306,18 @@ async def _update_incident(
|
||||
# acknowledging. The incident now ages from its last SUBSTANTIVE call, and
|
||||
# thin traffic rides along without extending its life.
|
||||
if refresh_activity:
|
||||
updates["updated_at"] = _floor_at_started_at(inc, now).isoformat()
|
||||
# Never move updated_at BACKWARD: a back-dated `now` (sweep, or a late
|
||||
# upload's clock pin, server-26#301) would make a live incident look
|
||||
# idle, failing on-time calls' recency gates and timer-closing it early.
|
||||
when = _floor_at_started_at(inc, now)
|
||||
try:
|
||||
prev = datetime.fromisoformat(str(inc.get("updated_at") or "").replace("Z", "+00:00"))
|
||||
if prev.tzinfo is None:
|
||||
prev = prev.replace(tzinfo=timezone.utc)
|
||||
when = max(when, prev)
|
||||
except ValueError:
|
||||
pass
|
||||
updates["updated_at"] = when.isoformat()
|
||||
if is_new_call: # a second scene of the same call is not a second call
|
||||
updates["substantive_call_count"] = (
|
||||
inc.get("substantive_call_count")
|
||||
@@ -2233,6 +2427,7 @@ async def _create_incident(
|
||||
"vehicles": call_vehicles,
|
||||
"srcaddrs": [call_srcaddr] if call_srcaddr else [],
|
||||
"severity": call_severity,
|
||||
"severity_peak": call_severity,
|
||||
"summary": None,
|
||||
"summary_stale": True,
|
||||
"summary_last_run": None,
|
||||
|
||||
@@ -1,23 +1,23 @@
|
||||
"""
|
||||
LLM-based incident correlator using Gemini.
|
||||
LLM incident correlator (Gemini, settings.corr_cheap_model).
|
||||
|
||||
Two functions are exposed:
|
||||
decide(call_id, ctx) — cheap first-pass (corr_cheap_model)
|
||||
tiebreak(rules_decision, llm_decision, ctx) — smart tiebreaker (corr_smart_model)
|
||||
One function is exposed:
|
||||
decide(call_id, ctx) — called by routers/upload.py ONLY when the rules
|
||||
engine marked its decision ambiguous
|
||||
(incident_correlator._ambiguity), at most once per
|
||||
scene. server-26#299 removed the second "smart"
|
||||
tiebreak tier and the cheap/smart consensus vote.
|
||||
|
||||
Both return a decision dict compatible with _run_decision() in incident_correlator:
|
||||
It returns a decision dict compatible with incident_correlator._apply_decision:
|
||||
{"action": "link"|"new"|"orphan",
|
||||
"matched_incident": dict|None,
|
||||
"incident_type": str|None,
|
||||
"corr_debug": dict,
|
||||
"reasoning": str} ← extra field for logging/tiebreak comparison
|
||||
"reasoning": str}
|
||||
|
||||
decide() is skipped for thin calls (no content to reason about) and when
|
||||
GEMINI_API_KEY is not set — in those cases returns None so the caller knows
|
||||
to fall back to the rules decision.
|
||||
|
||||
Error handling: any Gemini failure returns None from decide() and the
|
||||
rules_decision from tiebreak() so the pipeline never stalls.
|
||||
or None — no API key, nothing to choose from, an answer that fails
|
||||
validation, or any Gemini failure — and the caller then commits the rules
|
||||
decision unchanged, so the pipeline never stalls on the LLM.
|
||||
"""
|
||||
import asyncio
|
||||
from datetime import datetime, timezone
|
||||
@@ -155,8 +155,9 @@ call that reports a new event, below.):
|
||||
same one — especially once that incident has been open a long time (see
|
||||
open:/calls:).
|
||||
- Reserve "new" for a call that clearly describes a DIFFERENT event from every
|
||||
recent incident — a different place, different units, and a different subject,
|
||||
not merely a different transmission about the same job.
|
||||
recent incident — a different place or subject, not merely a different
|
||||
transmission about the same job. Shared units alone do not make it the same
|
||||
event (see EXCEPT above).
|
||||
- "orphan" a call that is not an incident at all: radio checks, roll call,
|
||||
a unit marking on/off duty or 10-8/10-98, mileage/log entries, a bare
|
||||
acknowledgement. Do not open a "new" incident for these.
|
||||
@@ -166,47 +167,15 @@ call that reports a new event, below.):
|
||||
"""
|
||||
|
||||
|
||||
def _build_decide_prompt(ctx: dict) -> str:
|
||||
now = ctx["now"]
|
||||
recent = ctx["recent"]
|
||||
inc_block = (
|
||||
"\n".join(_inc_summary(inc, now) for inc in _prompt_incidents(recent))
|
||||
if recent else "(none)"
|
||||
)
|
||||
def _build_decide_prompt(ctx: dict, candidates: list[dict]) -> str:
|
||||
now = ctx["now"]
|
||||
inc_block = "\n".join(_inc_summary(inc, now) for inc in _prompt_incidents(candidates))
|
||||
return (
|
||||
"You are an incident correlator for a public safety radio monitoring system.\n\n"
|
||||
"A new radio call has arrived. Decide whether it belongs to an existing active incident, "
|
||||
"represents a new incident, or should be orphaned (not enough information).\n\n"
|
||||
"represents a new incident, or should be orphaned (not an incident at all).\n\n"
|
||||
f"NEW CALL:\n{_call_block(ctx)}\n\n"
|
||||
f"ACTIVE INCIDENTS ({len(recent)} recent):\n{inc_block}\n"
|
||||
f"{_RULES}\n"
|
||||
f"Respond with JSON only (no markdown):\n{_SCHEMA}"
|
||||
)
|
||||
|
||||
|
||||
def _build_tiebreak_prompt(rules_decision: dict, llm_decision: dict, ctx: dict) -> str:
|
||||
now = ctx["now"]
|
||||
recent = ctx["recent"]
|
||||
inc_block = (
|
||||
"\n".join(_inc_summary(inc, now) for inc in _prompt_incidents(recent))
|
||||
if recent else "(none)"
|
||||
)
|
||||
|
||||
def _fmt(d: dict, name: str) -> str:
|
||||
action = d.get("action", "?")
|
||||
inc = d.get("matched_incident")
|
||||
inc_id = inc["incident_id"] if inc else (d.get("incident_id") or "null")
|
||||
reason = d.get("reasoning") or (d.get("corr_debug") or {}).get("corr_fit_signal") or "—"
|
||||
return f" {name}: action={action}, incident_id={inc_id}, reasoning={reason!r}"
|
||||
|
||||
return (
|
||||
"You are a senior incident correlator for a public safety radio monitoring system.\n\n"
|
||||
"Two correlation engines disagree. You must make the final decision.\n\n"
|
||||
f"NEW CALL:\n{_call_block(ctx)}\n\n"
|
||||
f"ACTIVE INCIDENTS ({len(recent)} recent):\n{inc_block}\n\n"
|
||||
"DISAGREEMENT:\n"
|
||||
f"{_fmt(rules_decision, 'Rules engine')}\n"
|
||||
f"{_fmt(llm_decision, 'LLM correlator')}\n"
|
||||
f"ACTIVE INCIDENTS ({len(candidates)} recent):\n{inc_block}\n"
|
||||
f"{_RULES}\n"
|
||||
f"Respond with JSON only (no markdown):\n{_SCHEMA}"
|
||||
)
|
||||
@@ -225,44 +194,48 @@ def _sync_gemini(model_name: str, prompt: str) -> dict:
|
||||
# Decision parsing
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _parse_response(raw: dict, ctx: dict) -> dict:
|
||||
def _parse_response(raw: dict, ctx: dict, candidates: list[dict]) -> Optional[dict]:
|
||||
"""
|
||||
Convert raw Gemini JSON output to a decision dict compatible with _run_decision().
|
||||
Resolves incident_id → full incident doc from ctx["all_active"].
|
||||
Handles type inference for "new" actions the same way as the rules engine.
|
||||
"""
|
||||
from app.internal.incident_correlator import _infer_type_from_tags # same-package import
|
||||
Convert raw Gemini JSON to a decision dict, or None when the answer is
|
||||
unusable and the rules decision should stand.
|
||||
|
||||
action = raw.get("action", "orphan")
|
||||
A "link" must name one of the candidates the model was shown — not any
|
||||
active incident: the shown list is the one with the #189 weak-link cap
|
||||
applied — and pass the distance guard (incident_correlator.
|
||||
llm_link_allowed). A "new" takes its type from the same creation gate the
|
||||
rules use, so it is refused (orphan) exactly when the rules would refuse
|
||||
it. An unknown action is unusable, not an orphan: the old "default to
|
||||
orphan" silently dropped calls on a malformed reply.
|
||||
"""
|
||||
from app.internal.incident_correlator import llm_link_allowed, resolve_new_incident_type
|
||||
|
||||
action = raw.get("action")
|
||||
reasoning = raw.get("reasoning", "")
|
||||
|
||||
if action not in ("link", "new", "orphan"):
|
||||
action = "orphan"
|
||||
logger.warning(f"LLM correlator: unusable action {action!r} — keeping the rules decision")
|
||||
return None
|
||||
|
||||
matched_incident: Optional[dict] = None
|
||||
|
||||
incident_type: Optional[str] = None
|
||||
if action == "link":
|
||||
inc_id = raw.get("incident_id")
|
||||
if inc_id:
|
||||
matched_incident = next(
|
||||
(i for i in ctx["all_active"] if i.get("incident_id") == inc_id),
|
||||
None,
|
||||
)
|
||||
matched_incident = next((i for i in candidates if i.get("incident_id") == inc_id), None)
|
||||
if not matched_incident:
|
||||
logger.warning(
|
||||
f"LLM correlator: incident_id={inc_id!r} not in active incidents — orphaning"
|
||||
f"LLM correlator: incident_id={inc_id!r} is not a shown candidate — keeping the rules decision"
|
||||
)
|
||||
action = "orphan"
|
||||
|
||||
incident_type: Optional[str] = None
|
||||
if action in ("link", "new"):
|
||||
return None
|
||||
if not llm_link_allowed(matched_incident, ctx):
|
||||
logger.info(
|
||||
f"LLM correlator: link to {inc_id} refused — call is geocoded beyond "
|
||||
f"incident_max_spread_km of its pin (server-26#189)"
|
||||
)
|
||||
return None
|
||||
incident_type = ctx["incident_type"]
|
||||
elif action == "new":
|
||||
incident_type = resolve_new_incident_type(ctx)
|
||||
if not incident_type:
|
||||
incident_type = _infer_type_from_tags(ctx["tags"])
|
||||
if action == "new" and not incident_type:
|
||||
# Can't create an incident without a type — demote to orphan
|
||||
action = "orphan"
|
||||
matched_incident = None
|
||||
|
||||
return {
|
||||
"action": action,
|
||||
@@ -279,54 +252,34 @@ def _parse_response(raw: dict, ctx: dict) -> dict:
|
||||
|
||||
async def decide(call_id: str, ctx: dict) -> Optional[dict]:
|
||||
"""
|
||||
Run the cheap LLM correlator (corr_cheap_model) on the call.
|
||||
|
||||
Returns a decision dict or None if:
|
||||
- GEMINI_API_KEY is not configured
|
||||
- the call is thin (content-free — no value from LLM)
|
||||
- there are no recent active incidents to reason about
|
||||
- Gemini fails
|
||||
|
||||
Callers should treat None as "fall back to rules decision".
|
||||
One cheap-model correlation decision for an ambiguous scene. Returns None
|
||||
(keep the rules decision) when GEMINI_API_KEY is unset, there is no
|
||||
candidate to show, the answer fails validation, or Gemini fails.
|
||||
"""
|
||||
from app.internal.incident_correlator import llm_candidates
|
||||
|
||||
if not settings.gemini_api_key:
|
||||
return None
|
||||
|
||||
if ctx["is_thin_call"]:
|
||||
return None # thin calls have no transcript/units/coords to reason about
|
||||
|
||||
if _is_clearance_only(ctx):
|
||||
# "45-9, I'm clear." carries one fact: which unit is done. Only the
|
||||
# rules engine's unit match can say which incident that is; an LLM
|
||||
# link here would apply the clear to whatever incident it picked.
|
||||
candidates = llm_candidates(ctx)
|
||||
if not candidates:
|
||||
return None
|
||||
|
||||
if not ctx["recent"]:
|
||||
return None # no incidents to correlate against — rules handles new-only
|
||||
|
||||
try:
|
||||
prompt = _build_decide_prompt(ctx)
|
||||
prompt = _build_decide_prompt(ctx, candidates)
|
||||
raw = await asyncio.to_thread(_sync_gemini, settings.corr_cheap_model, prompt)
|
||||
decision = _parse_response(raw, ctx)
|
||||
except Exception as e:
|
||||
await _log_llm_failure("LLM correlator", "correlation_cheap", call_id, settings.corr_cheap_model, e)
|
||||
return None
|
||||
await ai_health.report_healthy("correlation_cheap")
|
||||
decision = _parse_response(raw if isinstance(raw, dict) else {}, ctx, candidates)
|
||||
if decision:
|
||||
_id = (decision["matched_incident"] or {}).get("incident_id", "null")
|
||||
logger.info(
|
||||
f"LLM correlator ({settings.corr_cheap_model}): call {call_id} → "
|
||||
f"action={decision['action']} incident={_id} "
|
||||
f"reasoning={decision['reasoning']!r}"
|
||||
)
|
||||
await ai_health.report_healthy("correlation_cheap")
|
||||
return decision
|
||||
except Exception as e:
|
||||
await _log_llm_failure("LLM correlator", "correlation_cheap", call_id, settings.corr_cheap_model, e)
|
||||
return None
|
||||
|
||||
|
||||
def _is_clearance_only(ctx: dict) -> bool:
|
||||
units = ctx.get("call_units") or []
|
||||
cleared = ctx.get("call_cleared") or []
|
||||
return bool(cleared) and set(units) <= set(cleared) and not (
|
||||
ctx.get("tags") or ctx.get("location") or ctx.get("call_vehicles") or ctx.get("incident_type")
|
||||
)
|
||||
return decision
|
||||
|
||||
|
||||
_dead_models: set[str] = set()
|
||||
@@ -350,7 +303,7 @@ async def _log_llm_failure(where: str, tier: str, call_id: str, model: str, exc:
|
||||
|
||||
if kind == "dead_model":
|
||||
await _log_tier_down(where, tier, model, "model is unavailable",
|
||||
"Update CORR_CHEAP_MODEL/CORR_SMART_MODEL in config.py", text)
|
||||
"Update CORR_CHEAP_MODEL in config.py", text)
|
||||
return
|
||||
|
||||
# A depleted balance reads as 429, the same status as an ordinary rate limit,
|
||||
@@ -379,37 +332,3 @@ async def _log_tier_down(where: str, tier: str, model: str, problem: str, fix: s
|
||||
f"{where}: {problem} ({model!r}) -- the LLM correlation tier is DISABLED "
|
||||
f"and every call is falling back to rules-only. {fix}. API said: {text}"
|
||||
)
|
||||
|
||||
|
||||
async def tiebreak(rules_decision: dict, llm_decision: dict, ctx: dict) -> dict:
|
||||
"""
|
||||
Run the smart tiebreaker (corr_smart_model) when rules and LLM disagree.
|
||||
Falls back to rules_decision on any error.
|
||||
"""
|
||||
call_id = ctx["call_id"]
|
||||
try:
|
||||
prompt = _build_tiebreak_prompt(rules_decision, llm_decision, ctx)
|
||||
raw = await asyncio.to_thread(_sync_gemini, settings.corr_smart_model, prompt)
|
||||
decision = _parse_response(raw, ctx)
|
||||
_id = (decision["matched_incident"] or {}).get("incident_id", "null")
|
||||
logger.info(
|
||||
f"LLM tiebreak ({settings.corr_smart_model}): call {call_id} → "
|
||||
f"action={decision['action']} incident={_id} "
|
||||
f"reasoning={decision['reasoning']!r}"
|
||||
)
|
||||
await ai_health.report_healthy("correlation_smart")
|
||||
return decision
|
||||
except Exception as e:
|
||||
await _log_llm_failure("LLM tiebreak", "correlation_smart", call_id, settings.corr_smart_model, e)
|
||||
return rules_decision
|
||||
|
||||
|
||||
def decisions_agree(rules: dict, llm: dict) -> bool:
|
||||
"""True if both decisions agree on action and (when action=="link") on the target incident."""
|
||||
if rules["action"] != llm["action"]:
|
||||
return False
|
||||
if rules["action"] == "link":
|
||||
r_id = (rules.get("matched_incident") or {}).get("incident_id")
|
||||
l_id = (llm.get("matched_incident") or {}).get("incident_id")
|
||||
return r_id == l_id
|
||||
return True
|
||||
|
||||
@@ -72,7 +72,7 @@ def pipeline_worst_case_seconds(n_scenes: int = _BOUND_SCENES) -> float:
|
||||
correction = settings.gemini_timeout_seconds
|
||||
extraction = openai_attempts * settings.openai_timeout_seconds
|
||||
embeddings = n_scenes * openai_attempts * settings.openai_timeout_seconds
|
||||
correlation = 2 * settings.gemini_timeout_seconds # decide + tiebreak, scene 0
|
||||
correlation = settings.gemini_timeout_seconds # at most one LLM call, scene 0 (#299)
|
||||
return stt + correction + extraction + embeddings + correlation + _NON_AI_SLACK_SECONDS
|
||||
|
||||
|
||||
@@ -83,21 +83,25 @@ def _derive_min_minutes() -> int:
|
||||
# 21 with the defaults (stt 120s, openai 60s x2 attempts, gemini 60s, 3 scenes).
|
||||
MIN_MINUTES_SINCE_PIPELINE_START = _derive_min_minutes()
|
||||
|
||||
# Standard link-only retry budget before a call is tombstoned corr_path="unlinked".
|
||||
MAX_SWEEP_ATTEMPTS = 3
|
||||
# server-26#115 — a call the consensus LLM-orphan gate parked (llm=orphan vs
|
||||
# rules=new, no substance) gets a longer budget. The gate fires before any
|
||||
# incident for the job may exist, so the substantive call that would justify
|
||||
# linking can land well after the standard ~6 min. Still link-only: a genuinely
|
||||
# thin call must not mint an incident, and the rules creation gate would re-orphan
|
||||
# it anyway.
|
||||
GATED_ORPHAN_SWEEP_ATTEMPTS = 10
|
||||
|
||||
def _last_chance(call: dict, now: datetime) -> bool:
|
||||
"""
|
||||
server-26#27 — the retry budget is wall-clock, not an attempt count.
|
||||
|
||||
def _max_sweep_attempts(call: dict) -> int:
|
||||
if call.get("corr_consensus") == "llm_orphan_gate":
|
||||
return GATED_ORPHAN_SWEEP_ATTEMPTS
|
||||
return MAX_SWEEP_ATTEMPTS
|
||||
The old budget was 3 attempts, which with the #131 hold-back expired ~19
|
||||
minutes after the pipeline started, while the incident the orphan belongs
|
||||
to may be opened later than that (`f321b047` was tombstoned three minutes
|
||||
after its own incident was created). An orphan is now retried on every
|
||||
pass for as long as the scan query can see it, and tombstoned
|
||||
corr_path="unlinked" only on its LAST visible pass: the one after which
|
||||
its ended_at falls out of `recorrelation_scan_minutes`. A call with no
|
||||
parseable ended_at gets no further retries.
|
||||
"""
|
||||
ended = _parse_dt(call.get("ended_at"))
|
||||
if ended is None:
|
||||
return True
|
||||
remaining = timedelta(minutes=settings.recorrelation_scan_minutes) - (now - ended)
|
||||
return remaining <= timedelta(minutes=settings.summary_interval_minutes)
|
||||
|
||||
|
||||
async def recorrelation_loop() -> None:
|
||||
@@ -149,9 +153,8 @@ async def _run_sweep_pass() -> None:
|
||||
("status", "==", "ended"),
|
||||
("ended_at", ">=", cutoff),
|
||||
])
|
||||
# corr_path="unlinked" is written after the attempt budget is exhausted.
|
||||
# Allows a few retries so a welfare-check call can link to an escalation
|
||||
# incident that is created a few minutes later, without sweeping 30× forever.
|
||||
# corr_path="unlinked" is the tombstone written on an orphan's last
|
||||
# visible pass (_last_chance); the scan window itself is the budget.
|
||||
orphans = [
|
||||
c for c in recent_ended
|
||||
if not c.get("incident_ids") and not c.get("incident_id")
|
||||
@@ -163,7 +166,6 @@ async def _run_sweep_pass() -> None:
|
||||
# the thin path minutes later and attached to whatever was most recent —
|
||||
# a second route into the over-merge the thin fix above addresses.
|
||||
and not c.get("skip_reason")
|
||||
and c.get("corr_sweep_count", 0) < _max_sweep_attempts(c)
|
||||
and not _pipeline_likely_still_running(c, now)
|
||||
]
|
||||
|
||||
@@ -173,14 +175,14 @@ async def _run_sweep_pass() -> None:
|
||||
logger.info(f"Re-correlation sweep: {len(orphans)} orphaned call(s) to check")
|
||||
linked = 0
|
||||
for call in orphans:
|
||||
if await _recorrelate_orphan(call):
|
||||
if await _recorrelate_orphan(call, now):
|
||||
linked += 1
|
||||
|
||||
if linked:
|
||||
logger.info(f"Re-correlation sweep: linked {linked}/{len(orphans)} orphaned call(s)")
|
||||
|
||||
|
||||
async def _recorrelate_orphan(call: dict) -> bool:
|
||||
async def _recorrelate_orphan(call: dict, now: datetime) -> bool:
|
||||
"""
|
||||
Attempt to link a single orphaned call to an existing incident.
|
||||
Returns True if a match was found and the call was linked.
|
||||
@@ -223,12 +225,10 @@ async def _recorrelate_orphan(call: dict) -> bool:
|
||||
)
|
||||
return True
|
||||
|
||||
# Increment the attempt counter. Once the budget is reached the orphan filter
|
||||
# above will stop picking this call up, and we write corr_path="unlinked" as
|
||||
# a permanent tombstone.
|
||||
attempts = call.get("corr_sweep_count", 0) + 1
|
||||
update: dict = {"corr_sweep_count": attempts}
|
||||
if attempts >= _max_sweep_attempts(call):
|
||||
# The attempt count is kept for the admin debug view only; the budget is
|
||||
# the scan window (_last_chance), after which the call is tombstoned.
|
||||
update: dict = {"corr_sweep_count": call.get("corr_sweep_count", 0) + 1}
|
||||
if _last_chance(call, now):
|
||||
update["corr_path"] = "unlinked"
|
||||
await fstore.doc_set("calls", call_id, update)
|
||||
return False
|
||||
|
||||
@@ -60,7 +60,8 @@ USD_WHISPER_PER_MIN = 0.006
|
||||
USD_PER_EXTRACTION = 0.0005 # gpt-4o-mini scene extraction + embedding
|
||||
USD_PER_CORRECTION = 0.0003 # Gemini flash transcript correction
|
||||
USD_PER_GEOCODE = 0.005 # Google geocode, roughly one per located scene
|
||||
USD_PER_LLM_CORRELATE = 0.0005 # Gemini flash consensus decision
|
||||
USD_PER_LLM_CORRELATE = 0.0005 # one Gemini flash correlation decision (ambiguous scenes only)
|
||||
_LLM_CONSENSUS = ("llm", "rules_guard", "agreed", "tiebreak", "llm_orphan_gate")
|
||||
|
||||
# Fields the pipeline writes onto a call doc. Stripped when a call is copied
|
||||
# into the sandbox so the replay recomputes them instead of inheriting the
|
||||
@@ -636,7 +637,11 @@ def compute_metrics(incidents: list[dict], calls: list[dict]) -> dict:
|
||||
corr_path[r.get("corr_path") or "none"] += 1
|
||||
consensus[r.get("corr_consensus") or "none"] += 1
|
||||
linked = sum(1 for c in calls if c.get("incident_ids"))
|
||||
llm = sum(n for k, n in consensus.items() if k not in ("none", "rules_only"))
|
||||
# Scene decisions that made an LLM call (server-26#299: at most one each).
|
||||
# "rules_fallback" is excluded — the call failed or was unusable, so it
|
||||
# was mostly not billed. "agreed"/"tiebreak" are pre-#299 runs' values
|
||||
# (a tiebreak was two calls, so old runs under-count by their tiebreaks).
|
||||
llm = sum(n for k, n in consensus.items() if k in _LLM_CONSENSUS)
|
||||
return {
|
||||
"calls": len(calls),
|
||||
"calls_linked": linked,
|
||||
|
||||
Reference in New Issue
Block a user