correlation: rules decide, one cheap LLM call only when ambiguous; drop the consensus tiebreak (server-26#299)
The rules engine now commits every confident decision alone. _run_decision marks a decision ambiguous (new_with_candidates, unfit_tg_candidate, unverified_unit_link) and only then does upload._correlate make ONE llm_correlator.decide call; a usable answer is committed, otherwise the rules decision stands. Removed: _correlate_with_consensus, llm_correlator.tiebreak / decisions_agree / tiebreak prompt, corr_smart_model, the correlation_smart health tier, the LLM-orphan gate (_call_is_substanceless, effectively dead: it could only fire on minor-severity untyped calls) and its sweep budget. corr_consensus is now rules_only | llm | rules_fallback | rules_guard, with corr_band naming why; admin debug and replay metrics read the new values. Section fixes in the same pass: - #115: no tiebreaker left to manufacture incidents; an LLM orphan of a rules "new" is honoured only for untyped, substanceless, routine/minor calls. - #86: divergence veto bar moved to config (embedding_divergence_threshold); the band between it and the match bar is ambiguous, not a silent merge. - #103: a unit link with no embedding is "unit_overlap_unverified" -> ambiguous, instead of skipping the veto and linking as verified. - #189: unit-only and LLM links refused once an incident is open 90 min; unit links on incidents open >30 min are ambiguous; LLM links distance- guarded (incident_max_spread_km) and only to shown candidates. - #190: severity steps down on an explicit stand-down phrase; new severity_peak keeps the monotonic max. - #88: parse_llm_bool fails closed on "false"; scene closures honoured only on confident links or the incident the scene opened. - #27: sweep budget is the scan window (wall clock), not 3 attempts. - LLM prompt: "Reserve new" bullet no longer contradicts the #189 EXCEPT. - updated_at never moves backward on a back-dated link. Unmeasured: no live AI or replay (credits unfunded); band frequency, F1 and thresholds are unverified. Must be re-scored on both answer keys in Replay before merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
f4d78aba77
commit
3246fc97b0
@@ -109,9 +109,9 @@ async def debug_correlation(
|
||||
"scene_index": scene_index,
|
||||
"transcript": scene.get("transcript"),
|
||||
"incident_id": scene.get("incident_id"),
|
||||
# server-26#139: this scene's OWN incident_type/severity, as seen
|
||||
# by _call_is_substanceless at decision time — not the call doc's
|
||||
# flat top-level field, which is last-scene-wins (server-26#96).
|
||||
# server-26#139: this scene's OWN incident_type/severity at decision
|
||||
# time — not the call doc's flat top-level field, which is
|
||||
# last-scene-wins (server-26#96).
|
||||
"incident_type": scene.get("incident_type"),
|
||||
"severity": scene.get("severity"),
|
||||
"corr_path": corr_debug.get("corr_path"),
|
||||
@@ -126,7 +126,7 @@ async def debug_correlation(
|
||||
"corr_llm_reasoning": corr_debug.get("corr_llm_reasoning"),
|
||||
"corr_llm_action": corr_debug.get("corr_llm_action"),
|
||||
"corr_rules_action": corr_debug.get("corr_rules_action"),
|
||||
"corr_gate_veto": corr_debug.get("corr_gate_veto"),
|
||||
"corr_band": corr_debug.get("corr_band"),
|
||||
}
|
||||
|
||||
def _call_summary(call: dict) -> dict:
|
||||
@@ -173,20 +173,18 @@ async def debug_correlation(
|
||||
"corr_matched_units": call.get("corr_matched_units"),
|
||||
"corr_sweep_count": call.get("corr_sweep_count"),
|
||||
"skip_reason": call.get("skip_reason"),
|
||||
# LLM consensus tier fields — written by upload.py's
|
||||
# _correlate_with_consensus / llm_correlator.py, but previously
|
||||
# dropped here, making it impossible to tell from this endpoint
|
||||
# whether the LLM correlation tier is actually running (server-26#24).
|
||||
# LLM tier fields — written by upload.py's _correlate /
|
||||
# llm_correlator.py, but previously dropped here, making it
|
||||
# impossible to tell from this endpoint whether the LLM
|
||||
# correlation tier is actually running (server-26#24).
|
||||
"corr_consensus": call.get("corr_consensus"),
|
||||
"corr_llm_reasoning": call.get("corr_llm_reasoning"),
|
||||
"corr_llm_action": call.get("corr_llm_action"),
|
||||
"corr_rules_action": call.get("corr_rules_action"),
|
||||
# server-26#115 — why an llm=orphan/rules=new disagreement escalated
|
||||
# to tiebreak instead of being gated (see upload.py's
|
||||
# _call_is_substanceless). Present only on that disagreement shape;
|
||||
# written here specifically so a live measurement window can read
|
||||
# the reason instead of reconstructing it by hand from the dump.
|
||||
"corr_gate_veto": call.get("corr_gate_veto"),
|
||||
# server-26#299 — why the rules marked this decision ambiguous and
|
||||
# sent it to the LLM (incident_correlator._ambiguity). Absent on a
|
||||
# confident rules decision.
|
||||
"corr_band": call.get("corr_band"),
|
||||
# server-26#127 — shadow-mode upstream chatter classifier verdict.
|
||||
# Written by intelligence.extract_scenes on every transcript that
|
||||
# reaches real scene extraction (not on garbage/too-short skips).
|
||||
@@ -298,7 +296,8 @@ async def debug_correlation(
|
||||
orphans_by_tg[tg_key]["count"] += 1
|
||||
if not o.get("incident_type") and not o.get("tags"):
|
||||
orphans_by_tg[tg_key]["no_type_count"] += 1
|
||||
if (o.get("corr_sweep_count") or 0) >= 3:
|
||||
# Tombstoned by the sweep's wall-clock budget (server-26#27).
|
||||
if o.get("corr_path") == "unlinked":
|
||||
orphans_by_tg[tg_key]["sweep_exhausted_count"] += 1
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────
|
||||
@@ -374,9 +373,8 @@ async def debug_correlation(
|
||||
"corr_fit_signal": _tally(e.get("corr_fit_signal") for e in scene_entries),
|
||||
"corr_consensus": _tally(e.get("corr_consensus") for e in scene_entries),
|
||||
"corr_llm_action": _tally(e.get("corr_llm_action") for e in scene_entries),
|
||||
# server-26#115 — this IS the number the escape-hatch fix exists to
|
||||
# produce: why each llm=orphan/rules=new call escaped the gate.
|
||||
"corr_gate_veto": _tally(e.get("corr_gate_veto") for e in scene_entries),
|
||||
# server-26#299 — how often, and why, the LLM tier was consulted.
|
||||
"corr_band": _tally(e.get("corr_band") for e in scene_entries),
|
||||
# server-26#127 — shadow-mode chatter classifier. The target
|
||||
# population is non-events, which land as orphans or single-call
|
||||
# incidents, NOT as a slice of every linked call -- tally `orphans`
|
||||
|
||||
Reference in New Issue
Block a user