correlation: rules decide, one cheap LLM call only when ambiguous; drop the consensus tiebreak (server-26#299)

The rules engine now commits every confident decision alone. _run_decision
marks a decision ambiguous (new_with_candidates, unfit_tg_candidate,
unverified_unit_link) and only then does upload._correlate make ONE
llm_correlator.decide call; a usable answer is committed, otherwise the rules
decision stands. Removed: _correlate_with_consensus, llm_correlator.tiebreak /
decisions_agree / tiebreak prompt, corr_smart_model, the correlation_smart
health tier, the LLM-orphan gate (_call_is_substanceless, effectively dead:
it could only fire on minor-severity untyped calls) and its sweep budget.
corr_consensus is now rules_only | llm | rules_fallback | rules_guard, with
corr_band naming why; admin debug and replay metrics read the new values.

Section fixes in the same pass:
- #115: no tiebreaker left to manufacture incidents; an LLM orphan of a
  rules "new" is honoured only for untyped, substanceless, routine/minor calls.
- #86: divergence veto bar moved to config (embedding_divergence_threshold);
  the band between it and the match bar is ambiguous, not a silent merge.
- #103: a unit link with no embedding is "unit_overlap_unverified" ->
  ambiguous, instead of skipping the veto and linking as verified.
- #189: unit-only and LLM links refused once an incident is open 90 min;
  unit links on incidents open >30 min are ambiguous; LLM links distance-
  guarded (incident_max_spread_km) and only to shown candidates.
- #190: severity steps down on an explicit stand-down phrase; new
  severity_peak keeps the monotonic max.
- #88: parse_llm_bool fails closed on "false"; scene closures honoured only
  on confident links or the incident the scene opened.
- #27: sweep budget is the scan window (wall clock), not 3 attempts.
- LLM prompt: "Reserve new" bullet no longer contradicts the #189 EXCEPT.
- updated_at never moves backward on a back-dated link.

Unmeasured: no live AI or replay (credits unfunded); band frequency, F1 and
thresholds are unverified. Must be re-scored on both answer keys in Replay
before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-27 20:26:38 -04:00
co-authored by Claude Opus 5.5
parent f4d78aba77
commit 3246fc97b0
18 changed files with 1207 additions and 1062 deletions
+7 -2
View File
@@ -60,7 +60,8 @@ USD_WHISPER_PER_MIN = 0.006
USD_PER_EXTRACTION = 0.0005 # gpt-4o-mini scene extraction + embedding
USD_PER_CORRECTION = 0.0003 # Gemini flash transcript correction
USD_PER_GEOCODE = 0.005 # Google geocode, roughly one per located scene
USD_PER_LLM_CORRELATE = 0.0005 # Gemini flash consensus decision
USD_PER_LLM_CORRELATE = 0.0005 # one Gemini flash correlation decision (ambiguous scenes only)
_LLM_CONSENSUS = ("llm", "rules_guard", "agreed", "tiebreak", "llm_orphan_gate")
# Fields the pipeline writes onto a call doc. Stripped when a call is copied
# into the sandbox so the replay recomputes them instead of inheriting the
@@ -636,7 +637,11 @@ def compute_metrics(incidents: list[dict], calls: list[dict]) -> dict:
corr_path[r.get("corr_path") or "none"] += 1
consensus[r.get("corr_consensus") or "none"] += 1
linked = sum(1 for c in calls if c.get("incident_ids"))
llm = sum(n for k, n in consensus.items() if k not in ("none", "rules_only"))
# Scene decisions that made an LLM call (server-26#299: at most one each).
# "rules_fallback" is excluded — the call failed or was unusable, so it
# was mostly not billed. "agreed"/"tiebreak" are pre-#299 runs' values
# (a tiebreak was two calls, so old runs under-count by their tiebreaks).
llm = sum(n for k, n in consensus.items() if k in _LLM_CONSENSUS)
return {
"calls": len(calls),
"calls_linked": linked,