logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

correlation: rules decide, one cheap LLM call only when ambiguous; drop the consensus tiebreak (server-26#299)

Browse Source 
The rules engine now commits every confident decision alone. _run_decision
marks a decision ambiguous (new_with_candidates, unfit_tg_candidate,
unverified_unit_link) and only then does upload._correlate make ONE
llm_correlator.decide call; a usable answer is committed, otherwise the rules
decision stands. Removed: _correlate_with_consensus, llm_correlator.tiebreak /
decisions_agree / tiebreak prompt, corr_smart_model, the correlation_smart
health tier, the LLM-orphan gate (_call_is_substanceless, effectively dead:
it could only fire on minor-severity untyped calls) and its sweep budget.
corr_consensus is now rules_only | llm | rules_fallback | rules_guard, with
corr_band naming why; admin debug and replay metrics read the new values.

Section fixes in the same pass:
- #115: no tiebreaker left to manufacture incidents; an LLM orphan of a
  rules "new" is honoured only for untyped, substanceless, routine/minor calls.
- #86: divergence veto bar moved to config (embedding_divergence_threshold);
  the band between it and the match bar is ambiguous, not a silent merge.
- #103: a unit link with no embedding is "unit_overlap_unverified" ->
  ambiguous, instead of skipping the veto and linking as verified.
- #189: unit-only and LLM links refused once an incident is open 90 min;
  unit links on incidents open >30 min are ambiguous; LLM links distance-
  guarded (incident_max_spread_km) and only to shown candidates.
- #190: severity steps down on an explicit stand-down phrase; new
  severity_peak keeps the monotonic max.
- #88: parse_llm_bool fails closed on "false"; scene closures honoured only
  on confident links or the incident the scene opened.
- #27: sweep budget is the scan window (wall clock), not 3 attempts.
- LLM prompt: "Reserve new" bullet no longer contradicts the #189 EXCEPT.
- updated_at never moves backward on a back-dated link.

Unmeasured: no live AI or replay (credits unfunded); band frequency, F1 and
thresholds are unverified. Must be re-scored on both answer keys in Replay
before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-27 20:26:38 -04:00
co-authored by Claude Opus 5.5
parent f4d78aba77
commit 3246fc97b0
18 changed files with 1207 additions and 1062 deletions
Download Patch File Download Diff File Expand all files Collapse all files
drb-c2-core/app/internal/recorrelation_sweep.py
+26 -26
View File
@@ -72,7 +72,7 @@ def pipeline_worst_case_seconds(n_scenes: int = _BOUND_SCENES) -> float:
correction = settings.gemini_timeout_seconds
extraction = openai_attempts * settings.openai_timeout_seconds
embeddings = n_scenes * openai_attempts * settings.openai_timeout_seconds
correlation = 2 * settings.gemini_timeout_seconds # decide + tiebreak, scene 0
correlation = settings.gemini_timeout_seconds # at most one LLM call, scene 0 (#299)
return stt + correction + extraction + embeddings + correlation + _NON_AI_SLACK_SECONDS
@@ -83,21 +83,25 @@ def _derive_min_minutes() -> int:
# 21 with the defaults (stt 120s, openai 60s x2 attempts, gemini 60s, 3 scenes).
MIN_MINUTES_SINCE_PIPELINE_START = _derive_min_minutes()
# Standard link-only retry budget before a call is tombstoned corr_path="unlinked".
MAX_SWEEP_ATTEMPTS = 3
# server-26#115 — a call the consensus LLM-orphan gate parked (llm=orphan vs
# rules=new, no substance) gets a longer budget. The gate fires before any
# incident for the job may exist, so the substantive call that would justify
# linking can land well after the standard ~6 min. Still link-only: a genuinely
# thin call must not mint an incident, and the rules creation gate would re-orphan
# it anyway.
GATED_ORPHAN_SWEEP_ATTEMPTS = 10
def _last_chance(call: dict, now: datetime) -> bool:
"""
server-26#27 — the retry budget is wall-clock, not an attempt count.
def _max_sweep_attempts(call: dict) -> int:
if call.get("corr_consensus") == "llm_orphan_gate":
return GATED_ORPHAN_SWEEP_ATTEMPTS
return MAX_SWEEP_ATTEMPTS
The old budget was 3 attempts, which with the #131 hold-back expired ~19
minutes after the pipeline started, while the incident the orphan belongs
to may be opened later than that (`f321b047` was tombstoned three minutes
after its own incident was created). An orphan is now retried on every
pass for as long as the scan query can see it, and tombstoned
corr_path="unlinked" only on its LAST visible pass: the one after which
its ended_at falls out of `recorrelation_scan_minutes`. A call with no
parseable ended_at gets no further retries.
"""
ended = _parse_dt(call.get("ended_at"))
if ended is None:
return True
remaining = timedelta(minutes=settings.recorrelation_scan_minutes) - (now - ended)
return remaining <= timedelta(minutes=settings.summary_interval_minutes)
async def recorrelation_loop() -> None:
@@ -149,9 +153,8 @@ async def _run_sweep_pass() -> None:
("status", "==", "ended"),
("ended_at", ">=", cutoff),
])
# corr_path="unlinked" is written after the attempt budget is exhausted.
# Allows a few retries so a welfare-check call can link to an escalation
# incident that is created a few minutes later, without sweeping 30× forever.
# corr_path="unlinked" is the tombstone written on an orphan's last
# visible pass (_last_chance); the scan window itself is the budget.
orphans = [
c for c in recent_ended
if not c.get("incident_ids") and not c.get("incident_id")
@@ -163,7 +166,6 @@ async def _run_sweep_pass() -> None:
# the thin path minutes later and attached to whatever was most recent —
# a second route into the over-merge the thin fix above addresses.
and not c.get("skip_reason")
and c.get("corr_sweep_count", 0) < _max_sweep_attempts(c)
and not _pipeline_likely_still_running(c, now)
]
@@ -173,14 +175,14 @@ async def _run_sweep_pass() -> None:
logger.info(f"Re-correlation sweep: {len(orphans)} orphaned call(s) to check")
linked = 0
for call in orphans:
if await _recorrelate_orphan(call):
if await _recorrelate_orphan(call, now):
linked += 1
if linked:
logger.info(f"Re-correlation sweep: linked {linked}/{len(orphans)} orphaned call(s)")
async def _recorrelate_orphan(call: dict) -> bool:
async def _recorrelate_orphan(call: dict, now: datetime) -> bool:
"""
Attempt to link a single orphaned call to an existing incident.
Returns True if a match was found and the call was linked.
@@ -223,12 +225,10 @@ async def _recorrelate_orphan(call: dict) -> bool:
)
return True
# Increment the attempt counter. Once the budget is reached the orphan filter
# above will stop picking this call up, and we write corr_path="unlinked" as
# a permanent tombstone.
attempts = call.get("corr_sweep_count", 0) + 1
update: dict = {"corr_sweep_count": attempts}
if attempts >= _max_sweep_attempts(call):
# The attempt count is kept for the admin debug view only; the budget is
# the scan window (_last_chance), after which the call is tombstoned.
update: dict = {"corr_sweep_count": call.get("corr_sweep_count", 0) + 1}
if _last_chance(call, now):
update["corr_path"] = "unlinked"
await fstore.doc_set("calls", call_id, update)
return False