incidents: severity-scaled quiet timer, reopen-on-link, thin calls don't fill the cap
Hand-labelling the 09-22 10:00-12:00 ET replay window (server-26#170, answer key replay_groundtruth_0922.json) found ~25 real incidents, of which only ~5 had an audible clear — most jobs clear by MDT, so the quiet timer is the close for most incidents and a flat 90 minutes left a lockout or a plate check "active" on the portal an hour after it ended. - summarizer: timer close after 30 min quiet for routine/minor, 60 moderate, 90 major/unknown. A timer close is provisional: reopenable=True. - correlator: reopenable incidents inside incident_reopen_window_minutes (90, since last substantive call) stay candidates; linking a call to one reopens it (status active, reopened_count++). The sweep expires the flag so the reopenable pool stays bounded. Real clears (units_cleared, llm_closure) are never reopenable. - cap: incident_max_calls counts substantive calls only (substantive_call_count). The bridge MVA hit 40 in 32 min with ~40% thin replies, split in half, and the second half took another job's title. c2-core: 467 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
731b54bed9
commit
969d175a67
@@ -142,15 +142,41 @@ async def _summarize_incident(inc: dict) -> None:
|
||||
await fstore.doc_set("incidents", incident_id, updates)
|
||||
|
||||
|
||||
def _auto_resolve_minutes(inc: dict) -> int:
|
||||
"""Quiet time before a timer close, by severity (see config: incident_auto_resolve_minutes_*)."""
|
||||
sev = (inc.get("severity") or "").lower()
|
||||
if sev in ("routine", "minor"):
|
||||
return settings.incident_auto_resolve_minutes_routine
|
||||
if sev == "moderate":
|
||||
return settings.incident_auto_resolve_minutes_moderate
|
||||
return settings.incident_auto_resolve_minutes
|
||||
|
||||
|
||||
async def _expire_reopen_windows(now) -> None:
|
||||
"""A timer-closed incident stops being reopenable once its window passes,
|
||||
so the correlator's reopenable pool stays bounded."""
|
||||
window = timedelta(minutes=settings.incident_reopen_window_minutes)
|
||||
for inc in await fstore.collection_list("incidents", status="resolved", reopenable=True):
|
||||
try:
|
||||
updated = datetime.fromisoformat(str(inc.get("updated_at", "")).replace("Z", "+00:00"))
|
||||
if updated.tzinfo is None:
|
||||
updated = updated.replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
updated = None
|
||||
if updated is None or now - updated > window:
|
||||
await fstore.doc_set("incidents", inc["incident_id"], {"reopenable": False})
|
||||
|
||||
|
||||
async def _resolve_stale_incidents() -> None:
|
||||
"""Auto-resolve active incidents that have had no new calls for incident_auto_resolve_minutes."""
|
||||
"""Timer-close active incidents that have been quiet longer than their severity allows."""
|
||||
from app.internal import clock
|
||||
await _expire_reopen_windows(clock.now())
|
||||
all_active = await fstore.collection_list("incidents", status="active")
|
||||
if not all_active:
|
||||
return
|
||||
|
||||
from app.internal import clock
|
||||
now = clock.now()
|
||||
cutoff = timedelta(minutes=settings.incident_auto_resolve_minutes)
|
||||
count = 0
|
||||
|
||||
for inc in all_active:
|
||||
@@ -164,11 +190,12 @@ async def _resolve_stale_incidents() -> None:
|
||||
if updated_dt.tzinfo is None:
|
||||
updated_dt = updated_dt.replace(tzinfo=timezone.utc)
|
||||
idle_minutes = (now - updated_dt).total_seconds() / 60
|
||||
if idle_minutes > settings.incident_auto_resolve_minutes:
|
||||
if idle_minutes > _auto_resolve_minutes(inc):
|
||||
await fstore.doc_set("incidents", incident_id, {
|
||||
"status": "resolved",
|
||||
"resolved_at": now.isoformat(),
|
||||
"resolved_via": "idle_timeout",
|
||||
"reopenable": True,
|
||||
})
|
||||
from app.internal.incident_correlator import maybe_resolve_parent
|
||||
await maybe_resolve_parent(incident_id)
|
||||
|
||||
Reference in New Issue
Block a user