Logan CusanoandClaude Opus 5
33a247d306
The 2026-08-20 production dump had 4 of 6 sampled incidents as junk chains,
the worst being f5190670: 68 calls over 4h09m, 44 units, 12 tags, at least
13 genuinely distinct events. 58 of 133 linked calls took the fast/thin
path, which is the one path that attaches a call with no fit test at all.
Three defects combined to produce that, and all three are fixed here.
1. What counted as thin was wrong.
is_thin_call was "not units and not vehicles and not coords". A real
dispatch qualified as thin whenever no unit ID parsed and the geocode
failed - six of them did in that dump, including "All units head over to
the powerhouse, 55 Hyman Hills Road ... she's 87 years old", a brand new
job that attached to the four-hour chain and then overwrote its location
and its title. A call is now substantive if it carries tags, a location
string, a severity above routine, or is a reassignment; only genuinely
content-free housekeeping ("10-4", "Copy") stays thin. Those calls now go
through _call_fits_incident like everything else, which on a dispatch
backbone with no positive signal means they open their own incident or
orphan rather than merging.
The reassignment clause closes a self-defeating guard: upload.py blanks
units when dispatch pulls a unit onto a NEW job, specifically to stop
unit-overlap chaining - and blanking units made the call thin, routing it
to the only path with no fit check. The guard produced the merge it
existed to prevent.
2. The thin path was bounded on dispatch channels only.
Every other talkgroup fell through to "thin_pool = tg_recent": any
incident idle up to tg_fast_path_idle_minutes (90), no single-candidate
requirement, no fit test. The 30-second tier-1 / single-candidate tier-2
structure now applies to all channels. Non-dispatch gets its own window,
TG_THIN_IDLE_MINUTES=15, rather than sharing the dispatch value: a
tactical channel really is dedicated to one scene so it earns longer, but
15 sits inside the 20-minute tactical-default window already used in
_call_fits_incident, so the no-evidence path is never more permissive than
the fit-tested path on the same channel.
Recency gates now compare the magnitude of the idle, not the signed value.
The re-correlation sweep anchors "now" to the call's own started_at, so
idle goes negative routinely - incident 9d376ffe recorded
corr_incident_idle_min: -4.1 - and every "idle <= window" test in this
module reads True for a negative number. Those gates had silently stopped
bounding anything for exactly the calls the sweep re-examines.
3. Nothing capped an incident's total size.
Every fit test in the correlator is pairwise: does this call belong with
that incident. Each of f5190670's 68 links was individually arguable; the
mistake was the accumulated shape, which no pairwise rule can see. Two
hard caps now remove an incident from the candidate pool entirely, before
any path can choose it - including the LLM tier, which reads the same
ctx lists.
INCIDENT_MAX_DURATION_MINUTES=120. The one incident in that dump that was
genuinely a single event ran 63 minutes (06:15 wrong-way driver to 07:18
closeout), so the cap has to clear an hour with real headroom. The four
junk chains ran 3h41m, 3h43m, 4h05m and 4h09m, so it has to sit well under
three hours. 120 also equals correlation_window_hours: the location and
slow paths already refuse a candidate older than that, and the fast path
was the only one exempt, so this removes an inconsistency rather than
inventing a number.
INCIDENT_MAX_CALLS=40. A backstop for a burst that fills up inside the
duration cap, not the primary bound. The worst chain averaged ~16
calls/hour while absorbing an entire dispatch backbone, so 40 calls in
under two hours means one incident is eating most of the channel. Set
deliberately above any plausible single-incident call volume (a
multi-alarm fire on its own tactical channel) so this cap errs toward
keeping real incidents whole and lets the duration cap do the cutting.
Capping is not truncation: the incident keeps every call it has and still
auto-resolves on the normal idle sweep. It just stops being a candidate.
Every ambiguous call here was resolved toward a separate incident rather
than a merge. A wrongly-separate incident is visibly wrong and can be
merged later; a wrongly-merged one silently corrupts every unit, tag,
severity and map pin on the incident it joined, and poisons the AI
summary written from them. The cost is some acknowledgements orphaning
instead of riding along on an incident, which is a small, visible loss.
Deliberately NOT changed, since both push toward more merging while the
current failure mode is entirely over-merging (every incident in the dump
has exactly one "new" call; there is no over-splitting left to trade
against):
- unit-overlap positive feedback on shared dispatch channels, which is
now bounded by the caps rather than fixed at its root
- the sweep retry budget expiring before the target incident exists
Tests: 31 new cases in tests/test_correlator_merge_caps.py, including a
replay of the f5190670 night - 13 unrelated jobs at their real offsets,
plus roster unit traffic and acknowledgements every two minutes. Without
the caps that traffic still builds a 125-call incident spanning 244
minutes; with the old thinness test on top, 153 calls over 247 minutes in
3 incidents. With this commit it is 13 incidents, largest 40 calls over 80
minutes. Each new case was checked to fail when the behaviour it covers is
reverted. Suite: 138 passed.
No AI feature flag was touched; correlation stays off in production.
Closes logan/server-26#22
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
150 lines
8.3 KiB
Python
150 lines
8.3 KiB
Python
from pydantic_settings import BaseSettings
|
|
from typing import Optional
|
|
|
|
|
|
class Settings(BaseSettings):
|
|
# MQTT
|
|
mqtt_broker: str = "localhost"
|
|
mqtt_port: int = 1883
|
|
mqtt_user: Optional[str] = None
|
|
mqtt_pass: Optional[str] = None
|
|
|
|
# mosquitto's built-in dynamic-security plugin (see app/internal/dynsec.py).
|
|
# "admin" is hardcoded by the plugin itself on first boot — not actually
|
|
# configurable — kept as a named setting rather than a literal for
|
|
# readability. mqtt_dynsec_admin_pass must equal the mosquitto
|
|
# container's own MOSQUITTO_DYNSEC_PASSWORD env var (root .env /
|
|
# root.env.j2) or c2-core can't administer node credentials at all.
|
|
mqtt_dynsec_admin_user: str = "admin"
|
|
mqtt_dynsec_admin_pass: Optional[str] = None
|
|
|
|
# GCP
|
|
gcp_credentials_path: Optional[str] = None # None → uses ADC
|
|
gcs_bucket: Optional[str] = None # None → audio upload disabled
|
|
firestore_database: str = "(default)"
|
|
|
|
# Node health
|
|
node_offline_threshold: int = 90 # seconds without checkin before marking offline
|
|
|
|
# OpenAI (STT + intelligence)
|
|
openai_api_key: Optional[str] = None
|
|
stt_model: str = "whisper-1" # whisper-1 | gpt-4o-mini-transcribe | gpt-4o-transcribe
|
|
|
|
# Google Maps (geocoding)
|
|
google_maps_api_key: Optional[str] = None
|
|
|
|
# Gemini (intelligence extraction, embeddings, incident summaries)
|
|
gemini_api_key: Optional[str] = None
|
|
# Correlation consensus models
|
|
# corr_cheap_model — first-pass LLM correlator (runs on every call)
|
|
# corr_smart_model — tiebreaker (only fires when rules and cheap LLM disagree)
|
|
# Both IDs below were retired by Google and returned 404 on every call from
|
|
# some point before 2026-08-18 until they were corrected. Because a failed
|
|
# LLM call falls back to the rules decision, nothing broke loudly -- the
|
|
# entire LLM tier and the consensus tiebreak were simply dead in production
|
|
# while correlation behaviour was being tuned against rules-only output.
|
|
# Verify against https://ai.google.dev/gemini-api/docs/models before changing.
|
|
corr_cheap_model: str = "gemini-3.6-flash" # was gemini-2.0-flash (shut down)
|
|
corr_smart_model: str = "gemini-2.5-pro" # was gemini-1.5-pro (shut down)
|
|
summary_interval_minutes: int = 2 # how often the summary loop runs
|
|
correlation_window_hours: int = 2 # slow/location path: max hours since last call
|
|
embedding_similarity_threshold: float = 0.93 # slow-path: requires location corroboration
|
|
embedding_no_location_threshold: float = 0.97 # slow-path: match without location (very high bar)
|
|
embedding_cross_tg_threshold: float = 0.85 # cross-TG path: same dept + 2+ shared units
|
|
location_proximity_km: float = 0.5 # radius for location-proximity matching
|
|
geocode_max_km: float = 40.0 # reject geocode results farther than this from the node
|
|
incident_auto_resolve_minutes: int = 90 # auto-resolve after N minutes with no new calls
|
|
unit_continuity_max_idle_minutes: int = 20 # unit-continuity path: skip if incident idle > this
|
|
recorrelation_scan_minutes: int = 60 # re-examine orphaned calls ended within this window
|
|
tg_fast_path_idle_minutes: int = 90 # fast path: max minutes since incident last updated
|
|
# Dispatch channels only: tier-2 thin calls attach to a lone candidate idle < this.
|
|
# Was 10, which is long enough for the channel to have moved on to something else:
|
|
# on 2026-08-16 a "72 at Holland Station" incident absorbed a Grand Central train
|
|
# meet 9.6 min later, and a status check absorbed a records lookup at 9.7 min.
|
|
# Across that dump every correct thin attach was <= 3.4 min idle and every wrong
|
|
# one was >= 8.2, so 5 separates them with room on both sides. Genuine
|
|
# back-and-forth is handled by the 30-second tier-1 path above this.
|
|
tg_dispatch_thin_idle_minutes: int = 5
|
|
# Every other channel: tier-2 thin calls attach to a lone candidate idle < this.
|
|
# Non-dispatch talkgroups previously had NO tier-2 bound at all — they used the
|
|
# whole 90-minute tg_fast_path_idle_minutes window with no single-candidate
|
|
# requirement and no fit test, which is the widest version of the 2026-08-20
|
|
# over-merge. A tactical channel really is dedicated to one scene, so it earns
|
|
# a longer window than a dispatch backbone, but not an unbounded one: 15 sits
|
|
# inside the 20-minute tactical-default window in _call_fits_incident, so the
|
|
# no-evidence thin path is never more permissive than the fit-tested path on
|
|
# the same channel.
|
|
tg_thin_idle_minutes: int = 15
|
|
|
|
# ── Hard caps: an incident past either of these stops accepting calls ──────
|
|
# Enforced on every correlation path (see _incident_at_capacity). Pairwise fit
|
|
# tests judge one call against one incident and cannot see the shape of the
|
|
# chain they are building, so these are the only guard against a "work shift"
|
|
# incident regardless of how individually plausible each link looked.
|
|
#
|
|
# 120 minutes: the one incident in the 2026-08-20 dump that was genuinely a
|
|
# single event ran 63 minutes (06:15 wrong-way driver → 07:18 closeout), so
|
|
# the cap has to clear an hour with real headroom. The four junk chains ran
|
|
# 3h41m, 3h43m, 4h05m and 4h09m, so it has to sit well under three hours.
|
|
# 120 also equals correlation_window_hours: the location and slow paths
|
|
# already refuse to consider a candidate older than that, and the fast path
|
|
# was the only one exempt. Making it agree removes that inconsistency rather
|
|
# than inventing a new number.
|
|
incident_max_duration_minutes: int = 120
|
|
# 40 calls: a backstop for a burst that fills up inside the duration cap
|
|
# rather than the primary bound. The worst observed chain averaged ~16
|
|
# calls/hour while absorbing an ENTIRE dispatch backbone, so 40 calls in
|
|
# under two hours means the incident is eating most of the channel — that is
|
|
# a chain, not an event. Set deliberately above any plausible single-incident
|
|
# call volume (a multi-alarm fire on its own tactical channel) so this cap
|
|
# errs toward keeping real incidents whole and lets the duration cap do the
|
|
# cutting.
|
|
incident_max_calls: int = 40
|
|
|
|
# Vocabulary learning
|
|
vocabulary_induction_interval_hours: int = 24 # how often the induction loop runs
|
|
vocabulary_induction_sample_tokens: int = 4000 # ~tokens of transcript text sampled per system
|
|
|
|
# Internal service key — allows server-side services (discord bot) to call C2 without Firebase
|
|
service_key: Optional[str] = None
|
|
|
|
# Fleet-wide token edge nodes present to POST /nodes/enroll on first boot.
|
|
# Not a per-node secret — see routers/enrollment.py for why a leaked copy
|
|
# of this alone can't steal an already-approved node's key.
|
|
enrollment_token: Optional[str] = None
|
|
|
|
# Upload size limit — reject audio files larger than this (bytes). Default 100 MB.
|
|
upload_max_bytes: int = 100 * 1024 * 1024
|
|
|
|
# Public origin this API is reachable on, e.g. "https://api.drb.example.com".
|
|
# Only used to build absolute call-audio playback links: an <audio src> is
|
|
# fetched by the browser directly, so a relative path would resolve against
|
|
# the frontend origin, not this one.
|
|
public_api_url: Optional[str] = None
|
|
|
|
# How long a minted call-audio playback link stays valid. Long enough for a
|
|
# browsing session, short enough that a copied link isn't durable access.
|
|
audio_link_ttl_seconds: int = 6 * 60 * 60
|
|
|
|
# Two nodes hearing the same transmission start recording within about a
|
|
# second of each other (measured across node-002/node-PI-2 on TG 9048).
|
|
# 10s is generous against clock skew while staying well under the gap
|
|
# between genuinely separate transmissions on a busy dispatch channel.
|
|
duplicate_window_seconds: int = 10
|
|
|
|
# CORS — set to your frontend origin(s) in production, e.g. ["https://app.example.com"]
|
|
# Defaults to "*" for local development only.
|
|
cors_origins: list[str] = ["*"]
|
|
|
|
# Discord webhook URL that app/internal/ai_health.py posts to when an AI
|
|
# tier (transcription/correlation) transitions into or out of degraded
|
|
# state. Empty disables the POST entirely — not every self-hosted
|
|
# deployment will set this up, and skipping it must be silent.
|
|
ai_alert_webhook_url: str = ""
|
|
|
|
class Config:
|
|
env_file = ".env"
|
|
|
|
|
|
settings = Settings()
|