Logan Cusano
97013e1505
Two independent sources of garbage in the AI pipeline, both visible in the 2026-08-16 correlation dump. 1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt text as preceding transcript, so on noisy or silent audio it continued the series, emitting transcripts that count upward from 10-4 to 10-99. The existing no_speech_prob filter could not catch these: the model is highly confident in text it invented by continuing a pattern. The prompt no longer contains a series to extend, and _is_degenerate() rejects the three shapes this failure takes: ascending ten-code runs, one phrase looping, and near-identical segments across a whole recording. Verified against 13 transcripts from production: all four known hallucinations rejected, all nine real ones kept, including terse traffic containing legitimate codes. 2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one dump. Each was transcribed, billed and correlated twice, and the resulting incident listed two units where there was one. Canonical selection is by earliest started_at, tie-broken on call_id, NOT by upload order: upload order varies with encode time and network latency, so it would make the authoritative recording non-deterministic. Call documents are created from MQTT call_start before uploads arrive, so both nodes independently reach the same verdict. The loser keeps its audio (it may be the cleaner capture) but is excluded from STT, correlation, the re-correlation sweep and the orphan debug view. Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is missing, where the caller unpacks two values. A missing key surfaced as a misleading "Transcription failed" instead of the real warning. Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an injected callable so it stays importable without firebase-admin present.
96 lines
4.7 KiB
Python
96 lines
4.7 KiB
Python
from pydantic_settings import BaseSettings
|
|
from typing import Optional
|
|
|
|
|
|
class Settings(BaseSettings):
|
|
# MQTT
|
|
mqtt_broker: str = "localhost"
|
|
mqtt_port: int = 1883
|
|
mqtt_user: Optional[str] = None
|
|
mqtt_pass: Optional[str] = None
|
|
|
|
# mosquitto's built-in dynamic-security plugin (see app/internal/dynsec.py).
|
|
# "admin" is hardcoded by the plugin itself on first boot — not actually
|
|
# configurable — kept as a named setting rather than a literal for
|
|
# readability. mqtt_dynsec_admin_pass must equal the mosquitto
|
|
# container's own MOSQUITTO_DYNSEC_PASSWORD env var (root .env /
|
|
# root.env.j2) or c2-core can't administer node credentials at all.
|
|
mqtt_dynsec_admin_user: str = "admin"
|
|
mqtt_dynsec_admin_pass: Optional[str] = None
|
|
|
|
# GCP
|
|
gcp_credentials_path: Optional[str] = None # None → uses ADC
|
|
gcs_bucket: Optional[str] = None # None → audio upload disabled
|
|
firestore_database: str = "(default)"
|
|
|
|
# Node health
|
|
node_offline_threshold: int = 90 # seconds without checkin before marking offline
|
|
|
|
# OpenAI (STT + intelligence)
|
|
openai_api_key: Optional[str] = None
|
|
stt_model: str = "whisper-1" # whisper-1 | gpt-4o-mini-transcribe | gpt-4o-transcribe
|
|
|
|
# Google Maps (geocoding)
|
|
google_maps_api_key: Optional[str] = None
|
|
|
|
# Gemini (intelligence extraction, embeddings, incident summaries)
|
|
gemini_api_key: Optional[str] = None
|
|
# Correlation consensus models
|
|
# corr_cheap_model — first-pass LLM correlator (runs on every call)
|
|
# corr_smart_model — tiebreaker (only fires when rules and cheap LLM disagree)
|
|
corr_cheap_model: str = "gemini-2.0-flash"
|
|
corr_smart_model: str = "gemini-1.5-pro"
|
|
summary_interval_minutes: int = 2 # how often the summary loop runs
|
|
correlation_window_hours: int = 2 # slow/location path: max hours since last call
|
|
embedding_similarity_threshold: float = 0.93 # slow-path: requires location corroboration
|
|
embedding_no_location_threshold: float = 0.97 # slow-path: match without location (very high bar)
|
|
embedding_cross_tg_threshold: float = 0.85 # cross-TG path: same dept + 2+ shared units
|
|
location_proximity_km: float = 0.5 # radius for location-proximity matching
|
|
geocode_max_km: float = 40.0 # reject geocode results farther than this from the node
|
|
incident_auto_resolve_minutes: int = 90 # auto-resolve after N minutes with no new calls
|
|
unit_continuity_max_idle_minutes: int = 20 # unit-continuity path: skip if incident idle > this
|
|
recorrelation_scan_minutes: int = 60 # re-examine orphaned calls ended within this window
|
|
tg_fast_path_idle_minutes: int = 90 # fast path: max minutes since incident last updated
|
|
tg_dispatch_thin_idle_minutes: int = 10 # dispatch channels only: thin calls only attach to incidents idle < this many minutes
|
|
|
|
# Vocabulary learning
|
|
vocabulary_induction_interval_hours: int = 24 # how often the induction loop runs
|
|
vocabulary_induction_sample_tokens: int = 4000 # ~tokens of transcript text sampled per system
|
|
|
|
# Internal service key — allows server-side services (discord bot) to call C2 without Firebase
|
|
service_key: Optional[str] = None
|
|
|
|
# Fleet-wide token edge nodes present to POST /nodes/enroll on first boot.
|
|
# Not a per-node secret — see routers/enrollment.py for why a leaked copy
|
|
# of this alone can't steal an already-approved node's key.
|
|
enrollment_token: Optional[str] = None
|
|
|
|
# Upload size limit — reject audio files larger than this (bytes). Default 100 MB.
|
|
upload_max_bytes: int = 100 * 1024 * 1024
|
|
|
|
# Public origin this API is reachable on, e.g. "https://api.drb.example.com".
|
|
# Only used to build absolute call-audio playback links: an <audio src> is
|
|
# fetched by the browser directly, so a relative path would resolve against
|
|
# the frontend origin, not this one.
|
|
public_api_url: Optional[str] = None
|
|
|
|
# How long a minted call-audio playback link stays valid. Long enough for a
|
|
# browsing session, short enough that a copied link isn't durable access.
|
|
audio_link_ttl_seconds: int = 6 * 60 * 60
|
|
|
|
# Two nodes hearing the same transmission start recording within about a
|
|
# second of each other (measured across node-002/node-PI-2 on TG 9048).
|
|
# 10s is generous against clock skew while staying well under the gap
|
|
# between genuinely separate transmissions on a busy dispatch channel.
|
|
duplicate_window_seconds: int = 10
|
|
|
|
# CORS — set to your frontend origin(s) in production, e.g. ["https://app.example.com"]
|
|
# Defaults to "*" for local development only.
|
|
cors_origins: list[str] = ["*"]
|
|
|
|
class Config:
|
|
env_file = ".env"
|
|
|
|
|
|
settings = Settings()
|