Two independent sources of garbage in the AI pipeline, both visible in the
2026-08-16 correlation dump.
1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run
of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt
text as preceding transcript, so on noisy or silent audio it continued the
series, emitting transcripts that count upward from 10-4 to 10-99. The
existing no_speech_prob filter could not catch these: the model is highly
confident in text it invented by continuing a pattern.
The prompt no longer contains a series to extend, and _is_degenerate()
rejects the three shapes this failure takes: ascending ten-code runs, one
phrase looping, and near-identical segments across a whole recording.
Verified against 13 transcripts from production: all four known
hallucinations rejected, all nine real ones kept, including terse traffic
containing legitimate codes.
2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both
uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one
dump. Each was transcribed, billed and correlated twice, and the resulting
incident listed two units where there was one.
Canonical selection is by earliest started_at, tie-broken on call_id, NOT
by upload order: upload order varies with encode time and network latency,
so it would make the authoritative recording non-deterministic. Call
documents are created from MQTT call_start before uploads arrive, so both
nodes independently reach the same verdict. The loser keeps its audio (it
may be the cleaner capture) but is excluded from STT, correlation, the
re-correlation sweep and the orphan debug view.
Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is
missing, where the caller unpacks two values. A missing key surfaced as a
misleading "Transcription failed" instead of the real warning.
Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an
injected callable so it stays importable without firebase-admin present.
upload_audio() could only sign a URL when GCP_CREDENTIALS_PATH pointed at a
service-account key file. The deployed VM runs on Application Default
Credentials with no key file, so every upload silently took the fallback
branch and returned a bare gs:// URI. That broke two things at once:
* Browsers cannot fetch a gs:// URI, so no recording was ever playable.
* _public_url_to_gcs_uri() only matched https://storage.googleapis.com/ and
returned None for it, so `if gcs_uri:` in the upload path was always false
and transcription never ran. Nothing was logged, which is why this looked
like an OpenAI credits problem rather than a storage one.
The fallback also interpolated the client-supplied filename instead of the
call_id-derived safe name, so the URI did not even name the object written.
Calls now store only the canonical gs:// location. A short-lived playback link
is minted per read as an HMAC over (call_id, expiry) keyed by SERVICE_KEY, and
audio is served from the private bucket by the new /media route. An <audio src>
cannot carry an Authorization header, so the link has to be the credential;
that router is therefore public with the check done inline, as enrollment.py
already does. Signing GCS URLs from the VM would have needed a
serviceAccountTokenCreator grant on its own service account — this avoids the
IAM change entirely and keeps the bucket private.
gcs_uri_for_call() reconstructs the object name from call_id, so recordings
made before this fix are reachable again without a data migration.
Frontend rows come straight from Firestore via onSnapshot and never see a
server-minted field, so CallRow fetches the link lazily on expand.
Also removes the last long-lived (1 year) signed URL and the log line that
printed it.
Edge nodes are deployed to arbitrary locations by arbitrary people, so the
broker has to be reachable from the internet and secured on its own merits
rather than by a VPN.
Three defects made that impossible. The broker only had a plaintext 1883
listener; every node shared one drb-node password; and the ACL pattern used
%c, the client-supplied client id, so any holder of that shared password
could set client_id to another node and take over its namespace. The comment
claiming this cryptographically prevented cross-node access was wrong and is
gone.
Authentication now uses mosquitto 2.x's built-in dynamic-security plugin on
the stock eclipse-mosquitto image. c2-core administers it over the control
topic, creating each node's client on approval with username=<node_id> and
password=<its node_keys api_key>, attached to a role whose ACL is nodes/%u/#
against the authenticated username. One credential, one revocation point.
An HTTP-callback plugin was implemented first and rejected: that project is
archived upstream, which is not an acceptable dependency on an
internet-facing broker.
Because dynsec state is a second source of truth alongside Firestore,
approve/reissue/delete now write to the broker first and surface a 502
rather than drifting, and c2-core reconciles every approved node into dynsec
on startup.
Adds node self-enrollment (POST /nodes/enroll, GET /nodes/{id}/credentials)
so a new node can obtain its key over HTTPS without an operator handling
secrets by hand. Enrolling an already-approved node_id is refused on the
fleet token alone — otherwise a leaked token plus a guessable id would let
an attacker steal a live node's key before the real node asked for it.
Pickup secrets are stored hashed and returned once, and the endpoint is rate
limited per source IP.
Infrastructure: an 8883 TLS listener fed by Caddy's certificate via a
systemd path unit, a firewall rule for it, and Caddy now 404s /internal/*
so the api vhost cannot proxy internal routes.
Also fixes CORS, which allowed https://app.<domain> while the frontend is
served on the bare domain — every call from the portal would have failed —
and widens the vault gitignore to a glob, since ansible-vault leaves
backup siblings that the exact-name rule left committable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
auth.py
secrets.compare_digest replaces == for service key comparison (timing-safe)
Added require_service_key — bot-only endpoints (trip/event join/leave)
Added require_service_key_or_admin — node commands/config (bot via service key OR dashboard admin via Firebase)
Added _RateLimiter with three shared instances: trip_chat_limiter (20/5min per user), summarize_limiter (5/10min per incident), bootstrap_limiter (2/hr per system)
nodes.py
send_command and assign_system now require require_service_key_or_admin — the Discord bot can still call them via service key, but regular Firebase users are blocked
tokens.py
add_token, flush_tokens, set_preferred_system, delete_token all require require_admin_token
Token masking changed from token[:10] + "…" + token[-4:] to "•••" + token[-4:]
systems.py
All write endpoints (create, update, delete, ai-flags, ten-codes, vocabulary writes, bootstrap) now require require_admin_token
bootstrap_vocabulary also calls bootstrap_limiter.check(system_id)
incidents.py
POST /incidents/summarize (bulk) now requires require_admin_token
POST /incidents/{id}/summarize now calls summarize_limiter.check(incident_id)
trips.py
join_trip, leave_trip, join_event, leave_event require require_service_key — only the Discord bot can set Discord attendee identity
delete_trip, delete_event require require_service_key_or_admin
trip_chat rate-limited per caller UID, history stripped to user/assistant roles only, user message truncated to 2000 chars, Maps query strings capped at 200 chars
upload.py
Rejects files larger than settings.upload_max_bytes (default 100MB) with 413
storage.py
_safe_audio_filename() derives GCS object name from call_id + allowlisted extension, completely ignoring the client-supplied filename
config.py
Added upload_max_bytes: int = 100 * 1024 * 1024
Both Dockerfiles — python:3.14-slim → python:3.12-slim
- correlator: unit_overlap on dispatch channels now applies content
divergence check when the call has geocoded coords but the incident
doesn't; previously this gap caused unrelated calls to merge into
stale incidents (e.g. patrol officer at a second scene 70 min later)
- STT: switch default model from gpt-4o-transcribe to whisper-1, which
faithfully transcribes all exchanges in multi-PTT recordings; gpt-4o
was silently dropping utterances, starving the correlation engine
- STT: remove vocabulary from the Whisper prompt; whisper-1 echoes
prompted terms into noise/silence, skewing extracted incident data;
vocabulary context is now applied exclusively in the GPT extraction
step (build_gpt_vocab_block) where it is used as reference only
Refactor incident_correlator.py to a decision/commit split (preview_correlation
/ apply_correlation) so the rules engine and LLM can both produce decisions before
anything is written to Firestore.
Add llm_correlator.py: cheap Gemini Flash first-pass + Gemini Pro tiebreaker.
Wire _correlate_with_consensus in upload.py — rules-only fallback when key is
absent or call is thin; agreed/tiebreak consensus written to corr_debug.
- Cap unit-continuity path at 20 min idle (unit_continuity_max_idle_minutes)
- Block time_fallback and unit-continuity matching on reassignment calls
- Expand reassignment detection to cover unit-initiated self-reassignment
- Skip GPT extraction entirely for transcripts ≤5 words (prevents hallucinated tags/units)
- Reduce geocode_max_km from 75 to 40 to reject far-out-of-area results
- Include county in geocoding query for tighter jurisdiction anchoring
geocode_max_km: 25 → 75 km. The node is a physical receiver, not the system boundary;
digital repeaters extend coverage well beyond 25km (North White Plains at 35.5km from
the Yorktown node is a legitimate Westchester County location).
Query now fully qualified: "High Street" → "High Street, Yorktown, New York".
Added _get_node_state() which reverse-geocodes the node position once (cached) using
Google Maps to get the state name, appended alongside the municipality.
Generic street names (High Street, Main Street) no longer resolve to wrong-country results.
Switch geocoding from Nominatim to Google Maps Geocoding API for accurate
local place name resolution (bounds-biased, with 25km distance rejection guard).
Remove the now-unused _get_node_place reverse-geocoder and _node_place_cache.
Map page (TOC improvements):
- Weather radar tiles auto-refresh every 5 minutes via radarEpoch key cycling
- Google Maps traffic overlay added to LayersControl
- Live 24h clock overlay at bottom-left for situational awareness
- Incident sidebar cards now show age (time since dispatch) and unit count
Nominatim's viewbox is advisory (bounded=0), so ambiguous place names like
"Pinebrook" can resolve to locations 30-40km away in the wrong town. Added
a post-geocode distance gate: results farther than geocode_max_km (default
25km) from the node are discarded with a warning log rather than written to
the incident.
Also logs distance on successful geocodes for easier audit.
New config setting: geocode_max_km (float, default 25.0)
Correlator
- Raise fast-path idle gate 30 → 90 min (tg_fast_path_idle_minutes)
- Fix disambiguate always-commits bug: run _call_fits_incident on winner
before committing; fall through to new-incident creation if it fails
- Add unit-continuity path (path 1.5): matches all_active by shared unit
IDs with a reassignment guard, bridges calls past the idle gate
- Add tag-based incident_type inference (_TAG_TYPE_HINTS) as GPT fallback,
rescuing tagged calls that would have been dropped (616 observed orphans)
- Add master/child incident model: _create_master_incident, _demote_to_child,
_add_child_to_master; new incidents stamped incident_type="master"
- Add cross-system parent detection (_find_cross_system_parent): two-signal
scoring (road overlap=0.4, embedding≥0.78=0.3, proximity=0.3, threshold=0.5)
wired into create-if-new path; creates master shell on first cross-system match
- Add maybe_resolve_parent: auto-resolves master when all children close;
called from upload pipeline (LLM closure) and summarizer stale sweep
- Add signal-based auto-resolve via units_active/units_cleared tracking:
GPT now extracts cleared_units per scene; _update_incident moves units
between active/cleared lists and resolves the incident when active empties;
stored on call doc for re-correlation sweep reuse
- Add _create_incident initialization of units_active/units_cleared fields
Re-correlation sweep
- Add corr_sweep_count + MAX_SWEEP_ATTEMPTS=3: orphans get 3 attempts
then are tombstoned as corr_path="unlinked", ending the re-sweep loop
(previously hammering each orphan 29-31 times per shift)
Intelligence extraction
- Add cleared_units to GPT prompt schema and rules
- Extract and propagate cleared_units per scene; merge across scenes;
store on call doc for re-correlation sweep
Token management
- Fix token release bug: remove release_token call on discord_connected=False
in MQTT checkin (transient Discord drops were orphaning bots mid-shift)
- Add PUT /tokens/{id}/prefer/{system_id} endpoint: lock a bot token to a
system; pass _none as system_id to clear; stored bidirectionally on both
token and system documents
- discord_join handler resolves preferred_token_id from system doc and passes
system_name in MQTT payload
- *`correlate_call`* — added units and vehicles optional params; when provided (per-scene from intelligence extraction), they take priority over the merged call-document values, preventing multi-scene unit contamination
- *Cross-TGID correlation path (2.5)* — *new path between location and slow paths*: when a call shares 2+ unit IDs with a recent same-system, same-type incident AND embedding similarity ≥ 0.85, it links them — catches multi-talkgroup pursuits like the bicycle search that split across dispatch/tactical/geographic channels
# `app/internal/intelligence.py`
- *`reassignment` field* — added to the GPT-4o-mini prompt schema and rules; `true` when dispatch is actively pulling a unit to a new, different call (not a status update or en route acknowledgement); returned in every processed scene dict
- *Tag location rule* — added explicit instruction to the prompt: tags must describe what happened, not where; place names, road names, and talkgroup names are explicitly forbidden as tags
# `app/routers/upload.py`
- Both scene correlation call sites (`_run_extraction_pipeline` and `_run_intelligence_pipeline`) now pass `units=corr_units` where `corr_units = [] if scene.get("reassignment") else scene.get("units") `— suppresses unit overlap matching when a unit is being reassigned to a new call, preventing chaining into their previous incident
- Both sites also pass `vehicles=scene.get("vehicles")` (per-scene vehicles, from the multi-scene units fix)
# `app/config.py`
- `embedding_cross_tg_threshold: float = 0.85` — threshold for the new cross-TGID path
incident_correlator.py — full rewrite: always runs on every call, fetches all active incidents cross-type, fast path collects all talkgroup matches and disambiguates by unit/vehicle overlap → location proximity → embedding, new location proximity path, slow path requires location corroboration, "Auto:" stripped from titles, "auto-generated" tag added, units/vehicles now accumulated on update
intelligence.py — resolved field in GPT schema, returned as 5th value
upload.py — both pipelines unpack 5-tuple, always call correlate, auto-resolve on resolved=True
summarizer.py — stale sweep runs each tick, resolves incidents idle for 90+ minutes
config.py — correlation_window_hours=2, embedding_similarity_threshold=0.93, location_proximity_km=0.5, incident_auto_resolve_minutes=90
Logan Cusano
