/upload compared the per-node API key with a plain !=, which short-circuits on
the first differing byte and so leaks a little information about how much of a
guess was correct.
The reason to fix it is less the timing channel itself -- an HTTP round trip is
noisy -- than the inconsistency: enrollment.py and dynsec.py both went out of
their way to use secrets.compare_digest for the same class of credential, so the
codebase contradicted itself on whether this mattered. Now it does not.
Also coalesces a missing api_key field to "" so compare_digest is never handed
None, which would raise TypeError and turn a malformed node_keys document into a
500 instead of a 401.
Closeslogan/server-26#12
#17: severity was written once at _create_incident and never touched again,
so an incident that opened routine and escalated to a working fire stayed
routine forever. _update_incident now merges call_severity into the incident
via _max_severity() on every link.
Severity is monotonic: it only ever rises, never falls. An incident briefly
assessed "major" genuinely was major at that moment; a later, calmer-sounding
call is evidence the situation is winding down, not that the earlier read was
wrong. status/resolved_at exist to retire an incident — severity should stay
as the high-water mark so the worst-first rail, "Major only" filter, and map
colouring never bury a call that was genuinely major. See _max_severity's
docstring in incident_correlator.py for the full argument.
#18: none of the resolution sites wrote resolved_at, so an incident's
lifespan couldn't be reconstructed for the history-scrub feature. Added
resolved_at alongside status="resolved" at all six sites that flip it:
- incident_correlator.py _update_incident (signal-based: units all cleared)
- incident_correlator.py maybe_resolve_parent (master auto-resolve)
- summarizer.py _stale_sweep (90-minute auto-resolve)
- upload.py, both scene-resolution loops (single- and multi-scene)
- calls.py reprocess/correction path
(_update_incident's signal-resolve and maybe_resolve_parent's master-resolve
weren't named in the issue's four call sites, but they set status the same
way and were missing resolved_at too.)
No backfill: existing resolved incidents keep resolved_at = null, which
means "resolved before this field existed," not "never resolved." Backfilling
from updated_at would be a guess dressed up as data.
Tests: added to tests/test_correlator_gate.py, which needs no Firestore for
the pure _max_severity cases and patches fstore for the _update_incident/
maybe_resolve_parent writes. Covers the escalation case (routine -> major),
the no-downgrade case, and resolved_at on both the signal-resolve and
master-resolve paths. 52/52 passing in that file; 83 passed / 10
pre-existing failures for drb-c2-core overall (baseline was 69/10 — the
+14 is exactly the new tests, no regressions).
Fixes#17, #18.
The previous commit shipped Firestore rules that reference an org_id claim
nothing issues yet, and an org_id filter nothing writes yet - this is the
commit that makes both real. Backend half of SAAS_PLAN.md B2/B2b/B2c.
Data model: organizations/{org_id} and org_members/{uid} are new
collections (models.py OrganizationRecord/OrgMember). org_id is now an
Optional field on NodeRecord, SystemRecord, CallRecord, IncidentRecord,
AlertRule, and AlertEvent - optional because every existing document
predates it; scripts/backfill_org_id.py (written, not run - it touches
production Firestore and Firebase Auth claims) is what closes that gap
later. plan_id/subscription_status/stripe_* on OrganizationRecord are
deliberately None: no billing or pricing model has been decided, so this is
a seam, not a promise. app/internal/tenancy.py holds FOUNDING_ORG_ID, the
org every pre-tenancy document and every legacy enrollment path resolves
into.
Where org_id comes from, end to end: a customer's node enrolls with a
per-org token (new enrollment_tokens/{token_hash} collection, minted via
POST /org/enrollment-tokens - new routers/org.py) instead of the old
fleet-wide ENROLLMENT_TOKEN, which still works as a fallback that resolves
to FOUNDING_ORG_ID so an already-deployed node's .env doesn't start failing
today. The node's org_id then flows onto every call it produces
(mqtt_handler.py's call_start/call_end, upload.py's /upload handler all
resolve it from the node doc), and onto every incident correlated from
those calls (incident_correlator.py's _create_incident/_create_master_incident).
That last one is the part that isn't just a read filter: _build_context's
`all_active = collection_list("incidents", status="active")` fed every
correlation candidate - fast-path talkgroup match, unit-continuity,
disambiguation - from the entire incidents collection, unscoped. Without
scoping it to the call's own org_id, a call from org A could link into an
incident org B already owns, which is a cross-tenant data merge at
correlation time, not just an over-broad read. Same shape of bug in
alerter.py: rule matching pulled every enabled alert_rule regardless of
org, so org A's keyword rule could fire (and POST org A's Discord webhook)
on org B's radio traffic. Both now resolve org_id from the call doc itself
rather than threading a new parameter through every caller.
Every list/get route gained org scoping via a new resolve_caller_org_id()
helper in internal/auth.py, which handles the three credential shapes those
routes accept (service key, node api_key, Firebase user) uniformly and
returns None (unrestricted) for the service key and platform admins -
preserving today's single-org behaviour exactly while closing the leak for
everyone else: GET /nodes, /systems, /calls, /incidents, /alerts,
/alert-rules. Write routes for nodes/systems (approve, create, delete, etc.)
deliberately stay platform-admin-only for now rather than being loosened to
org-owner/operator - that's a real gap called out in SAAS_PLAN.md 2.4's
"should be" column, but it's a separate authorization redesign the 12-item
build order doesn't actually enumerate, and doing it half-considered here
risked being exactly the "half-applied filter is worse than none" failure
mode the plan warns about. Today's founding org keeps working unchanged;
loosening node/system management to org owners is follow-up work, flagged
rather than guessed at.
Also closed the four spend/access-attack routes SAAS_PLAN.md B2c called out
by file and line: POST /calls/{id}/reprocess is now admin-only (was any
signed-in viewer looping the Whisper+Gemini pipeline for free - DEFERRED.md
had this as a live, independent-of-SaaS exploit) plus a per-call rate
limiter as a second guard; POST /alerts/{id}/acknowledge now checks the
alert's org_id; GET /admin/features moved from require_firebase_token to
require_admin_token; and trips.py's four unauthenticated mutation routes
(create_trip, update_trip_tags, create_event, update_event) are now
restricted to the founding org (or the bot's service key, or a platform
admin) - trips has no org_id of its own and isn't getting one, since
[[trips-feature-intentional]] says it's an internal utility riding along on
this stack, not a tenant-scoped product surface.
New public-but-scoped seam: POST /auth/signup (routers/links.py, alongside
the existing /auth/link* routes) provisions an organizations doc and an
owner org_members doc for a just-created Firebase user, then sets their
org_id/org_role claims - idempotent, so a double-submit doesn't create two
orgs. This is the only route that turns "has a Firebase account" into "can
read anything," which is what the frontend AuthProvider no-claim guard
(next commit) is built around.
Also new: GET/PATCH /org for the organization profile (closes the disabled
"Save changes" button noted in DEFERRED.md - there was no organizations
concept to save into before this), and POST /waitlist (public, source-IP
rate-limited, not coupled to any plan or tier - the commercial model is
still an open decision per SAAS_PLAN.md section 6).
Verified: all touched files py_compile clean; c2-core pytest is 69
passed / 10 failed, matching the documented pre-existing baseline exactly
(DEFERRED.md - mqtt_handler/node_sweeper test-vs-code drift, unrelated to
this change) - no new failures. flake8 --max-line-length=120 shows no new
violations in any touched file (checked each new E501/E221/E30x against
`git diff` to confirm it predates this commit); c2-core has no CI lint gate
regardless (CLAUDE.md - flake8 only runs in Client CI).
No new environment variables. Firestore composite indexes for the queries
this introduces were already shipped in the previous commit
(infra/firestore/firestore.indexes.json).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent sources of garbage in the AI pipeline, both visible in the
2026-08-16 correlation dump.
1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run
of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt
text as preceding transcript, so on noisy or silent audio it continued the
series, emitting transcripts that count upward from 10-4 to 10-99. The
existing no_speech_prob filter could not catch these: the model is highly
confident in text it invented by continuing a pattern.
The prompt no longer contains a series to extend, and _is_degenerate()
rejects the three shapes this failure takes: ascending ten-code runs, one
phrase looping, and near-identical segments across a whole recording.
Verified against 13 transcripts from production: all four known
hallucinations rejected, all nine real ones kept, including terse traffic
containing legitimate codes.
2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both
uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one
dump. Each was transcribed, billed and correlated twice, and the resulting
incident listed two units where there was one.
Canonical selection is by earliest started_at, tie-broken on call_id, NOT
by upload order: upload order varies with encode time and network latency,
so it would make the authoritative recording non-deterministic. Call
documents are created from MQTT call_start before uploads arrive, so both
nodes independently reach the same verdict. The loser keeps its audio (it
may be the cleaner capture) but is excluded from STT, correlation, the
re-correlation sweep and the orphan debug view.
Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is
missing, where the caller unpacks two values. A missing key surfaced as a
misleading "Transcription failed" instead of the real warning.
Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an
injected callable so it stays importable without firebase-admin present.
upload_audio() could only sign a URL when GCP_CREDENTIALS_PATH pointed at a
service-account key file. The deployed VM runs on Application Default
Credentials with no key file, so every upload silently took the fallback
branch and returned a bare gs:// URI. That broke two things at once:
* Browsers cannot fetch a gs:// URI, so no recording was ever playable.
* _public_url_to_gcs_uri() only matched https://storage.googleapis.com/ and
returned None for it, so `if gcs_uri:` in the upload path was always false
and transcription never ran. Nothing was logged, which is why this looked
like an OpenAI credits problem rather than a storage one.
The fallback also interpolated the client-supplied filename instead of the
call_id-derived safe name, so the URI did not even name the object written.
Calls now store only the canonical gs:// location. A short-lived playback link
is minted per read as an HMAC over (call_id, expiry) keyed by SERVICE_KEY, and
audio is served from the private bucket by the new /media route. An <audio src>
cannot carry an Authorization header, so the link has to be the credential;
that router is therefore public with the check done inline, as enrollment.py
already does. Signing GCS URLs from the VM would have needed a
serviceAccountTokenCreator grant on its own service account — this avoids the
IAM change entirely and keeps the bucket private.
gcs_uri_for_call() reconstructs the object name from call_id, so recordings
made before this fix are reachable again without a data migration.
Frontend rows come straight from Firestore via onSnapshot and never see a
server-minted field, so CallRow fetches the link lazily on expand.
Also removes the last long-lived (1 year) signed URL and the log line that
printed it.
auth.py
secrets.compare_digest replaces == for service key comparison (timing-safe)
Added require_service_key — bot-only endpoints (trip/event join/leave)
Added require_service_key_or_admin — node commands/config (bot via service key OR dashboard admin via Firebase)
Added _RateLimiter with three shared instances: trip_chat_limiter (20/5min per user), summarize_limiter (5/10min per incident), bootstrap_limiter (2/hr per system)
nodes.py
send_command and assign_system now require require_service_key_or_admin — the Discord bot can still call them via service key, but regular Firebase users are blocked
tokens.py
add_token, flush_tokens, set_preferred_system, delete_token all require require_admin_token
Token masking changed from token[:10] + "…" + token[-4:] to "•••" + token[-4:]
systems.py
All write endpoints (create, update, delete, ai-flags, ten-codes, vocabulary writes, bootstrap) now require require_admin_token
bootstrap_vocabulary also calls bootstrap_limiter.check(system_id)
incidents.py
POST /incidents/summarize (bulk) now requires require_admin_token
POST /incidents/{id}/summarize now calls summarize_limiter.check(incident_id)
trips.py
join_trip, leave_trip, join_event, leave_event require require_service_key — only the Discord bot can set Discord attendee identity
delete_trip, delete_event require require_service_key_or_admin
trip_chat rate-limited per caller UID, history stripped to user/assistant roles only, user message truncated to 2000 chars, Maps query strings capped at 200 chars
upload.py
Rejects files larger than settings.upload_max_bytes (default 100MB) with 413
storage.py
_safe_audio_filename() derives GCS object name from call_id + allowlisted extension, completely ignoring the client-supplied filename
config.py
Added upload_max_bytes: int = 100 * 1024 * 1024
Both Dockerfiles — python:3.14-slim → python:3.12-slim
Refactor incident_correlator.py to a decision/commit split (preview_correlation
/ apply_correlation) so the rules engine and LLM can both produce decisions before
anything is written to Firestore.
Add llm_correlator.py: cheap Gemini Flash first-pass + Gemini Pro tiebreaker.
Wire _correlate_with_consensus in upload.py — rules-only fallback when key is
absent or call is thin; agreed/tiebreak consensus written to corr_debug.
- Geocoding: reject GEOMETRIC_CENTER/APPROXIMATE results — vague location strings
(regions, city centroids) were resolving to node-area coords and creating false
proximity matches that merged unrelated incidents
- Thin path: on dispatch channels with multiple active incidents, skip attachment
rather than guessing — "10-4" with 3 active incidents is genuinely ambiguous
- Short transcripts (≤5 words) now write skip_reason="transcript_too_short" to
the call doc, matching garbage transcript behavior
- upload.py no-scenes fallback now checks skip_reason before running correlation —
flagged calls (garbage, too short) no longer attach via thin path
- Update Server README to reflect current project purpose, goals, and pipeline
- Cap unit-continuity path at 20 min idle (unit_continuity_max_idle_minutes)
- Block time_fallback and unit-continuity matching on reassignment calls
- Expand reassignment detection to cover unit-initiated self-reassignment
- Skip GPT extraction entirely for transcripts ≤5 words (prevents hallucinated tags/units)
- Reduce geocode_max_km from 75 to 40 to reject far-out-of-area results
- Include county in geocoding query for tighter jurisdiction anchoring
Correlator
- Raise fast-path idle gate 30 → 90 min (tg_fast_path_idle_minutes)
- Fix disambiguate always-commits bug: run _call_fits_incident on winner
before committing; fall through to new-incident creation if it fails
- Add unit-continuity path (path 1.5): matches all_active by shared unit
IDs with a reassignment guard, bridges calls past the idle gate
- Add tag-based incident_type inference (_TAG_TYPE_HINTS) as GPT fallback,
rescuing tagged calls that would have been dropped (616 observed orphans)
- Add master/child incident model: _create_master_incident, _demote_to_child,
_add_child_to_master; new incidents stamped incident_type="master"
- Add cross-system parent detection (_find_cross_system_parent): two-signal
scoring (road overlap=0.4, embedding≥0.78=0.3, proximity=0.3, threshold=0.5)
wired into create-if-new path; creates master shell on first cross-system match
- Add maybe_resolve_parent: auto-resolves master when all children close;
called from upload pipeline (LLM closure) and summarizer stale sweep
- Add signal-based auto-resolve via units_active/units_cleared tracking:
GPT now extracts cleared_units per scene; _update_incident moves units
between active/cleared lists and resolves the incident when active empties;
stored on call doc for re-correlation sweep reuse
- Add _create_incident initialization of units_active/units_cleared fields
Re-correlation sweep
- Add corr_sweep_count + MAX_SWEEP_ATTEMPTS=3: orphans get 3 attempts
then are tombstoned as corr_path="unlinked", ending the re-sweep loop
(previously hammering each orphan 29-31 times per shift)
Intelligence extraction
- Add cleared_units to GPT prompt schema and rules
- Extract and propagate cleared_units per scene; merge across scenes;
store on call doc for re-correlation sweep
Token management
- Fix token release bug: remove release_token call on discord_connected=False
in MQTT checkin (transient Discord drops were orphaning bots mid-shift)
- Add PUT /tokens/{id}/prefer/{system_id} endpoint: lock a bot token to a
system; pass _none as system_id to clear; stored bidirectionally on both
token and system documents
- discord_join handler resolves preferred_token_id from system doc and passes
system_name in MQTT payload
### Firestore read reductions
**1. `doc_get_cached()` in `firestore.py` — new 5-min TTL cache**
One place, benefits everything. System and node config documents almost never change during a monitoring session.
**2. System doc: 4 reads → 1 per call**
| Before | After |
|---|---|
| `upload.py` — `doc_get("systems")` for ai_flags | `doc_get_cached` |
| `transcription.py` — `get_vocabulary()` → `doc_get("systems")` | cache hit |
| `intelligence.py` — `get_vocabulary()` → `doc_get("systems")` | cache hit |
| `intelligence.py` — `doc_get("systems")` again for ten_codes | eliminated (reads same cached doc) |
**3. Node doc: cached in `_on_call_start` and `intelligence.py`**
The node is read every call event to get `assigned_system_id` and lat/lon for geocoding. Both now use the cache — node assignments and positions essentially never change at runtime.
**4. Node sweeper: 30s → 90s interval**
The sweeper was doing a full node collection scan 3× more often than necessary — the offline threshold is already 90s. Cuts sweeper reads by 66%.
**5. Vocabulary induction: scans all-time calls → last 7 days**
Previously fetched every ended call for a system (could be thousands). Now scoped to the last 7 days.
> **Note:** The vocabulary induction query `(system_id == X, ended_at >= cutoff)` needs a Firestore
> composite index on `(system_id ASC, ended_at ASC)`. When the induction loop first fires it will log
> an error with a Firebase Console link to create it in one click.
- *`correlate_call`* — added units and vehicles optional params; when provided (per-scene from intelligence extraction), they take priority over the merged call-document values, preventing multi-scene unit contamination
- *Cross-TGID correlation path (2.5)* — *new path between location and slow paths*: when a call shares 2+ unit IDs with a recent same-system, same-type incident AND embedding similarity ≥ 0.85, it links them — catches multi-talkgroup pursuits like the bicycle search that split across dispatch/tactical/geographic channels
# `app/internal/intelligence.py`
- *`reassignment` field* — added to the GPT-4o-mini prompt schema and rules; `true` when dispatch is actively pulling a unit to a new, different call (not a status update or en route acknowledgement); returned in every processed scene dict
- *Tag location rule* — added explicit instruction to the prompt: tags must describe what happened, not where; place names, road names, and talkgroup names are explicitly forbidden as tags
# `app/routers/upload.py`
- Both scene correlation call sites (`_run_extraction_pipeline` and `_run_intelligence_pipeline`) now pass `units=corr_units` where `corr_units = [] if scene.get("reassignment") else scene.get("units") `— suppresses unit overlap matching when a unit is being reassigned to a new call, preventing chaining into their previous incident
- Both sites also pass `vehicles=scene.get("vehicles")` (per-scene vehicles, from the multi-scene units fix)
# `app/config.py`
- `embedding_cross_tg_threshold: float = 0.85` — threshold for the new cross-TGID path
incident_correlator.py — full rewrite: always runs on every call, fetches all active incidents cross-type, fast path collects all talkgroup matches and disambiguates by unit/vehicle overlap → location proximity → embedding, new location proximity path, slow path requires location corroboration, "Auto:" stripped from titles, "auto-generated" tag added, units/vehicles now accumulated on update
intelligence.py — resolved field in GPT schema, returned as 5th value
upload.py — both pipelines unpack 5-tuple, always call correlate, auto-resolve on resolved=True
summarizer.py — stale sweep runs each tick, resolves incidents idle for 90+ minutes
config.py — correlation_window_hours=2, embedding_similarity_threshold=0.93, location_proximity_km=0.5, incident_auto_resolve_minutes=90
Upload 404 warning doc_set(merge=True) in upload.py — creates doc if missing
MQTT call_end 404 error doc_set(merge=True) in mqtt_handler.py — same root cause
Transcription 404 (saving transcript to nonexistent doc) doc_set(merge=True) in transcription.py
Transcription ADC credentials error Explicit service_account.Credentials from gcp-key.json in _sync_transcribe — same pattern as storage.py
Logan Cusano
