Stop Whisper hallucinations and dedupe recordings across nodes
Build & Deploy / Build & push images (push) Successful in 4m0s
Build & Deploy / Deploy to VM (push) Successful in 2m29s

Two independent sources of garbage in the AI pipeline, both visible in the
2026-08-16 correlation dump.

1. Hallucinated transcripts. The Whisper prompt opened with an enumerated run
   of ten-codes: 10-4, 10-23, 10-20, 10-97 and so on. Whisper treats prompt
   text as preceding transcript, so on noisy or silent audio it continued the
   series, emitting transcripts that count upward from 10-4 to 10-99. The
   existing no_speech_prob filter could not catch these: the model is highly
   confident in text it invented by continuing a pattern.

   The prompt no longer contains a series to extend, and _is_degenerate()
   rejects the three shapes this failure takes: ascending ten-code runs, one
   phrase looping, and near-identical segments across a whole recording.
   Verified against 13 transcripts from production: all four known
   hallucinations rejected, all nine real ones kept, including terse traffic
   containing legitimate codes.

2. Duplicate recordings. node-002 and node-PI-2 both cover TG 9048 and both
   uploaded the same transmissions, ~1.1s apart. Nine pairs appeared in one
   dump. Each was transcribed, billed and correlated twice, and the resulting
   incident listed two units where there was one.

   Canonical selection is by earliest started_at, tie-broken on call_id, NOT
   by upload order: upload order varies with encode time and network latency,
   so it would make the authoritative recording non-deterministic. Call
   documents are created from MQTT call_start before uploads arrive, so both
   nodes independently reach the same verdict. The loser keeps its audio (it
   may be the cleaner capture) but is excluded from STT, correlation, the
   re-correlation sweep and the orphan debug view.

Also fixes _sync_transcribe returning a bare None when OPENAI_API_KEY is
missing, where the caller unpacks two values. A missing key surfaced as a
misleading "Transcription failed" instead of the real warning.

Adds tests/test_dedup.py (15 cases). dedup.py reaches Firestore through an
injected callable so it stays importable without firebase-admin present.
This commit is contained in:
Logan Cusano
2026-08-16 17:28:27 -04:00
parent a2cd2c57ca
commit 97013e1505
8 changed files with 378 additions and 7 deletions
+14
View File
@@ -2,6 +2,7 @@ from typing import Optional
from fastapi import APIRouter, BackgroundTasks, UploadFile, File, Form, HTTPException, Security
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from app.internal.storage import upload_audio
from app.internal import dedup
from app.internal import firestore as fstore
from app.internal.logger import logger
from app.config import settings
@@ -57,6 +58,19 @@ async def upload_call_audio(
except Exception as e:
logger.warning(f"Could not update call {call_id} with audio_gcs_uri: {e}")
# Another node in range recorded the same transmission. Keep the audio
# (it may be the cleaner capture) but don't transcribe or correlate it
# a second time — see app/internal/dedup.py.
call_doc = await fstore.doc_get("calls", call_id)
duplicate_of = await dedup.find_duplicate_of(call_doc) if call_doc else None
if duplicate_of:
await fstore.doc_set("calls", call_id, {"duplicate_of": duplicate_of})
logger.info(
f"Call {call_id} from {node_id} duplicates {duplicate_of} "
f"— audio kept, AI pipeline skipped."
)
return {"url": gcs_uri, "duplicate_of": duplicate_of}
background_tasks.add_task(
_run_intelligence_pipeline,
call_id=call_id,