Correct the transcript before anything reads it

Correction existed, but as a line in intelligence.py's EXTRACTION_PROMPT --
which put it in the wrong place twice over. The same model call that extracted
units, location and severity emitted the correction afterwards, so extraction
reasoned over text already known to be wrong; and it sat behind
correlation_enabled, so during a cost-controlled STT-only window nothing was
ever corrected at all. That is the normal state during development.

internal/transcript_correction.py is now its own pass, between the degenerate
filter and the Firestore write. It receives an already-produced transcript plus
a reference list, so unlike a Whisper prompt it has no series to extend -- the
distinction that keeps vocabulary out of the recogniser's prompt, where an
enumerated ten-code list once made it hallucinate ten-code runs.

Reference data is merged from the talkgroup and the system, TALKGROUP FIRST. A
system spanning several counties can have a talkgroup covering one
municipality, and that municipality's streets must not be buried under a
county-wide list. A single-municipality system is the degenerate case: populate
the system level and every talkgroup inherits it. Area context is now SET --
municipality, county, roads, landmarks, on both scopes -- rather than guessed
from talkgroup names, which is what vocabulary_learner did and which is close
to useless across multiple counties.

Segments are corrected too, not just the joined text. extract_scenes builds its
prompt from numbered segments whenever there is more than one, so a correction
that only fixed the transcript would have been discarded on exactly the
multi-transmission calls carrying the most content. Alignment is enforced: an
array of the wrong length or type is dropped whole, because scenes map back to
transmissions by index and a shifted array would misattribute audio silently.

Whisper is also retried once on degenerate output. Call e49ea32c produced a
56-word ten-code counting run on one attempt and ordinary speech on the next --
same clip, same temperature=0 -- so a hallucination is a coin-flip, and
discarding on the first bad roll threw away a recoverable transcript.

Two things found on the way:

PUT /systems/{id} wiped ten_codes on every save. The systems form sends only
{name, type, config}, and model_dump() wrote every omitted field as its default
over the top. Now exclude_unset. area_context would have been the next victim,
which is why it gets its own route alongside ten-codes rather than a field on
that payload.

Closes server-26#36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

drb-c2-core/app/internal/transcription.py

@@ -12,6 +12,7 @@ from typing import Optional
 from app.internal.logger import logger
 from app.internal import firestore as fstore
 from app.internal import ai_health
+from app.internal import transcript_correction
 from app.config import settings
 
 # Whisper treats `prompt` as preceding transcript text, not instructions.
@@ -154,6 +155,7 @@ async def transcribe_call(
     gcs_uri: str,
     talkgroup_name: Optional[str] = None,
     system_id: Optional[str] = None,
+    talkgroup_id: Optional[int] = None,
 ) -> tuple[Optional[str], list[dict]]:
     """
     Transcribe audio at the given GCS URI and store the result in Firestore.
@@ -166,9 +168,23 @@ async def transcribe_call(
         return None, []
 
     try:
-        transcript, segments = await asyncio.to_thread(
+        transcript, segments, degenerate = await asyncio.to_thread(
             _sync_transcribe, gcs_uri, talkgroup_name
         )
+        # A hallucination is a coin-flip, not a property of the clip: call
+        # e49ea32c produced a 56-word ten-code counting run on one attempt and
+        # ordinary speech on the next, same audio and temperature=0. Discarding
+        # on the first bad roll threw away a recoverable transcript, so spend
+        # one more request before giving up.
+        if degenerate and settings.stt_retry_on_degenerate:
+            logger.info(f"Retrying transcription for call {call_id} after degenerate output")
+            transcript, segments, degenerate = await asyncio.to_thread(
+                _sync_transcribe, gcs_uri, talkgroup_name
+            )
+            if degenerate:
+                logger.warning(
+                    f"Transcription for call {call_id} was degenerate twice — giving up"
+                )
     except Exception as e:
         await _log_transcribe_failure(call_id, e)
         return None, []
@@ -183,23 +199,63 @@ async def transcribe_call(
         updates: dict = {"transcript": transcript}
         if segments:
             updates["segments"] = segments
+
+        # Second opinion, before anything downstream sees the text. Whisper
+        # mishears proper nouns confidently, and extraction/embedding/
+        # correlation all consume the transcript — correcting it afterwards
+        # (which is where it used to live, inside the extraction prompt) meant
+        # every one of them reasoned over known-bad text. server-26#36.
+        corrected, corrected_segments, not_speech = await transcript_correction.correct(
+            call_id, transcript, segments,
+            system_id=system_id,
+            talkgroup_id=talkgroup_id,
+            talkgroup_name=talkgroup_name,
+        )
+        if corrected_segments:
+            # Raw stays as evidence; the corrected copy is what extraction reads.
+            updates["segments_corrected"] = corrected_segments
+        if not_speech:
+            # The corrector sees what _is_degenerate misses — novel repetition
+            # shapes rather than the two it pattern-matches. Keep the raw text
+            # (it is evidence) but do not let it reach extraction as fact.
+            updates["transcript_not_speech"] = True
+            logger.info(
+                f"Corrector flagged call {call_id} as recogniser noise: {transcript[:80]!r}"
+            )
+        elif corrected:
+            updates["transcript_corrected"] = corrected
+
         try:
             await fstore.doc_set("calls", call_id, updates)
             logger.info(
                 f"Transcript saved for call {call_id} "
-                f"({len(transcript)} chars, {len(segments)} segment(s))"
+                f"({len(transcript)} chars, {len(segments)} segment(s)"
+                f"{', corrected' if corrected and not not_speech else ''})"
             )
         except Exception as e:
             logger.warning(f"Could not save transcript for {call_id}: {e}")
 
+        if not_speech:
+            return None, []
+        # Hand the corrected copies downstream. extract_scenes prefers numbered
+        # segments over the joined transcript, so returning corrected text with
+        # raw segments would have thrown the correction away on every call with
+        # more than one transmission.
+        return corrected or transcript, corrected_segments or segments
+
     return transcript, segments
 
 
 def _sync_transcribe(
     gcs_uri: str,
     talkgroup_name: Optional[str] = None,
-) -> tuple[Optional[str], list[dict]]:
-    """Download audio from GCS and transcribe with OpenAI Whisper."""
+) -> tuple[Optional[str], list[dict], bool]:
+    """Download audio from GCS and transcribe with OpenAI Whisper.
+
+    Third element is True when output was DISCARDED as degenerate, which the
+    caller distinguishes from ordinary silence so it can retry — the same clip
+    can hallucinate on one attempt and transcribe on the next.
+    """
     from google.cloud import storage as gcs
     from google.oauth2 import service_account
     from openai import OpenAI
@@ -210,7 +266,7 @@ def _sync_transcribe(
         # Tuple, not a bare None: the caller unpacks two values, so returning
         # None here raised a TypeError that surfaced as a misleading
         # "Transcription failed" instead of the real missing-key warning.
-        return None, []
+        return None, [], False
 
     without_scheme = gcs_uri[len("gs://"):]
     bucket_name, blob_path = without_scheme.split("/", 1)
@@ -281,16 +337,16 @@ def _sync_transcribe(
             text = " ".join(s["text"] for s in segments) or None
             if _is_degenerate(text or "", segments):
                 logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
-                return None, []
-            return text, segments
+                return None, [], True
+            return text, segments, False
         else:
             # json format returns just {"text": "..."} — no segments or timestamps.
             # Intelligence extraction falls back to treating the whole transcript as one block.
             text = (response.text or "").strip() or None
             if _is_degenerate(text or "", []):
                 logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
-                return None, []
-            return text, []
+                return None, [], True
+            return text, [], False
     finally:
         try:
             os.unlink(tmp_path)