Correct the transcript before anything reads it

Correction existed, but as a line in intelligence.py's EXTRACTION_PROMPT --
which put it in the wrong place twice over. The same model call that extracted
units, location and severity emitted the correction afterwards, so extraction
reasoned over text already known to be wrong; and it sat behind
correlation_enabled, so during a cost-controlled STT-only window nothing was
ever corrected at all. That is the normal state during development.

internal/transcript_correction.py is now its own pass, between the degenerate
filter and the Firestore write. It receives an already-produced transcript plus
a reference list, so unlike a Whisper prompt it has no series to extend -- the
distinction that keeps vocabulary out of the recogniser's prompt, where an
enumerated ten-code list once made it hallucinate ten-code runs.

Reference data is merged from the talkgroup and the system, TALKGROUP FIRST. A
system spanning several counties can have a talkgroup covering one
municipality, and that municipality's streets must not be buried under a
county-wide list. A single-municipality system is the degenerate case: populate
the system level and every talkgroup inherits it. Area context is now SET --
municipality, county, roads, landmarks, on both scopes -- rather than guessed
from talkgroup names, which is what vocabulary_learner did and which is close
to useless across multiple counties.

Segments are corrected too, not just the joined text. extract_scenes builds its
prompt from numbered segments whenever there is more than one, so a correction
that only fixed the transcript would have been discarded on exactly the
multi-transmission calls carrying the most content. Alignment is enforced: an
array of the wrong length or type is dropped whole, because scenes map back to
transmissions by index and a shifted array would misattribute audio silently.

Whisper is also retried once on degenerate output. Call e49ea32c produced a
56-word ten-code counting run on one attempt and ordinary speech on the next --
same clip, same temperature=0 -- so a hallucination is a coin-flip, and
discarding on the first bad roll threw away a recoverable transcript.

Two things found on the way:

PUT /systems/{id} wiped ten_codes on every save. The systems form sends only
{name, type, config}, and model_dump() wrote every omitted field as its default
over the top. Now exclude_unset. area_context would have been the next victim,
which is why it gets its own route alongside ten-codes rather than a field on
that payload.

Closes server-26#36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

drb-c2-core/app/internal/intelligence.py

@@ -49,7 +49,6 @@ Response format — a JSON object with a "scenes" array. Each scene:
   severity: one of "routine" | "minor" | "moderate" | "major"
   resolved: true if this scene explicitly signals incident closure, false otherwise
   reassignment: true if a unit is breaking from their current scene to respond to a completely different call — whether dispatch-initiated ("Baker, can you clear and respond to...", "Adam, break from that and go to...") OR unit-initiated ("Show me headed to the vehicle complaint", "Can you show me to that call", a unit going 10-8 and self-requesting a new assignment). False if the unit is reporting in on their current scene, giving a status update, or requesting information about their existing call.
-  transcript_corrected: corrected text for this scene's transmissions only, or null
 
 Rules:
 - location: prefer intersections > addresses > mile markers > route+town > route alone > town alone. Dispatch-provided addresses take priority over unit-reported positions. Empty string if none.
@@ -66,7 +65,6 @@ Rules:
 - resolved: true only when the scene explicitly signals "Code 4", "all clear", "10-42", "in custody", "patient transported", "fire out", "GOA", "negative contact", "scene clear".
 - cleared_units: only include units that explicitly stated their own back-in-service status in this recording (e.g. "Unit 7, 10-8", "Baker-1 available", "E-14 back in service", or the department ten-code for available/back-in-service listed above). Silence or absence of a unit is NOT clearance. A scene-wide Code 4 belongs in resolved=true, not here — cleared_units is for individual unit availability signals only.
 - reassignment: only true when a unit is explicitly being pulled to a completely new call or location. A unit going en route to their first dispatch is NOT a reassignment. Routine status updates, acknowledgements, and scene updates are NOT reassignments.
-- transcript_corrected: fix only clear STT/vocoder errors (e.g. "Several" → "10-4", misheard street names, garbled unit IDs). Keep all radio language as-is — do NOT decode codes into plain English. Return null if accurate.
 
 System: {system_id}
 Talkgroup: {talkgroup_name}

drb-c2-core/app/internal/transcript_correction.py

@@ -0,0 +1,277 @@
+"""
+Transcript correction — the second opinion on what was said.
+
+Whisper hears a P25 vocoder through a narrowband channel and guesses at proper
+nouns it has no reason to know: street names, business names, unit call signs.
+It guesses confidently, so the output reads like speech and is wrong in exactly
+the places that matter downstream — "Cool Parts, Illinois" and "Shout out to
+Optum" both became incident locations.
+
+Correction used to be a line in intelligence.py's EXTRACTION_PROMPT, which put
+it in the wrong place twice over (server-26#36): the same model call that
+extracted units, location and severity emitted the correction *afterwards*, so
+extraction reasoned over uncorrected text; and it sat behind
+`correlation_enabled`, so during a cost-controlled STT-only window nothing was
+ever corrected at all. It belongs here, between transcription and everything
+that consumes a transcript.
+
+WHY A SEPARATE PASS AND NOT A WHISPER PROMPT: Whisper treats its prompt as
+preceding transcript text and will happily continue a pattern it finds there —
+an enumerated ten-code prompt made it emit "10-4. 10-5. 10-6. …" over silence
+(see transcription.py). Vocabulary can never be a transcription prior. A
+corrector that receives an already-produced transcript plus a reference list has
+no series to extend; it can only substitute what it was given.
+
+SCOPE RESOLUTION: reference data is merged from the talkgroup and the system,
+**talkgroup first**. The specific beats the general — a system spanning several
+counties may have one talkgroup covering a single municipality, and that
+municipality's streets must not be buried under a county-wide list. A
+single-municipality system is the degenerate case: populate the system level and
+every talkgroup inherits it.
+"""
+
+import asyncio
+import json
+from typing import Any, Optional
+
+from app.config import settings
+from app.internal import firestore as fstore
+from app.internal.logger import logger
+
+# A transcript this short has no proper nouns to get wrong — "10-4.", "6-2,
+# stand by." — and 9 of 29 calls in the 2026-08-23 sample sat at or under this.
+# Skipping them is most of the cost saving for none of the value.
+MIN_WORDS_FOR_CORRECTION = 4
+
+_PROMPT = """You are correcting a police/fire radio transcript produced by an automatic speech recogniser.
+
+The recogniser hears a low-bitrate vocoded radio channel. It reliably mishears proper nouns — street names, business names, town names, unit call signs — and substitutes common words that sound similar. Your job is to put back what was almost certainly said.
+
+{context_block}
+Rules:
+- Change ONLY what is likely a mishearing. If a phrase is already plausible radio traffic, leave it exactly as it is.
+- Prefer a name from the reference lists above when the transcript contains something that sounds like it. That is the entire point of this pass.
+- NEVER add information. No new sentences, no invented units, no addresses that are not implied by the audio's own words.
+- Keep radio language as radio language. Do NOT expand ten-codes or signals into plain English: "10-4" stays "10-4".
+- Keep the speaker's structure and order. This is not a rewrite or a summary.
+- If the text is clearly not speech at all — a counting run like "10-11. 10-12. 10-13.", or one phrase repeating many times over static — set not_speech to true.
+
+Return JSON:
+  corrected:  the corrected transcript, or null if nothing needed changing
+  segments:   REQUIRED when numbered transmissions are given below — the corrected
+              text for each one, as an array of exactly the same length and order.
+              Never merge, split, reorder or drop a transmission; an unchanged one
+              is returned verbatim. Omit this field entirely when no transmissions
+              are numbered.
+  not_speech: true if this is recogniser noise rather than a transmission
+  changed:    list of ["heard" -> "corrected"] pairs you applied, for audit
+
+{transcript}"""
+
+
+def _render_input(text: str, segments: Optional[list[dict]]) -> str:
+    """Numbered transmissions when we have them, so corrections stay aligned."""
+    if segments and len(segments) > 1:
+        lines = [f"{i + 1}. {s.get('text', '')}" for i, s in enumerate(segments)]
+        body = "\n".join(lines)
+        return f"Transmissions ({len(segments)}):\n{body}"
+    return f"Transcript:\n{text}"
+
+
+def _dedupe(items: list[str]) -> list[str]:
+    """Preserve order, drop case-insensitive duplicates."""
+    seen: set[str] = set()
+    out: list[str] = []
+    for item in items:
+        key = (item or "").strip().lower()
+        if key and key not in seen:
+            seen.add(key)
+            out.append(item.strip())
+    return out
+
+
+def _talkgroup_entry(system_doc: dict, talkgroup_id: Optional[int]) -> dict:
+    """The config.talkgroups[] entry for this talkgroup, or {}."""
+    if talkgroup_id is None:
+        return {}
+    try:
+        wanted = int(talkgroup_id)
+    except (TypeError, ValueError):
+        return {}
+    for tg in (system_doc.get("config") or {}).get("talkgroups", []) or []:
+        try:
+            if int(tg.get("id", -1)) == wanted:
+                return tg
+        except (TypeError, ValueError):
+            continue
+    return {}
+
+
+def _area_lines(area: dict, label: str) -> list[str]:
+    """Render an area_context dict as prompt lines. Empty when nothing is set."""
+    if not area:
+        return []
+    parts: list[str] = []
+    for key, heading in (
+        ("municipality", "Municipality"),
+        ("county", "County"),
+        ("roads", "Roads and highways"),
+        ("landmarks", "Landmarks, businesses and facilities"),
+    ):
+        value = area.get(key)
+        if not value:
+            continue
+        if isinstance(value, list):
+            value = ", ".join(str(v) for v in value if v)
+        if value:
+            parts.append(f"  {heading}: {value}")
+    return [f"{label}:"] + parts if parts else []
+
+
+async def resolve_context(system_id: Optional[str], talkgroup_id: Optional[int]) -> dict:
+    """
+    Merge the reference data a corrector needs, talkgroup ahead of system.
+
+    Returns {"vocabulary": [...], "ten_codes": {...}, "area_lines": [...]}.
+    Empty everywhere is legitimate — a system nobody has configured yet.
+    """
+    empty: dict[str, Any] = {"vocabulary": [], "ten_codes": {}, "area_lines": []}
+    if not system_id:
+        return empty
+
+    system_doc = await fstore.doc_get_cached("systems", system_id)
+    if not system_doc:
+        return empty
+
+    tg = _talkgroup_entry(system_doc, talkgroup_id)
+
+    # Talkgroup terms first so they survive any downstream truncation.
+    vocabulary = _dedupe(
+        list(tg.get("vocabulary") or []) + list(system_doc.get("vocabulary") or [])
+    )
+
+    # Ten-codes: system-wide reference, with talkgroup entries overriding a
+    # code that means something different on this channel.
+    ten_codes = dict(system_doc.get("ten_codes") or {})
+    ten_codes.update(tg.get("ten_codes") or {})
+
+    area_lines = (
+        _area_lines(tg.get("area_context") or {}, "Area covered by this talkgroup")
+        + _area_lines(system_doc.get("area_context") or {}, "Area covered by this system")
+    )
+
+    return {"vocabulary": vocabulary, "ten_codes": ten_codes, "area_lines": area_lines}
+
+
+def build_context_block(context: dict, talkgroup_name: Optional[str]) -> str:
+    """Render resolved context into the prompt's reference section."""
+    lines: list[str] = []
+    if talkgroup_name:
+        lines.append(f"Channel: {talkgroup_name}")
+    lines.extend(context.get("area_lines") or [])
+    vocabulary = context.get("vocabulary") or []
+    if vocabulary:
+        lines.append("Known local names and terms: " + ", ".join(vocabulary))
+    ten_codes = context.get("ten_codes") or {}
+    if ten_codes:
+        rendered = ", ".join(f"{code}={meaning}" for code, meaning in sorted(ten_codes.items()))
+        lines.append(f"Ten-codes used on this system: {rendered}")
+    return ("\n".join(lines) + "\n") if lines else ""
+
+
+def _sync_gemini(model_name: str, prompt: str) -> dict:
+    import google.generativeai as genai  # lazy import — only when needed
+
+    genai.configure(api_key=settings.gemini_api_key)
+    model = genai.GenerativeModel(
+        model_name,
+        generation_config={"response_mime_type": "application/json"},
+    )
+    return json.loads(model.generate_content(prompt).text)
+
+
+async def correct(
+    call_id: str,
+    text: str,
+    segments: Optional[list[dict]] = None,
+    system_id: Optional[str] = None,
+    talkgroup_id: Optional[int] = None,
+    talkgroup_name: Optional[str] = None,
+) -> tuple[Optional[str], Optional[list[dict]], bool]:
+    """
+    Second-opinion pass over a transcript.
+
+    Returns (corrected_text, corrected_segments, not_speech). ``None`` for
+    either correction means "no change" — the corrector found nothing to fix,
+    could not run, or returned segments that did not line up. Callers keep the
+    original in that case; correction is an improvement, never a dependency.
+
+    Segments matter as much as the joined text: intelligence.py builds its
+    extraction prompt from NUMBERED SEGMENTS whenever there is more than one,
+    so a correction that only fixed the joined transcript would never reach the
+    model on exactly the multi-transmission calls that carry the most content.
+    """
+    if not settings.gemini_api_key or not settings.transcript_correction_enabled:
+        return None, None, False
+
+    if len((text or "").split()) < MIN_WORDS_FOR_CORRECTION:
+        return None, None, False
+
+    context = await resolve_context(system_id, talkgroup_id)
+    prompt = _PROMPT.format(
+        context_block=build_context_block(context, talkgroup_name),
+        transcript=_render_input(text, segments),
+    )
+
+    try:
+        raw = await asyncio.to_thread(
+            _sync_gemini, settings.transcript_correction_model, prompt
+        )
+    except Exception as e:
+        # Never fail the transcript over a failed correction — the raw text is
+        # still worth having. ai_health reporting is the caller's business.
+        logger.warning(f"Transcript correction failed for call {call_id}: {e}")
+        return None, None, False
+
+    not_speech = bool(raw.get("not_speech"))
+
+    corrected = raw.get("corrected")
+    if not isinstance(corrected, str) or not corrected.strip():
+        corrected = None
+    elif corrected.strip() == (text or "").strip():
+        corrected = None
+
+    # Segment alignment is non-negotiable: scene extraction maps scenes back to
+    # transmissions by INDEX (segment_indices), so a returned array of the wrong
+    # length would silently attribute the wrong audio to a scene. Wrong length,
+    # wrong type, or any non-string entry and the segments are discarded whole —
+    # the joined correction still stands.
+    corrected_segments: Optional[list[dict]] = None
+    if segments and len(segments) > 1:
+        returned = raw.get("segments")
+        if (
+            isinstance(returned, list)
+            and len(returned) == len(segments)
+            and all(isinstance(x, str) for x in returned)
+        ):
+            corrected_segments = [
+                {**seg, "text": new.strip() or seg.get("text", "")}
+                for seg, new in zip(segments, returned)
+            ]
+            if all(s["text"] == o.get("text") for s, o in zip(corrected_segments, segments)):
+                corrected_segments = None
+        elif returned is not None:
+            logger.warning(
+                f"Transcript correction for call {call_id} returned "
+                f"{len(returned) if isinstance(returned, list) else type(returned).__name__} "
+                f"segment(s) against {len(segments)} — discarding segment corrections"
+            )
+
+    if corrected or corrected_segments or not_speech:
+        changed = raw.get("changed") or []
+        logger.info(
+            f"Transcript correction ({settings.transcript_correction_model}): call {call_id} "
+            f"not_speech={not_speech} segments={'yes' if corrected_segments else 'no'} "
+            f"changes={changed if isinstance(changed, list) else '?'}"
+        )
+    return corrected, corrected_segments, not_speech

drb-c2-core/app/internal/transcription.py

@@ -12,6 +12,7 @@ from typing import Optional
 from app.internal.logger import logger
 from app.internal import firestore as fstore
 from app.internal import ai_health
+from app.internal import transcript_correction
 from app.config import settings
 
 # Whisper treats `prompt` as preceding transcript text, not instructions.
@@ -154,6 +155,7 @@ async def transcribe_call(
     gcs_uri: str,
     talkgroup_name: Optional[str] = None,
     system_id: Optional[str] = None,
+    talkgroup_id: Optional[int] = None,
 ) -> tuple[Optional[str], list[dict]]:
     """
     Transcribe audio at the given GCS URI and store the result in Firestore.
@@ -166,9 +168,23 @@ async def transcribe_call(
         return None, []
 
     try:
-        transcript, segments = await asyncio.to_thread(
+        transcript, segments, degenerate = await asyncio.to_thread(
             _sync_transcribe, gcs_uri, talkgroup_name
         )
+        # A hallucination is a coin-flip, not a property of the clip: call
+        # e49ea32c produced a 56-word ten-code counting run on one attempt and
+        # ordinary speech on the next, same audio and temperature=0. Discarding
+        # on the first bad roll threw away a recoverable transcript, so spend
+        # one more request before giving up.
+        if degenerate and settings.stt_retry_on_degenerate:
+            logger.info(f"Retrying transcription for call {call_id} after degenerate output")
+            transcript, segments, degenerate = await asyncio.to_thread(
+                _sync_transcribe, gcs_uri, talkgroup_name
+            )
+            if degenerate:
+                logger.warning(
+                    f"Transcription for call {call_id} was degenerate twice — giving up"
+                )
     except Exception as e:
         await _log_transcribe_failure(call_id, e)
         return None, []
@@ -183,23 +199,63 @@ async def transcribe_call(
         updates: dict = {"transcript": transcript}
         if segments:
             updates["segments"] = segments
+
+        # Second opinion, before anything downstream sees the text. Whisper
+        # mishears proper nouns confidently, and extraction/embedding/
+        # correlation all consume the transcript — correcting it afterwards
+        # (which is where it used to live, inside the extraction prompt) meant
+        # every one of them reasoned over known-bad text. server-26#36.
+        corrected, corrected_segments, not_speech = await transcript_correction.correct(
+            call_id, transcript, segments,
+            system_id=system_id,
+            talkgroup_id=talkgroup_id,
+            talkgroup_name=talkgroup_name,
+        )
+        if corrected_segments:
+            # Raw stays as evidence; the corrected copy is what extraction reads.
+            updates["segments_corrected"] = corrected_segments
+        if not_speech:
+            # The corrector sees what _is_degenerate misses — novel repetition
+            # shapes rather than the two it pattern-matches. Keep the raw text
+            # (it is evidence) but do not let it reach extraction as fact.
+            updates["transcript_not_speech"] = True
+            logger.info(
+                f"Corrector flagged call {call_id} as recogniser noise: {transcript[:80]!r}"
+            )
+        elif corrected:
+            updates["transcript_corrected"] = corrected
+
         try:
             await fstore.doc_set("calls", call_id, updates)
             logger.info(
                 f"Transcript saved for call {call_id} "
-                f"({len(transcript)} chars, {len(segments)} segment(s))"
+                f"({len(transcript)} chars, {len(segments)} segment(s)"
+                f"{', corrected' if corrected and not not_speech else ''})"
             )
         except Exception as e:
             logger.warning(f"Could not save transcript for {call_id}: {e}")
 
+        if not_speech:
+            return None, []
+        # Hand the corrected copies downstream. extract_scenes prefers numbered
+        # segments over the joined transcript, so returning corrected text with
+        # raw segments would have thrown the correction away on every call with
+        # more than one transmission.
+        return corrected or transcript, corrected_segments or segments
+
     return transcript, segments
 
 
 def _sync_transcribe(
     gcs_uri: str,
     talkgroup_name: Optional[str] = None,
-) -> tuple[Optional[str], list[dict]]:
-    """Download audio from GCS and transcribe with OpenAI Whisper."""
+) -> tuple[Optional[str], list[dict], bool]:
+    """Download audio from GCS and transcribe with OpenAI Whisper.
+
+    Third element is True when output was DISCARDED as degenerate, which the
+    caller distinguishes from ordinary silence so it can retry — the same clip
+    can hallucinate on one attempt and transcribe on the next.
+    """
     from google.cloud import storage as gcs
     from google.oauth2 import service_account
     from openai import OpenAI
@@ -210,7 +266,7 @@ def _sync_transcribe(
         # Tuple, not a bare None: the caller unpacks two values, so returning
         # None here raised a TypeError that surfaced as a misleading
         # "Transcription failed" instead of the real missing-key warning.
-        return None, []
+        return None, [], False
 
     without_scheme = gcs_uri[len("gs://"):]
     bucket_name, blob_path = without_scheme.split("/", 1)
@@ -281,16 +337,16 @@ def _sync_transcribe(
             text = " ".join(s["text"] for s in segments) or None
             if _is_degenerate(text or "", segments):
                 logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
-                return None, []
-            return text, segments
+                return None, [], True
+            return text, segments, False
         else:
             # json format returns just {"text": "..."} — no segments or timestamps.
             # Intelligence extraction falls back to treating the whole transcript as one block.
             text = (response.text or "").strip() or None
             if _is_degenerate(text or "", []):
                 logger.info(f"Discarded hallucinated transcript for {gcs_uri}: {(text or '')[:80]!r}")
-                return None, []
-            return text, []
+                return None, [], True
+            return text, [], False
     finally:
         try:
             os.unlink(tmp_path)