transcript_correction: reject a correction that changes a ten-code (#162)
correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add information", but nothing checked the model's output against its own rules -- raw["corrected"] was accepted verbatim past a non-empty/changed check, and the "changed" field it returns was logged for audit and never validated. Caught live: the same call's raw vs corrected transcript showed "10-7" rewritten to "10-13" in one place and "10-4" in another, plus a bare "7" expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes Street and 4th and Rowe -> Home Street and Forest Ave, from this system's own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly as trustworthy as a correct transcript, which is worse than leaving the raw mishearing in place. _code_tokens() extracts every ten-code/signal-shaped token from the original and corrected text/segments; any change to that set discards the correction and falls back to raw. Checked independently for the joined text and for segments, consistent with the existing all-or-nothing segment-alignment rule. Does not catch a wrong word swapped for another equally plausible non-code word -- that class still depends entirely on the model following its own prompt. Filed server-26#161 for the broader STT/audio quality initiative this belongs alongside. Verified: 426 pass, 0 fail (4 new tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
f91d4559f3
commit
241a15b8da
@@ -28,10 +28,23 @@ counties may have one talkgroup covering a single municipality, and that
|
||||
municipality's streets must not be buried under a county-wide list. A
|
||||
single-municipality system is the degenerate case: populate the system level and
|
||||
every talkgroup inherits it.
|
||||
|
||||
THE PROMPT'S OWN RULES ARE NOT ENFORCED (server-26#162). "Do NOT expand
|
||||
ten-codes" and "NEVER add information" are instructions to the model, not
|
||||
checks on its output — `correct()` used to accept `raw["corrected"]` verbatim.
|
||||
Caught live: the same call came back with "10-7" rewritten to "10-13" in one
|
||||
place and "10-4" in another, and "7" expanded into "ShotSpotter" — a real code
|
||||
swapped for a different real code reads exactly as confident and trustworthy
|
||||
as a correct one, which is worse than leaving the raw mishearing in place. The
|
||||
model isn't graded on this at write time; `_code_tokens()` is a
|
||||
verify-what-you-can-cheaply-check backstop, not a fix to the model's judgment:
|
||||
it only catches a code-shaped token changing, not a wrong word substituted for
|
||||
another equally plausible word.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Optional
|
||||
|
||||
from app.config import settings
|
||||
@@ -96,6 +109,19 @@ def _dedupe(items: list[str]) -> list[str]:
|
||||
return out
|
||||
|
||||
|
||||
# Ten-codes ("10-4"), unit/signal shorthand ("4-2"), and the digit-group
|
||||
# fragments radio traffic reads out loud ("7-2-1" of a case number) all share
|
||||
# this shape. The guard below does not need to know which of those a given
|
||||
# token is — it only needs the SET of them to survive a "correction"
|
||||
# unchanged, in order. A model rewriting "10-7" as "10-13" is not the kind of
|
||||
# mishearing this pass exists to fix (server-26#162).
|
||||
_CODE_TOKEN_RE = re.compile(r"\b\d{1,3}(?:-\d{1,3})+\b")
|
||||
|
||||
|
||||
def _code_tokens(text: str) -> list[str]:
|
||||
return _CODE_TOKEN_RE.findall(text or "")
|
||||
|
||||
|
||||
def _talkgroup_entry(system_doc: dict, talkgroup_id: Optional[int]) -> dict:
|
||||
"""The config.talkgroups[] entry for this talkgroup, or {}."""
|
||||
if talkgroup_id is None:
|
||||
@@ -317,6 +343,31 @@ async def correct(
|
||||
if verified_segments:
|
||||
corrected_segments = verified_segments
|
||||
|
||||
# server-26#162: a code-shaped token ("10-7", "4-2", a case-number
|
||||
# fragment like "7-2-1") changing at all — not just going missing, any
|
||||
# change — means the model touched something this pass has no business
|
||||
# touching. Reject that half of the correction outright rather than trust
|
||||
# a rewrite that already broke its own instructions once. Checked against
|
||||
# the ORIGINAL text/segment, not each other, so a joined-text correction
|
||||
# and a segment correction are judged independently, same as everywhere
|
||||
# else in this function.
|
||||
if corrected is not None and _code_tokens(corrected) != _code_tokens(text):
|
||||
logger.warning(
|
||||
f"Transcript correction for call {call_id} changed code-shaped "
|
||||
f"tokens ({_code_tokens(text)} -> {_code_tokens(corrected)}) — "
|
||||
f"discarding the joined correction"
|
||||
)
|
||||
corrected = None
|
||||
if corrected_segments is not None:
|
||||
for seg, orig in zip(corrected_segments, segments or []):
|
||||
if _code_tokens(seg["text"]) != _code_tokens(orig.get("text", "")):
|
||||
logger.warning(
|
||||
f"Transcript correction for call {call_id} changed "
|
||||
f"code-shaped tokens in a segment — discarding segment corrections"
|
||||
)
|
||||
corrected_segments = None
|
||||
break
|
||||
|
||||
if corrected or corrected_segments or not_speech:
|
||||
changed = raw.get("changed") or []
|
||||
logger.info(
|
||||
|
||||
Reference in New Issue
Block a user