transcript_correction: reject a correction that changes a ten-code (#162)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy Firestore rules & indexes (push) Failing after 3s
Build & Deploy / Deploy to VM (push) Successful in 2m1s
Build & Deploy / Report a failed deploy (push) Successful in 1s

correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add
information", but nothing checked the model's output against its own
rules -- raw["corrected"] was accepted verbatim past a non-empty/changed
check, and the "changed" field it returns was logged for audit and never
validated.

Caught live: the same call's raw vs corrected transcript showed "10-7"
rewritten to "10-13" in one place and "10-4" in another, plus a bare "7"
expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes
Street and 4th and Rowe -> Home Street and Forest Ave, from this system's
own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly
as trustworthy as a correct transcript, which is worse than leaving the
raw mishearing in place.

_code_tokens() extracts every ten-code/signal-shaped token from the
original and corrected text/segments; any change to that set discards
the correction and falls back to raw. Checked independently for the
joined text and for segments, consistent with the existing
all-or-nothing segment-alignment rule.

Does not catch a wrong word swapped for another equally plausible
non-code word -- that class still depends entirely on the model
following its own prompt. Filed server-26#161 for the broader STT/audio
quality initiative this belongs alongside.

Verified: 426 pass, 0 fail (4 new tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-09-20 21:51:41 -04:00
co-authored by Claude Sonnet 5
parent f91d4559f3
commit 241a15b8da
2 changed files with 105 additions and 0 deletions
@@ -211,6 +211,60 @@ async def test_model_failure_leaves_the_transcript_alone():
assert await tc.correct("c1", "x y z w", SEGS, system_id="sys-1") == (None, None, False)
# ── Code-token guard (server-26#162) ────────────────────────────────────────
# Caught live: the same call came back with "10-7" rewritten to "10-13" in one
# place and "10-4" in another. A real code swapped for a different real code
# reads exactly as trustworthy as a correct one — worse than leaving the raw
# mishearing in place, since nothing downstream can tell it happened.
@pytest.mark.asyncio
async def test_changed_ten_code_is_discarded():
payload = {"corrected": "10-13, we're back in town."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "10-7, we're back in town.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_invented_code_token_is_discarded():
"""Nothing code-shaped in the original — the model added one from nothing."""
payload = {"corrected": "ShotSpotter, 10-4, group of 3 shooting outside."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct("c1", "Seven, group of 3 shooting outside.", None, system_id="sys-1")
assert text is None
@pytest.mark.asyncio
async def test_legitimate_place_correction_with_unchanged_codes_still_applies():
"""The guard must not collateral-damage a correction that never touches
a code token — Home/Forest for Holmes/4th-and-Rowe is exactly the kind of
fix this pass exists to make."""
payload = {"corrected": "10-13 coming over on Home Street and Forest Ave, 4-2."}
with _system(), _gemini(payload):
text, _, _ = await tc.correct(
"c1", "10-13 coming over on Holmes Street and 4th and Rowe, 4-2.",
None, system_id="sys-1",
)
assert text == "10-13 coming over on Home Street and Forest Ave, 4-2."
@pytest.mark.asyncio
async def test_segment_code_change_discards_segments_only():
"""A code change in one segment discards the whole segments array (same
all-or-nothing rule as a length mismatch), but the independently-checked
joined correction still stands if it kept its own codes intact. The
joined `text`/`corrected` pair here is deliberately code-free — this test
isolates the segment-level guard, not the joined-text one."""
payload = {
"corrected": "Show it out to Ossining, back to Route 9.",
"segments": ["Headquarters, 10-13.", "Show it out to Ossining.", "360 north, back to Route 9."],
}
with _system(), _gemini(payload):
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1", talkgroup_id=9048)
assert segs is None
assert text == "Show it out to Ossining, back to Route 9."
@pytest.mark.asyncio
async def test_reference_data_reaches_the_prompt():
seen = {}