transcript_correction: reject a correction that changes a ten-code (#162)
correct()'s prompt says "Do NOT expand ten-codes" and "NEVER add information", but nothing checked the model's output against its own rules -- raw["corrected"] was accepted verbatim past a non-empty/changed check, and the "changed" field it returns was logged for audit and never validated. Caught live: the same call's raw vs corrected transcript showed "10-7" rewritten to "10-13" in one place and "10-4" in another, plus a bare "7" expanded into "ShotSpotter" -- alongside genuinely good fixes (Holmes Street and 4th and Rowe -> Home Street and Forest Ave, from this system's own vocabulary). A wrong 10-13 standing in for a real 10-7 reads exactly as trustworthy as a correct transcript, which is worse than leaving the raw mishearing in place. _code_tokens() extracts every ten-code/signal-shaped token from the original and corrected text/segments; any change to that set discards the correction and falls back to raw. Checked independently for the joined text and for segments, consistent with the existing all-or-nothing segment-alignment rule. Does not catch a wrong word swapped for another equally plausible non-code word -- that class still depends entirely on the model following its own prompt. Filed server-26#161 for the broader STT/audio quality initiative this belongs alongside. Verified: 426 pass, 0 fail (4 new tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
f91d4559f3
commit
241a15b8da
@@ -211,6 +211,60 @@ async def test_model_failure_leaves_the_transcript_alone():
|
||||
assert await tc.correct("c1", "x y z w", SEGS, system_id="sys-1") == (None, None, False)
|
||||
|
||||
|
||||
# ── Code-token guard (server-26#162) ────────────────────────────────────────
|
||||
# Caught live: the same call came back with "10-7" rewritten to "10-13" in one
|
||||
# place and "10-4" in another. A real code swapped for a different real code
|
||||
# reads exactly as trustworthy as a correct one — worse than leaving the raw
|
||||
# mishearing in place, since nothing downstream can tell it happened.
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_changed_ten_code_is_discarded():
|
||||
payload = {"corrected": "10-13, we're back in town."}
|
||||
with _system(), _gemini(payload):
|
||||
text, _, _ = await tc.correct("c1", "10-7, we're back in town.", None, system_id="sys-1")
|
||||
assert text is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invented_code_token_is_discarded():
|
||||
"""Nothing code-shaped in the original — the model added one from nothing."""
|
||||
payload = {"corrected": "ShotSpotter, 10-4, group of 3 shooting outside."}
|
||||
with _system(), _gemini(payload):
|
||||
text, _, _ = await tc.correct("c1", "Seven, group of 3 shooting outside.", None, system_id="sys-1")
|
||||
assert text is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legitimate_place_correction_with_unchanged_codes_still_applies():
|
||||
"""The guard must not collateral-damage a correction that never touches
|
||||
a code token — Home/Forest for Holmes/4th-and-Rowe is exactly the kind of
|
||||
fix this pass exists to make."""
|
||||
payload = {"corrected": "10-13 coming over on Home Street and Forest Ave, 4-2."}
|
||||
with _system(), _gemini(payload):
|
||||
text, _, _ = await tc.correct(
|
||||
"c1", "10-13 coming over on Holmes Street and 4th and Rowe, 4-2.",
|
||||
None, system_id="sys-1",
|
||||
)
|
||||
assert text == "10-13 coming over on Home Street and Forest Ave, 4-2."
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_segment_code_change_discards_segments_only():
|
||||
"""A code change in one segment discards the whole segments array (same
|
||||
all-or-nothing rule as a length mismatch), but the independently-checked
|
||||
joined correction still stands if it kept its own codes intact. The
|
||||
joined `text`/`corrected` pair here is deliberately code-free — this test
|
||||
isolates the segment-level guard, not the joined-text one."""
|
||||
payload = {
|
||||
"corrected": "Show it out to Ossining, back to Route 9.",
|
||||
"segments": ["Headquarters, 10-13.", "Show it out to Ossining.", "360 north, back to Route 9."],
|
||||
}
|
||||
with _system(), _gemini(payload):
|
||||
text, segs, _ = await tc.correct("c1", "x y z w", SEGS, system_id="sys-1", talkgroup_id=9048)
|
||||
assert segs is None
|
||||
assert text == "Show it out to Ossining, back to Route 9."
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reference_data_reaches_the_prompt():
|
||||
seen = {}
|
||||
|
||||
Reference in New Issue
Block a user