transcript_correction.py accepted the model's rewrite verbatim — a real ten-code got swapped for a different real ten-code #162

New Issue

2026-09-20T21:51:25-04:00

logan commented

2026-09-20 21:51:25 -04:00

Found live while reviewing the correction pass's output against raw transcripts (2026-09-21). correct()'s prompt tells the model "Do NOT expand ten-codes" and "NEVER add information" — but nothing in the code checked that. raw["corrected"] was accepted verbatim once it passed basic non-empty/changed checks; the changed field it returns is logged "for audit" and never validated against anything.

Caught on the "Shots Fired at Home Street and Forest Ave" call (talkgroup 8026): the same call's raw vs corrected transcript showed "10-7" rewritten to "10-13" in one place and to "10-4" in another, and a bare "7" expanded into "ShotSpotter". The place-name fixes alongside it were good (Holmes Street and 4th and Rowe → Home Street and Forest Ave, using this system's own vocabulary) — the corrector isn't broken in general, it just has no leash on what kind of token it's allowed to touch.

Why this is worse than it sounds for a public-safety tool: a wrong 10-13 (officer needs assistance) standing in for an actual 10-7 (out of service) reads exactly as confident and trustworthy as a correct transcript. There is no visible sign downstream that anything was invented.

Fixed in this pass: _code_tokens() extracts every ten-code/signal-shaped token (\d{1,3}(?:-\d{1,3})+ — covers "10-4", "4-2", and case-number fragments like "7-2-1") from the original and the corrected text/segments; if the set changes at all, that half of the correction is discarded and the raw text is kept instead. Independently checked for the joined text and for segments, matching the existing all-or-nothing segment-alignment rule. Tests added to tests/test_transcript_correction.py covering: a changed code, an invented code, a legitimate place-only correction that must still pass, and a segment-level code change discarding only the segments (not the independently-clean joined text).

What this does NOT fix, and can't cheaply: a wrong word substituted for another equally plausible word that isn't code-shaped (a street name changed to a different real street name, say). That class of error still relies entirely on the model following its own prompt. Tracked as part of the broader STT-quality initiative filed alongside this.

Found live while reviewing the correction pass's output against raw transcripts (2026-09-21). `correct()`'s prompt tells the model *"Do NOT expand ten-codes"* and *"NEVER add information"* — but nothing in the code checked that. `raw["corrected"]` was accepted verbatim once it passed basic non-empty/changed checks; the `changed` field it returns is logged "for audit" and never validated against anything. Caught on the "Shots Fired at Home Street and Forest Ave" call (talkgroup 8026): the same call's raw vs corrected transcript showed **"10-7" rewritten to "10-13" in one place and to "10-4" in another**, and a bare "7" expanded into "ShotSpotter". The place-name fixes alongside it were good (Holmes Street and 4th and Rowe → Home Street and Forest Ave, using this system's own vocabulary) — the corrector isn't broken in general, it just has no leash on *what kind* of token it's allowed to touch. Why this is worse than it sounds for a public-safety tool: a wrong 10-13 (officer needs assistance) standing in for an actual 10-7 (out of service) reads exactly as confident and trustworthy as a correct transcript. There is no visible sign downstream that anything was invented. Fixed in this pass: `_code_tokens()` extracts every ten-code/signal-shaped token (`\d{1,3}(?:-\d{1,3})+` — covers "10-4", "4-2", and case-number fragments like "7-2-1") from the original and the corrected text/segments; if the set changes at all, that half of the correction is discarded and the raw text is kept instead. Independently checked for the joined text and for segments, matching the existing all-or-nothing segment-alignment rule. Tests added to `tests/test_transcript_correction.py` covering: a changed code, an invented code, a legitimate place-only correction that must still pass, and a segment-level code change discarding only the segments (not the independently-clean joined text). What this does NOT fix, and can't cheaply: a wrong word substituted for another equally plausible word that isn't code-shaped (a street name changed to a different real street name, say). That class of error still relies entirely on the model following its own prompt. Tracked as part of the broader STT-quality initiative filed alongside this.

logan referenced this issue from a commit

2026-09-20 21:51:43 -04:00

transcript_correction: reject a correction that changes a ten-code (#162)

logan referenced this issue

2026-09-20 21:56:50 -04:00

STT quality initiative: raw transcription accuracy is the ceiling every downstream fix (correction, extraction, correlation) is hostage to — research + candidate angles #163

logan commented

2026-09-20 21:57:07 -04:00

Correction: the commit message for this fix (241a15b) says "Filed server-26#161 for the broader STT/audio quality initiative" — that's wrong, #161 is the unrelated FIREBASE_TOKEN/Firestore-rules CI issue. The STT initiative this fix belongs alongside is #163, filed after this commit landed. Not amending the pushed commit for a message-only error; leaving this comment as the correction.

Correction: the commit message for this fix (241a15b) says "Filed server-26#161 for the broader STT/audio quality initiative" — that's wrong, #161 is the unrelated FIREBASE_TOKEN/Firestore-rules CI issue. The STT initiative this fix belongs alongside is **#163**, filed after this commit landed. Not amending the pushed commit for a message-only error; leaving this comment as the correction.

logan added the area/transcription

labels 2026-10-05 02:04:55 -04:00

logan referenced this issue

2026-10-27 20:03:49 -04:00

Transcription: move to an OpenAI-compatible cheaper provider (Groq whisper-large-v3-turbo) with a domain vocabulary prompt #297

logan commented