STT quality initiative: raw transcription accuracy is the ceiling every downstream fix (correction, extraction, correlation) is hostage to — research + candidate angles #163

New Issue

2026-09-20T21:56:50-04:00

logan commented

2026-09-20 21:56:50 -04:00

Prompted by server-26#162 (transcript_correction.py silently swapping ten-codes) and the underlying observation that drove it: raw Whisper (whisper-1) output on this project's P25 audio contains real, non-code mishearings — "Holmes Street and 4th and Rowe" for "Home Street and Forest Ave", "Oklahoma City" for "10-4 Central", "That was a heavy centrifuge" (no clear real referent), "Someone's wearing an alleyway" — that a post-hoc correction pass can only partially and riskily fix. #162 made the correction pass safer; it does not make the underlying transcription more accurate. This issue is the research pass on the actual STT layer, not the correction layer.

Current implementation (drb-c2-core/app/internal/transcription.py): OpenAI hosted whisper-1, language="en", temperature=0, a fixed generic priming prompt (deliberately excludes real per-system vocabulary — documented reason: whisper-1 treats the prompt as a transcription prior and echoes vocabulary into silence/noise, which is how the "10-4. 10-5. 10-6…" hallucination bug happened previously). verbose_json gives per-segment no_speech_prob, which _is_degenerate()/the silence filter both depend on for hallucination rejection. Audio reaching Whisper is already lossless FLAC at capture's native 22050 Hz (deliberately not pre-resampled — correct, already reasoned through in call_recorder.py's comments) — this part of the pipeline is already solid, not a target for rework.

The floor under all of this: P25 uses the IMBE vocoder (4400 bps voice + 2800 bps FEC). Published vocoder research is blunt about it — intelligibility "starts to degrade" as SNR approaches 0 dB and "degrades significantly" below it. No STT model, ours or anyone's, recovers information the vocoder itself never encoded. Some of what reads as "bad STT" may be weak-signal reception at the antenna, not a model problem — worth ruling in or out before spending on the model side (see Angle 5).

Angle 1 — Switch hosted model: `gpt-4o-transcribe` / `gpt-4o-mini-transcribe`

OpenAI's own published benchmark (Robust Speech Bench, noisy conditions): gpt-4o-mini-transcribe 0.046 WER, gpt-4o-transcribe 0.047 WER, vs whisper-1 0.053 WER. Real, quantified, and already supported in config.py's stt_model enum — but not a drop-in swap: verbose_json (segment timestamps + no_speech_prob) is whisper-1-only per OpenAI's API (already noted in a comment in transcription.py). Switching means _is_degenerate() and the leading/trailing-silence hallucination filters lose their input entirely and need an equivalent built for plain json output, or the switch trades one failure mode (mishearings) for another (unfiltered hallucinations). Also untested: whether these newer models have the same "prompt-as-continuation" hallucination risk that made this project exclude real vocabulary from the Whisper prompt — they're a different underlying architecture (GPT-4o audio, not Whisper's encoder-decoder) and may not inherit that failure mode, which would reopen vocabulary-primed prompting as an option. Needs an A/B eval on real saved DRB call audio before committing, not just trusting the published benchmark (which isn't P25-specific).

Angle 2 — Self-hosted, larger Whisper (large-v3 / large-v3-turbo)

The closest real prior art found (a hobbyist project transcribing P25 police scanner audio directly, not through this codebase) used local Whisper large-v3, then large-v3-turbo for speed — not the hosted whisper-1 API, which is roughly large-v2-class. Their explicit finding: street names were still the main error class even on the bigger model ("would be nice to fine-tune the model with the streets of my city" — never implemented). Self-hosting removes per-minute OpenAI cost and may raise the accuracy ceiling further than Angle 1, but is a real infra commitment (GPU capacity, a service to run and keep up, deploy/ops ownership) — this is a CIO-scale decision, not a code change, and shouldn't be started without CIO sign-off on the infra cost/ops tradeoff.

Angle 3 — In-model vocabulary/keyword boosting (Deepgram Nova-3, AssemblyAI, others)

This is the angle that most directly targets the root cause of #162's whole incident: Whisper has no safe way to bias toward known local vocabulary (the project already tried and rejected prompt-based priming for exactly this reason), which pushed all vocabulary correction into a post-hoc LLM rewrite pass that turned out to be unconstrained enough to swap real ten-codes. Deepgram Nova-3's "Keyterm Prompting" is built into the model's decoding, not a prompt hack: up to 100 terms per request, the model biases decoding toward them "when the acoustic evidence supports them" (i.e., it isn't supposed to have Whisper's continue-the-pattern failure mode), with a vendor-published "up to 625% uplift" in correct recognition for boosted terms, at roughly $0.26/hr batch pricing. Feeding it this system's own local_knowledge/vocabulary/unit-callsign list as keyterms could shrink transcription-time mishearings directly, reducing how much weight the correction pass has to carry (and how much room it has to do damage). AssemblyAI has an equivalent word-boost feature.
Real, load-bearing caveat: published research on P25 specifically found AWS Transcribe and Azure Speech both produced "largely gibberish" on this exact audio type — general-purpose ASR vendors do not uniformly handle vocoded radio well, and none of the vendor benchmarks above were measured on P25 audio. Deepgram/AssemblyAI's fit for this project is unverified and must be tested on real saved DRB call samples before any commitment, not assumed from generic noisy-office/phone-call benchmarks.

Angle 4 — Audio preprocessing (denoise, gain normalization) before STT

Mixed evidence, explicitly NOT a free win: one recent study (SAM-Audio preprocessing vs raw noisy speech) found denoising consistently increased WER/CER despite improving signal-level/perceptual quality — a generic denoiser can remove information an ASR model was actually using. If this is pursued at all, it needs controlled A/B testing on DRB's own vocoded audio (denoised vs raw, same clips, compare transcripts against a human-verified reference), not a blind pipeline addition.

Angle 5 — The RF/decode side ("better radio audio" upstream of any software fix)

P25/IMBE intelligibility falls off a cliff below 0 dB SNR. That's a receive-chain property: SDR antenna placement/gain, OP25 discriminator tap and squelch settings, distance/terrain to the transmitter. No STT model change fixes audio that was already unintelligible at the antenna. Worth a dedicated, separate look at node-002's current OP25 gain/squelch config and whether weak-signal talkgroups (vs strong ones) correlate with the worst transcripts — if they do, the highest-leverage fix might be a fleet/RF change, not a model change. Out of scope for this issue to resolve; flagging so it isn't missed while everyone's attention is on the software side.

Angle 6 — Already solid, do not rework

FLAC lossless capture at the native sample rate (fixed from a prior lossy-MP3 mistake, well-reasoned in call_recorder.py's own comments), the single resample happening inside Whisper rather than client-side, language="en" + temperature=0, and the existing _is_degenerate()/no_speech_prob hallucination filtering are all sound. Nothing here should be touched as part of this initiative.

Recommended sequencing (not a ruling — flagging for a CTO/board call given the cost and infra tradeoffs across angles 1–3)

Build a small held-out eval set: N real DRB calls with a human-verified reference transcript (this doesn't exist today as far as I could find — confirm before assuming one does).
A/B Angle 1 (gpt-4o-transcribe/mini) against current whisper-1 on that set, including designing the non-verbose_json hallucination-filter replacement it would need.
A/B Angle 3 (Deepgram Nova-3 keyterm prompting, seeded from a system's real vocabulary/local_knowledge) on the same set — this is the one most likely to reduce reliance on the correction pass that #162 just had to guard.
Only after 2–3 have real numbers, decide whether Angle 2 (self-hosting) is worth the infra commitment.
Separately, sanity-check Angle 5 against real signal-strength data — cheap to check, could reframe the whole priority order if weak signal turns out to be the dominant driver on the worst-performing talkgroups.

Related: #36 (per-talkgroup transcript correction — appears already implemented, flagged separately) and #6/#159 (geocode_max_km rejecting legitimate far-talkgroup results — fixed, same underlying "no anchor / no reference point" family of gap this project keeps hitting on the AI pipeline's edges).

Sources consulted (web research, 2026-09-21, general ASR benchmarks — none P25-specific except where noted):

OpenAI gpt-4o-transcribe/whisper-1 Robust Speech Bench WER comparison
Deepgram Nova-3 Keyterm Prompting docs and pricing
AssemblyAI custom vocabulary / word boost
P25 IMBE vocoder SNR-intelligibility relationship (patent/technical literature)
A hobbyist P25 police-scanner transcription project's model choice and street-name error findings (self-hosted Whisper large-v3/large-v3-turbo)
Denoising-before-ASR WER impact research (positive and negative results both found — no universal answer)

Prompted by server-26#162 (transcript_correction.py silently swapping ten-codes) and the underlying observation that drove it: raw Whisper (`whisper-1`) output on this project's P25 audio contains real, non-code mishearings — "Holmes Street and 4th and Rowe" for "Home Street and Forest Ave", "Oklahoma City" for "10-4 Central", "That was a heavy centrifuge" (no clear real referent), "Someone's wearing an alleyway" — that a post-hoc correction pass can only partially and riskily fix. #162 made the correction pass safer; it does not make the underlying transcription more accurate. This issue is the research pass on the actual STT layer, not the correction layer. **Current implementation** (`drb-c2-core/app/internal/transcription.py`): OpenAI hosted `whisper-1`, `language="en"`, `temperature=0`, a fixed generic priming prompt (deliberately excludes real per-system vocabulary — documented reason: `whisper-1` treats the prompt as a transcription prior and echoes vocabulary into silence/noise, which is how the "10-4. 10-5. 10-6…" hallucination bug happened previously). `verbose_json` gives per-segment `no_speech_prob`, which `_is_degenerate()`/the silence filter both depend on for hallucination rejection. Audio reaching Whisper is already lossless FLAC at capture's native 22050 Hz (deliberately not pre-resampled — correct, already reasoned through in `call_recorder.py`'s comments) — **this part of the pipeline is already solid, not a target for rework.** **The floor under all of this**: P25 uses the IMBE vocoder (4400 bps voice + 2800 bps FEC). Published vocoder research is blunt about it — intelligibility "starts to degrade" as SNR approaches 0 dB and "degrades significantly" below it. No STT model, ours or anyone's, recovers information the vocoder itself never encoded. Some of what reads as "bad STT" may be weak-signal reception at the antenna, not a model problem — worth ruling in or out before spending on the model side (see Angle 5). --- ### Angle 1 — Switch hosted model: `gpt-4o-transcribe` / `gpt-4o-mini-transcribe` OpenAI's own published benchmark (Robust Speech Bench, noisy conditions): `gpt-4o-mini-transcribe` 0.046 WER, `gpt-4o-transcribe` 0.047 WER, vs `whisper-1` 0.053 WER. Real, quantified, and already supported in `config.py`'s `stt_model` enum — but **not a drop-in swap**: `verbose_json` (segment timestamps + `no_speech_prob`) is `whisper-1`-only per OpenAI's API (already noted in a comment in `transcription.py`). Switching means `_is_degenerate()` and the leading/trailing-silence hallucination filters lose their input entirely and need an equivalent built for plain `json` output, or the switch trades one failure mode (mishearings) for another (unfiltered hallucinations). Also untested: whether these newer models have the same "prompt-as-continuation" hallucination risk that made this project exclude real vocabulary from the Whisper prompt — they're a different underlying architecture (GPT-4o audio, not Whisper's encoder-decoder) and may not inherit that failure mode, which would reopen vocabulary-primed prompting as an option. Needs an A/B eval on real saved DRB call audio before committing, not just trusting the published benchmark (which isn't P25-specific). ### Angle 2 — Self-hosted, larger Whisper (large-v3 / large-v3-turbo) The closest real prior art found (a hobbyist project transcribing P25 police scanner audio directly, not through this codebase) used local Whisper `large-v3`, then `large-v3-turbo` for speed — not the hosted `whisper-1` API, which is roughly `large-v2`-class. Their explicit finding: street names were still the main error class even on the bigger model ("would be nice to fine-tune the model with the streets of my city" — never implemented). Self-hosting removes per-minute OpenAI cost and may raise the accuracy ceiling further than Angle 1, but is a real infra commitment (GPU capacity, a service to run and keep up, deploy/ops ownership) — this is a CIO-scale decision, not a code change, and shouldn't be started without CIO sign-off on the infra cost/ops tradeoff. ### Angle 3 — In-model vocabulary/keyword boosting (Deepgram Nova-3, AssemblyAI, others) This is the angle that most directly targets the root cause of #162's whole incident: Whisper has **no safe way** to bias toward known local vocabulary (the project already tried and rejected prompt-based priming for exactly this reason), which pushed all vocabulary correction into a post-hoc LLM rewrite pass that turned out to be unconstrained enough to swap real ten-codes. Deepgram Nova-3's "Keyterm Prompting" is built into the model's decoding, not a prompt hack: up to 100 terms per request, the model biases decoding toward them "when the acoustic evidence supports them" (i.e., it isn't supposed to have Whisper's continue-the-pattern failure mode), with a vendor-published "up to 625% uplift" in correct recognition for boosted terms, at roughly $0.26/hr batch pricing. Feeding it this system's own `local_knowledge`/vocabulary/unit-callsign list as keyterms could shrink transcription-time mishearings directly, reducing how much weight the correction pass has to carry (and how much room it has to do damage). AssemblyAI has an equivalent word-boost feature. **Real, load-bearing caveat**: published research on P25 specifically found AWS Transcribe and Azure Speech both produced "largely gibberish" on this exact audio type — general-purpose ASR vendors do not uniformly handle vocoded radio well, and none of the vendor benchmarks above were measured on P25 audio. Deepgram/AssemblyAI's fit for this project is **unverified** and must be tested on real saved DRB call samples before any commitment, not assumed from generic noisy-office/phone-call benchmarks. ### Angle 4 — Audio preprocessing (denoise, gain normalization) before STT Mixed evidence, explicitly NOT a free win: one recent study (SAM-Audio preprocessing vs raw noisy speech) found denoising **consistently increased** WER/CER despite improving signal-level/perceptual quality — a generic denoiser can remove information an ASR model was actually using. If this is pursued at all, it needs controlled A/B testing on DRB's own vocoded audio (denoised vs raw, same clips, compare transcripts against a human-verified reference), not a blind pipeline addition. ### Angle 5 — The RF/decode side ("better radio audio" upstream of any software fix) P25/IMBE intelligibility falls off a cliff below 0 dB SNR. That's a receive-chain property: SDR antenna placement/gain, OP25 discriminator tap and squelch settings, distance/terrain to the transmitter. No STT model change fixes audio that was already unintelligible at the antenna. Worth a dedicated, separate look at node-002's current OP25 gain/squelch config and whether weak-signal talkgroups (vs strong ones) correlate with the worst transcripts — if they do, the highest-leverage fix might be a fleet/RF change, not a model change. Out of scope for this issue to resolve; flagging so it isn't missed while everyone's attention is on the software side. ### Angle 6 — Already solid, do not rework FLAC lossless capture at the native sample rate (fixed from a prior lossy-MP3 mistake, well-reasoned in `call_recorder.py`'s own comments), the single resample happening inside Whisper rather than client-side, `language="en"` + `temperature=0`, and the existing `_is_degenerate()`/`no_speech_prob` hallucination filtering are all sound. Nothing here should be touched as part of this initiative. --- ### Recommended sequencing (not a ruling — flagging for a CTO/board call given the cost and infra tradeoffs across angles 1–3) 1. Build a small held-out eval set: N real DRB calls with a human-verified reference transcript (this doesn't exist today as far as I could find — confirm before assuming one does). 2. A/B Angle 1 (`gpt-4o-transcribe`/`mini`) against current `whisper-1` on that set, including designing the non-`verbose_json` hallucination-filter replacement it would need. 3. A/B Angle 3 (Deepgram Nova-3 keyterm prompting, seeded from a system's real vocabulary/local_knowledge) on the same set — this is the one most likely to reduce reliance on the correction pass that #162 just had to guard. 4. Only after 2–3 have real numbers, decide whether Angle 2 (self-hosting) is worth the infra commitment. 5. Separately, sanity-check Angle 5 against real signal-strength data — cheap to check, could reframe the whole priority order if weak signal turns out to be the dominant driver on the worst-performing talkgroups. Related: #36 (per-talkgroup transcript correction — appears already implemented, flagged separately) and #6/#159 (geocode_max_km rejecting legitimate far-talkgroup results — fixed, same underlying "no anchor / no reference point" family of gap this project keeps hitting on the AI pipeline's edges). Sources consulted (web research, 2026-09-21, general ASR benchmarks — none P25-specific except where noted): - OpenAI gpt-4o-transcribe/whisper-1 Robust Speech Bench WER comparison - Deepgram Nova-3 Keyterm Prompting docs and pricing - AssemblyAI custom vocabulary / word boost - P25 IMBE vocoder SNR-intelligibility relationship (patent/technical literature) - A hobbyist P25 police-scanner transcription project's model choice and street-name error findings (self-hosted Whisper large-v3/large-v3-turbo) - Denoising-before-ASR WER impact research (positive and negative results both found — no universal answer)

logan referenced this issue

2026-09-20 21:57:07 -04:00

transcript_correction.py accepted the model's rewrite verbatim — a real ten-code got swapped for a different real ten-code #162

logan referenced this issue from a commit

2026-09-21 00:04:43 -04:00

admin: STT eval harness — record human-verified transcripts, measure real WER (#163)

logan added the area/transcription

labels 2026-10-05 02:04:55 -04:00