Logan CusanoandClaude Sonnet 5
5f85a878fa
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.
GET /calls/eval-queue -- calls with a transcript but no
eval_transcript yet, paged (same bounded-
window-plus-cursor shape as /search)
PUT /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
never touches transcript/transcript_corrected
GET /calls/eval-stats -- eval_count + average word error rate of
the raw and corrected machine transcripts
against the human-verified ones
internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.
Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.
Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
45 lines
1.5 KiB
Python
45 lines
1.5 KiB
Python
"""
|
|
Word error rate — server-26#163's eval harness needs a real number to compare
|
|
against, not a vibe. Standard definition: word-level Levenshtein distance
|
|
between a human-verified reference and the machine hypothesis, divided by the
|
|
reference's own word count. Case-insensitive, punctuation-insensitive — this
|
|
measures whether the right WORDS came out, not transcript formatting.
|
|
"""
|
|
import re
|
|
from typing import Optional
|
|
|
|
|
|
def _tokenize(text: str) -> list[str]:
|
|
return re.findall(r"[\w']+", (text or "").lower())
|
|
|
|
|
|
def word_error_rate(reference: str, hypothesis: str) -> Optional[float]:
|
|
"""
|
|
(substitutions + deletions + insertions) / len(reference words).
|
|
|
|
None when the reference has no words — WER is undefined there, not 0.0;
|
|
a caller that defaults a None to 0.0 would report a perfect score for a
|
|
call nobody actually transcribed.
|
|
"""
|
|
ref = _tokenize(reference)
|
|
hyp = _tokenize(hypothesis)
|
|
if not ref:
|
|
return None
|
|
if not hyp:
|
|
return 1.0
|
|
|
|
n, m = len(ref), len(hyp)
|
|
# Single-row DP over Levenshtein distance — O(n*m) time, O(m) space.
|
|
row = list(range(m + 1))
|
|
for i in range(1, n + 1):
|
|
prev_diag = row[0]
|
|
row[0] = i
|
|
for j in range(1, m + 1):
|
|
prev_row_j = row[j]
|
|
if ref[i - 1] == hyp[j - 1]:
|
|
row[j] = prev_diag
|
|
else:
|
|
row[j] = 1 + min(prev_diag, row[j], row[j - 1])
|
|
prev_diag = prev_row_j
|
|
return row[m] / n
|