admin: STT eval harness — record human-verified transcripts, measure real WER (#163)
Backend: three new routes on the calls router, deliberately separate from
PATCH /{call_id}/transcript (a production correction with real side effects
-- re-extraction, incident unlinking, vocabulary learning). This is pure
measurement and must never share that code path.
GET /calls/eval-queue -- calls with a transcript but no
eval_transcript yet, paged (same bounded-
window-plus-cursor shape as /search)
PUT /{call_id}/eval-transcript -- records eval_transcript/_by/_at only;
never touches transcript/transcript_corrected
GET /calls/eval-stats -- eval_count + average word error rate of
the raw and corrected machine transcripts
against the human-verified ones
internal/wer.py: standard word-level Levenshtein WER. Returns None (not 0.0)
when the reference is empty -- a call nobody transcribed must not score as a
perfect match.
Frontend: a new "STT Eval" tab on /admin -- one call at a time, audio player,
a textarea pre-filled with the machine transcript to correct into ground
truth, Save & next / Skip, running WER stats at the top. Built for working a
handful of calls at a time over however many sittings it takes, not a
one-shot form: the queue auto-refills from where the last save left off.
Verified: 438 pass, 0 fail (12 new backend tests). Frontend is UNVERIFIED --
this box has no Node.js/npm (confirmed absent), so neither typecheck nor the
dev server could be run. Matches existing code patterns and the CallRecord/
c2api types by manual review only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
241a15b8da
commit
5f85a878fa
@@ -100,6 +100,30 @@ export const c2api = {
|
||||
closeStallCalls: (olderThanMinutes: number, dryRun: boolean) =>
|
||||
request<{ dry_run: boolean; older_than_minutes: number; count: number; call_ids: string[] }>(`/calls/close-stale?older_than_minutes=${olderThanMinutes}&dry_run=${dryRun}`, { method: "POST" }),
|
||||
|
||||
// STT eval harness (server-26#163) — separate from patchTranscript above,
|
||||
// which is a production correction with real side effects (re-extraction,
|
||||
// incident unlinking, vocabulary learning). This is pure measurement.
|
||||
getEvalQueue: (limit: number, cursor?: string | null) => {
|
||||
const qs = new URLSearchParams({ limit: String(limit) });
|
||||
if (cursor) qs.set("cursor", cursor);
|
||||
return request<{
|
||||
calls: import("@/lib/types").CallRecord[];
|
||||
next_cursor: string | null;
|
||||
scanned: number;
|
||||
matched: number;
|
||||
window_exhausted: boolean;
|
||||
}>(`/calls/eval-queue?${qs.toString()}`);
|
||||
},
|
||||
getEvalStats: () =>
|
||||
request<{ eval_count: number; raw_wer: number | null; corrected_wer: number | null }>(
|
||||
"/calls/eval-stats"
|
||||
),
|
||||
putEvalTranscript: (callId: string, text: string) =>
|
||||
request<{ ok: boolean; call_id: string }>(`/calls/${callId}/eval-transcript`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify({ text }),
|
||||
}),
|
||||
|
||||
// Incidents
|
||||
getIncidents: (params?: { status?: string; type?: string }) => {
|
||||
const qs = params ? "?" + new URLSearchParams(params as Record<string, string>).toString() : "";
|
||||
|
||||
Reference in New Issue
Block a user