Stop throwing away the audio Whisper has to read
Build edge-node / build (push) Successful in 35s
CI / lint (push) Successful in 7s
CI / test (push) Successful in 42s

The single encode at save time was %mp3(bitrate=16), and the comment said why:
it matched what Liquidsoap pushes to Icecast. That was the wrong thing to
match. Icecast is the LISTENING path and 16 kbps is a bandwidth budget for a
live stream; this file is the ACCURACY path -- it is what Whisper transcribes,
and CLAUDE.md is explicit that everything downstream is hostage to it. P25 has
already been through a vocoder, so 16 kbps MP3 stacked a second lossy stage on
the one copy that had to stay faithful.

FLAC instead. Lossless, so the bytes Whisper receives are the bytes PulseAudio
captured. ~1.3 MB/min against 120 KB/min, which keeps a 600 s call (the time
cap) around 13 MB -- inside Whisper's 25 MB request cap and well inside
upload_max_bytes. Icecast's own 16 kbps stream is untouched; nothing about
live listening changes.

Capture, buffering, silence detection and the byte-offset trim are all
unchanged: they operate on raw PCM and never saw the encode. The sample rate
stays pinned to pcm.SAMPLE_RATE so the encode remains a straight pass -- the
trim arithmetic depends on that, and Whisper resamples to 16 kHz itself.

encode_mp3 is now encode_recording, the upload sends audio/flac, and the test
that pinned the old contract now pins losslessness instead, including an
assertion that no bitrate constant comes back.

This is the before/after boundary for STT quality. Last night's window is the
16 kbps baseline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-23 12:48:32 -04:00
co-authored by Claude Opus 5
parent fb13bb8ae3
commit 0c08275482
2 changed files with 64 additions and 31 deletions
+46 -23
View File
@@ -7,14 +7,16 @@ A persistent capture process runs for the lifetime of the node. Spawning FFmpeg
per call used to lose the first 1-2 s to process startup, which meant short per call used to lose the first 1-2 s to process startup, which meant short
transmissions produced empty files, so capture never stops. transmissions produced empty files, so capture never stops.
RAW PCM, NOT MP3 — this is the change everything else hangs off. FFmpeg is RAW PCM, NOT A COMPRESSED STREAM — this is the change everything else hangs
asked for s16le/22050/mono on stdout instead of an MP3 stream, so: off. FFmpeg is asked for s16le/22050/mono on stdout instead of an encoded
stream, so:
* silence detection is integer arithmetic over each chunk as it arrives, with * silence detection is integer arithmetic over each chunk as it arrives, with
no decode, which is what makes AUDIO-DRIVEN call boundaries possible; no decode, which is what makes AUDIO-DRIVEN call boundaries possible;
* trimming is a byte-offset slice, not a second FFmpeg pass; * trimming is a byte-offset slice, not a second FFmpeg pass;
* MP3 encoding happens exactly ONCE, at save time, so uploads are no longer * encoding happens exactly ONCE, at save time, so uploads are no longer
double-encoded. double-encoded. That encode is now FLAC (lossless) rather than 16 kbps MP3
— see the AUDIO_* constants below for why.
TWO BUFFERS, TWO JOBS — this split is load-bearing: TWO BUFFERS, TWO JOBS — this split is load-bearing:
@@ -94,14 +96,32 @@ RING_BUFFER_SECONDS = 30
# under PRE_ROLL_SECONDS and well under the shortest utterance we care about. # under PRE_ROLL_SECONDS and well under the shortest utterance we care about.
READ_CHUNK_BYTES = 2048 READ_CHUNK_BYTES = 2048
# Encoder settings for the single encode at save time, matched on purpose to # Encoder settings for the single encode at save time.
# what Liquidsoap already pushes to Icecast — %mp3(bitrate=16, samplerate=22050, #
# stereo=false) — so the C2 /upload endpoint keeps receiving exactly the kind of # This used to be %mp3(bitrate=16) chosen to match what Liquidsoap pushes to
# MP3 it has always received (multipart "audio/mpeg", stored to GCS as .mp3, # Icecast. That was the wrong thing to match: Icecast is the LISTENING path and
# then fed to Whisper). MP3_SAMPLE_RATE MUST equal pcm.SAMPLE_RATE: the encode # 16 kbps is a bandwidth budget for a live stream, while this file is the
# is a straight pass with no resampling. # ACCURACY path — it is what Whisper transcribes, and the transcript is what
MP3_BITRATE = "16k" # every downstream stage is hostage to. P25 audio has already been through a
MP3_SAMPLE_RATE = str(pcm.SAMPLE_RATE) # vocoder; 16 kbps MP3 stacked a second lossy stage on top of that, on the one
# copy that had to stay faithful.
#
# FLAC instead: lossless, so the bytes Whisper receives are the bytes PulseAudio
# captured. Roughly 1.3 MB/min against 120 KB/min for 16k MP3 — larger, but a
# 600 s call is still ~13 MB, inside both Whisper's 25 MB request cap and the
# C2 upload_max_bytes (100 MB). Icecast's own 16 kbps stream is untouched;
# nothing about the listening path changes.
#
# AUDIO_SAMPLE_RATE MUST equal pcm.SAMPLE_RATE: the encode is a straight pass
# with no resampling. Whisper resamples to 16 kHz itself, so handing it 22050
# unresampled keeps the one resample in the pipeline inside the model.
AUDIO_SAMPLE_RATE = str(pcm.SAMPLE_RATE)
AUDIO_FORMAT = "flac"
AUDIO_SUFFIX = ".flac"
AUDIO_MIME = "audio/flac"
# -compression_level 5 is ffmpeg's default: near-best ratio, and the encode is
# off the hot path anyway (once per call, at save).
FLAC_COMPRESSION_LEVEL = "5"
# Bounded so a wedged encoder can never stall the upload path. # Bounded so a wedged encoder can never stall the upload path.
ENCODE_TIMEOUT_SECONDS = 60.0 ENCODE_TIMEOUT_SECONDS = 60.0
@@ -218,9 +238,12 @@ class Recording:
all_silence: bool = False all_silence: bool = False
async def encode_mp3(audio: bytes, path: Path) -> bool: async def encode_recording(audio: bytes, path: Path) -> bool:
""" """
The one and only encode in the pipeline: raw PCM in, MP3 file out. The one and only encode in the pipeline: raw PCM in, FLAC file out.
Lossless on purpose — see the AUDIO_* constants above. This file is what
Whisper transcribes, so the encode must not throw anything away.
Module-level rather than a method so tests can substitute it without Module-level rather than a method so tests can substitute it without
needing FFmpeg, and so the "exactly one encode per call" property is needing FFmpeg, and so the "exactly one encode per call" property is
@@ -233,13 +256,13 @@ async def encode_mp3(audio: bytes, path: Path) -> bool:
"-hide_banner", "-nostdin", "-nostats", "-hide_banner", "-nostdin", "-nostats",
"-loglevel", "warning", "-y", "-loglevel", "warning", "-y",
"-f", "s16le", "-f", "s16le",
"-ar", MP3_SAMPLE_RATE, "-ar", AUDIO_SAMPLE_RATE,
"-ac", str(pcm.CHANNELS), "-ac", str(pcm.CHANNELS),
"-i", "pipe:0", "-i", "pipe:0",
"-ar", MP3_SAMPLE_RATE, "-ar", AUDIO_SAMPLE_RATE,
"-ac", str(pcm.CHANNELS), "-ac", str(pcm.CHANNELS),
"-b:a", MP3_BITRATE, "-compression_level", FLAC_COMPRESSION_LEVEL,
"-f", "mp3", str(path), "-f", AUDIO_FORMAT, str(path),
] ]
try: try:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
@@ -327,7 +350,7 @@ class CallRecorder:
"-loglevel", "warning", "-loglevel", "warning",
"-f", "pulse", "-i", settings.pulse_source, "-f", "pulse", "-i", settings.pulse_source,
"-ac", str(pcm.CHANNELS), "-ac", str(pcm.CHANNELS),
"-ar", MP3_SAMPLE_RATE, "-ar", AUDIO_SAMPLE_RATE,
# Raw PCM on stdout. No muxer, so no -flush_packets games: s16le is # Raw PCM on stdout. No muxer, so no -flush_packets games: s16le is
# a bare byte stream and every byte FFmpeg produces is immediately # a bare byte stream and every byte FFmpeg produces is immediately
# readable, which is what keeps arrival timestamps honest. # readable, which is what keeps arrival timestamps honest.
@@ -360,7 +383,7 @@ class CallRecorder:
async def _run_capture(self) -> None: async def _run_capture(self) -> None:
cmd = self._ffmpeg_command() cmd = self._ffmpeg_command()
logger.info(f"Starting capture: ffmpeg -f pulse -i {settings.pulse_source} (s16le/{MP3_SAMPLE_RATE}/mono)") logger.info(f"Starting capture: ffmpeg -f pulse -i {settings.pulse_source} (s16le/{AUDIO_SAMPLE_RATE}/mono)")
self._last_stderr_lines.clear() self._last_stderr_lines.clear()
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
*cmd, *cmd,
@@ -674,9 +697,9 @@ class CallRecorder:
self._recordings_dir.mkdir(parents=True, exist_ok=True) self._recordings_dir.mkdir(parents=True, exist_ok=True)
ts_str = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") ts_str = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
output_path = self._recordings_dir / f"{ts_str}_{recording.call_id}.mp3" output_path = self._recordings_dir / f"{ts_str}_{recording.call_id}{AUDIO_SUFFIX}"
if not await encode_mp3(audio, output_path): if not await encode_recording(audio, output_path):
output_path.unlink(missing_ok=True) output_path.unlink(missing_ok=True)
return None return None
@@ -766,7 +789,7 @@ class CallRecorder:
with open(file_path, "rb") as f: with open(file_path, "rb") as f:
r = await client.post( r = await client.post(
upload_url, upload_url,
files={"file": (file_path.name, f, "audio/mpeg")}, files={"file": (file_path.name, f, AUDIO_MIME)},
data=form, data=form,
headers=headers, headers=headers,
) )
+18 -8
View File
@@ -59,7 +59,7 @@ def encodes(monkeypatch):
path.write_bytes(audio) path.write_bytes(audio)
return True return True
monkeypatch.setattr(recorder_mod, "encode_mp3", _encode) monkeypatch.setattr(recorder_mod, "encode_recording", _encode)
return calls return calls
@@ -412,21 +412,31 @@ async def test_a_failed_encode_leaves_no_file_and_no_recording(recorder, monkeyp
async def _fail(audio, path): async def _fail(audio, path):
return False return False
monkeypatch.setattr(recorder_mod, "encode_mp3", _fail) monkeypatch.setattr(recorder_mod, "encode_recording", _fail)
ingest(recorder, T0, T0 + 5.0) ingest(recorder, T0, T0 + 5.0)
await recorder.start_recording("call-1", start_epoch=T0 + 1.0) await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
assert await recorder.stop_recording(end_epoch=T0 + 3.0) is None assert await recorder.stop_recording(end_epoch=T0 + 3.0) is None
assert list(recorder._recordings_dir.glob("*.mp3")) == [] assert list(recorder._recordings_dir.glob("*.flac")) == []
def test_encoder_command_contract_matches_what_c2_expects(): def test_encoder_command_contract_matches_what_c2_expects():
""" """
/upload has always received mono MP3 at 22050 Hz / 16 kbps, and Whisper The saved file is what Whisper transcribes, so the encode must stay
consumes it downstream. The single encode must not quietly change that. LOSSLESS and must not resample. It was 16 kbps MP3 — a bitrate copied from
Icecast's live stream, i.e. the listening path's budget applied to the
accuracy path — which put a second lossy stage on top of the P25 vocoder.
The sample rate must equal pcm.SAMPLE_RATE or the encode stops being a
straight pass and the byte-offset trim arithmetic no longer lines up.
""" """
assert recorder_mod.MP3_SAMPLE_RATE == str(pcm.SAMPLE_RATE) == "22050" assert recorder_mod.AUDIO_SAMPLE_RATE == str(pcm.SAMPLE_RATE) == "22050"
assert recorder_mod.MP3_BITRATE == "16k" assert recorder_mod.AUDIO_FORMAT == "flac"
assert recorder_mod.AUDIO_SUFFIX == ".flac"
assert recorder_mod.AUDIO_MIME == "audio/flac"
# No bitrate constant should exist: a bitrate on a lossless codec would mean
# someone reintroduced lossy encoding.
assert not hasattr(recorder_mod, "MP3_BITRATE")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -523,7 +533,7 @@ async def test_discard_drops_the_audio_without_writing_anything(recorder, encode
assert not recorder.is_recording assert not recorder.is_recording
assert encodes == [] assert encodes == []
assert list(recorder._recordings_dir.glob("*.mp3")) == [] assert list(recorder._recordings_dir.glob("*.flac")) == []
# ...and the recorder is immediately reusable. # ...and the recorder is immediately reusable.
assert await recorder.start_recording("call-next", start_epoch=T0 + 2.0) is True assert await recorder.start_recording("call-next", start_epoch=T0 + 2.0) is True