Drive call boundaries from audio, use the console only for the label
The control channel was wrong in both directions. Grants fire 0.84-1.62s
before anyone speaks, and srcaddr can drop to 0 while someone is still
talking - one recording came back "-1.61s lead, -0.00s tail", the trim
finding nothing to remove because the window had closed on live speech.
Confirmed by ear: the cut lands at a word boundary on an unfinished word.
Audio is ground truth for WHEN. The console remains the only source of
WHO, so it still supplies talkgroup, alias and rid.
START voice onset in the captured audio, with a 0.25s pre-roll that
now covers only chunk quantisation and threshold ramp-up rather
than a variable control-channel offset.
STOP call_silence_timeout seconds of silence heard in the audio.
LABEL resolved AT CLOSE from a bounded rolling history of console
observations overlapping the window, +4s/-2s, because there is
no guaranteed ordering between a grant and its audio.
SPLIT a console talkgroup change still forces a cut, since two calls
with no silence between them would otherwise merge into one.
Capture now emits raw PCM instead of MP3. Silence detection becomes
integer arithmetic per chunk with no decode, trimming becomes a byte
offset slice rather than a second ffmpeg pass, and MP3 encoding happens
exactly once at save - uploads are no longer double-encoded.
Audio with no talkgroup anywhere in its window is discarded rather than
uploaded: an untagged call silently poisons incident correlation, which
is worse than losing the audio. Logged at ERROR and counted on
/api/status.
When capture produces no audio at all the old console state machine
still runs, so a node with a broken audio path keeps reporting radio
activity. That is now the only consumer of call_idle_timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,160 +1,218 @@
|
||||
"""
|
||||
Unit tests for silence-trim decision logic.
|
||||
Unit tests for silence trimming, now a byte-offset slice of raw PCM.
|
||||
|
||||
`speech_bounds` is pure on purpose so the "what do we keep" decision — the part
|
||||
that can destroy a transmission if it is wrong — is testable without FFmpeg.
|
||||
The numbers below come from ffmpeg silencedetect run against six real recordings
|
||||
off a live P25 node: 1.71–2.45 s of leading silence and 0.00–1.11 s trailing.
|
||||
`keep_window` is pure on purpose so the "what do we keep" decision — the part
|
||||
that can destroy a transmission if it is wrong — stays testable without any
|
||||
audio at all. The rest of the file drives the real detector over synthesised
|
||||
buffers shaped like the six real recordings measured off a live P25 node:
|
||||
1.71-2.45 s of leading silence and 0.00-1.11 s trailing.
|
||||
|
||||
The old implementation shelled out to FFmpeg twice (silencedetect, then a
|
||||
re-encode) and these tests parsed its stderr. Both passes are gone; the recorder
|
||||
buffers PCM, so detection is arithmetic and the cut is a slice.
|
||||
"""
|
||||
from array import array
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.internal import audio_trim, pcm
|
||||
from app.internal.audio_trim import (
|
||||
TrimResult,
|
||||
_parse_duration,
|
||||
_parse_silences,
|
||||
speech_bounds,
|
||||
first_signal_offset,
|
||||
keep_window,
|
||||
last_signal_offset,
|
||||
trim_pcm,
|
||||
)
|
||||
|
||||
GUARD = 0.25
|
||||
SPEECH_LEVEL = 4096 # -18 dBFS, the measured field average
|
||||
FLOOR_LEVEL = 1 # -90.3 dBFS, the measured digital-silence floor
|
||||
|
||||
|
||||
def test_leading_silence_is_trimmed_with_a_guard_margin():
|
||||
# Real shape of file f4bfaa1f: 1.85s lead, 0.34s trail, 4.54s total.
|
||||
regions = [(0.0, 1.85), (4.20, None)]
|
||||
start, end, all_silence = speech_bounds(regions, duration=4.54, guard=GUARD)
|
||||
|
||||
assert not all_silence
|
||||
assert start == pytest.approx(1.85 - GUARD)
|
||||
assert end == pytest.approx(4.20 + GUARD)
|
||||
# The guard must never eat into detected speech.
|
||||
assert start < 1.85 and end > 4.20
|
||||
def speech(seconds: float) -> bytes:
|
||||
count = int(pcm.SAMPLE_RATE * seconds)
|
||||
return array("h", [SPEECH_LEVEL, -SPEECH_LEVEL] * (count // 2)).tobytes()
|
||||
|
||||
|
||||
def test_guard_margin_never_runs_past_the_file_bounds():
|
||||
regions = [(0.0, 0.10), (3.95, None)]
|
||||
start, end, _ = speech_bounds(regions, duration=4.0, guard=1.0)
|
||||
|
||||
assert start == 0.0
|
||||
assert end == 4.0
|
||||
|
||||
|
||||
def test_trailing_silence_is_trimmed_when_ffmpeg_closes_the_region_at_eof():
|
||||
"""
|
||||
FFmpeg 6.x flushes a `silence_end` at EOF, so a trailing region looks closed.
|
||||
Treating "no silence_end" as the only trailing signal silently disabled tail
|
||||
trimming entirely — verified against ffmpeg 6.1.1.
|
||||
"""
|
||||
# Real ffmpeg 6.1.1 output for a 2s-silence + 1.5s-tone + 1s-silence file.
|
||||
regions = [(0.0, 2.05361), (3.56367, 4.63102)]
|
||||
start, end, all_silence = speech_bounds(regions, duration=4.65, guard=GUARD)
|
||||
|
||||
assert not all_silence
|
||||
assert start == pytest.approx(2.05361 - GUARD)
|
||||
assert end == pytest.approx(3.56367 + GUARD), "the trailing second must be trimmed"
|
||||
|
||||
|
||||
def test_all_silence_survives_ffmpeg_closing_the_region_at_eof():
|
||||
# Real ffmpeg 6.1.1 output for a 4s file of pure silence.
|
||||
_, _, all_silence = speech_bounds([(0.0, 4.0)], duration=4.03, guard=GUARD)
|
||||
assert all_silence
|
||||
|
||||
|
||||
def test_trailing_silence_that_does_not_reach_eof_is_left_alone():
|
||||
"""
|
||||
A silence region with a closing silence_end is an internal pause between
|
||||
transmissions, not dead air at the tail. Trimming it would cut the middle
|
||||
out of a conversation.
|
||||
"""
|
||||
regions = [(0.0, 1.9), (5.0, 7.5)]
|
||||
start, end, _ = speech_bounds(regions, duration=12.0, guard=GUARD)
|
||||
|
||||
assert start == pytest.approx(1.9 - GUARD)
|
||||
assert end == 12.0, "an internal pause must not shorten the file"
|
||||
|
||||
|
||||
def test_no_silence_detected_keeps_the_whole_file():
|
||||
start, end, all_silence = speech_bounds([], duration=6.0, guard=GUARD)
|
||||
|
||||
assert (start, end) == (0.0, 6.0)
|
||||
assert not all_silence
|
||||
|
||||
|
||||
def test_silence_starting_late_is_not_treated_as_leading():
|
||||
"""Only a region at the very head counts as leading silence."""
|
||||
regions = [(1.20, 2.00)]
|
||||
start, end, _ = speech_bounds(regions, duration=5.0, guard=GUARD)
|
||||
|
||||
assert start == 0.0, "speech before 1.20s must not be trimmed away"
|
||||
assert end == 5.0
|
||||
|
||||
|
||||
def test_all_silence_is_reported_not_trimmed_to_nothing():
|
||||
# One region covering the whole file and running to EOF.
|
||||
regions = [(0.0, None)]
|
||||
start, end, all_silence = speech_bounds(regions, duration=4.0, guard=GUARD)
|
||||
|
||||
assert all_silence
|
||||
assert (start, end) == (0.0, 4.0), "an all-silence file must not become zero-length"
|
||||
|
||||
|
||||
def test_all_silence_when_head_and_tail_regions_overlap():
|
||||
regions = [(0.0, 3.2), (3.0, None)]
|
||||
_, _, all_silence = speech_bounds(regions, duration=4.0, guard=GUARD)
|
||||
|
||||
assert all_silence
|
||||
def silence(seconds: float, level: int = FLOOR_LEVEL) -> bytes:
|
||||
count = int(pcm.SAMPLE_RATE * seconds)
|
||||
return array("h", [level, -level] * (count // 2)).tobytes()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# FFmpeg output parsing
|
||||
# keep_window — the pure decision
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Verbatim shape of ffmpeg 6.1.1 output.
|
||||
FFMPEG_STDERR = """
|
||||
Input #0, mp3, from '/recordings/x.mp3':
|
||||
Duration: 00:00:04.70, start: 0.050113, bitrate: 16 kb/s
|
||||
[silencedetect @ 0000029160e63f40] silence_start: 0
|
||||
[silencedetect @ 0000029160e63f40] silence_end: 2.05361 | silence_duration: 2.05361
|
||||
[silencedetect @ 0000029160e63f40] silence_start: 3.56367
|
||||
[silencedetect @ 0000029160e63f40] silence_end: 4.63102 | silence_duration: 1.06735
|
||||
[out#0/null @ 0x2] video:0kB audio:97kB
|
||||
"""
|
||||
|
||||
FFMPEG_STDERR_OPEN_TAIL = """
|
||||
Duration: 00:00:04.54, start: 0.000000, bitrate: 16 kb/s
|
||||
[silencedetect @ 0x1] silence_start: 0
|
||||
[silencedetect @ 0x1] silence_end: 1.85042 | silence_duration: 1.85042
|
||||
[silencedetect @ 0x1] silence_start: 4.20134
|
||||
"""
|
||||
def test_guard_margin_is_kept_around_detected_speech():
|
||||
guard = pcm.byte_offset(GUARD)
|
||||
start, end = keep_window(
|
||||
first_signal=pcm.byte_offset(1.85),
|
||||
last_signal=pcm.byte_offset(4.20),
|
||||
total_bytes=pcm.byte_offset(4.54),
|
||||
guard_bytes=guard,
|
||||
)
|
||||
assert pcm.seconds(start) == pytest.approx(1.85 - GUARD, abs=0.001)
|
||||
assert pcm.seconds(end) == pytest.approx(4.20 + GUARD, abs=0.001)
|
||||
|
||||
|
||||
def test_duration_is_corrected_for_the_mp3_container_start_offset():
|
||||
def test_guard_margin_never_runs_past_the_buffer_bounds():
|
||||
total = pcm.byte_offset(4.0)
|
||||
start, end = keep_window(
|
||||
first_signal=pcm.byte_offset(0.10),
|
||||
last_signal=pcm.byte_offset(3.95),
|
||||
total_bytes=total,
|
||||
guard_bytes=pcm.byte_offset(1.0),
|
||||
)
|
||||
assert (start, end) == (0, total)
|
||||
|
||||
|
||||
def test_keep_window_offsets_are_sample_aligned():
|
||||
start, end = keep_window(3, 9, 21, 1)
|
||||
assert start % pcm.FRAME_BYTES == 0
|
||||
assert end % pcm.FRAME_BYTES == 0
|
||||
|
||||
|
||||
def test_no_signal_found_keeps_everything():
|
||||
total = pcm.byte_offset(6.0)
|
||||
assert keep_window(None, None, total, pcm.byte_offset(GUARD)) == (0, total)
|
||||
|
||||
|
||||
def test_an_inverted_window_degrades_to_keeping_everything():
|
||||
"""Never return an empty slice, whatever the inputs say."""
|
||||
total = pcm.byte_offset(4.0)
|
||||
assert keep_window(pcm.byte_offset(3.0), pcm.byte_offset(0.5), total, 0) == (0, total)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scanning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_first_and_last_signal_are_found_in_a_realistic_recording():
|
||||
audio = silence(1.85) + speech(2.35) + silence(0.34)
|
||||
threshold = -40.0
|
||||
|
||||
first = first_signal_offset(audio, threshold)
|
||||
last = last_signal_offset(audio, threshold)
|
||||
|
||||
assert pcm.seconds(first) == pytest.approx(1.85, abs=audio_trim.ANALYSIS_WINDOW_SECONDS)
|
||||
assert pcm.seconds(last) == pytest.approx(4.20, abs=audio_trim.ANALYSIS_WINDOW_SECONDS)
|
||||
|
||||
|
||||
def test_internal_pauses_are_not_treated_as_the_tail():
|
||||
"""Trimming the middle out of a conversation would be unrecoverable."""
|
||||
audio = silence(1.9) + speech(3.1) + silence(2.5) + speech(4.5)
|
||||
last = last_signal_offset(audio, -40.0)
|
||||
assert pcm.seconds(last) == pytest.approx(12.0, abs=audio_trim.ANALYSIS_WINDOW_SECONDS)
|
||||
|
||||
|
||||
def test_all_silence_returns_no_signal_offset():
|
||||
assert first_signal_offset(silence(4.0), -40.0) is None
|
||||
assert last_signal_offset(silence(4.0), -40.0) is None
|
||||
|
||||
|
||||
def test_the_scan_is_bounded_so_a_long_buffer_cannot_stall_the_upload():
|
||||
"""The per-sample loop is the only unbounded cost; it must have a ceiling."""
|
||||
audio = silence(2.0)
|
||||
assert first_signal_offset(audio, -40.0, limit_seconds=0.5) is None
|
||||
assert first_signal_offset(speech(0.1) + silence(1.9), -40.0, limit_seconds=0.5) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# trim_pcm — end to end over synthesised audio
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_leading_and_trailing_silence_are_trimmed_to_the_guard_margin():
|
||||
audio = silence(1.85) + speech(2.35) + silence(0.34)
|
||||
kept, result = trim_pcm(audio, threshold_db=-40.0, guard=GUARD)
|
||||
|
||||
assert result.applied and not result.all_silence
|
||||
assert result.lead == pytest.approx(1.85 - GUARD, abs=0.05)
|
||||
assert result.tail == pytest.approx(0.34 - GUARD, abs=0.05)
|
||||
assert pcm.seconds(len(kept)) == pytest.approx(result.duration_after, abs=0.001)
|
||||
assert result.duration_after < result.duration_before
|
||||
|
||||
|
||||
def test_the_guard_margin_never_eats_into_speech():
|
||||
audio = silence(2.0) + speech(1.0) + silence(2.0)
|
||||
kept, result = trim_pcm(audio, threshold_db=-40.0, guard=GUARD)
|
||||
|
||||
# Everything removed from the head must be silence, and the first sample of
|
||||
# real speech must survive.
|
||||
assert result.lead < 2.0
|
||||
assert pcm.seconds(len(kept)) > 1.0
|
||||
|
||||
|
||||
def test_measured_trailing_silence_is_reported_for_field_tuning():
|
||||
"""
|
||||
MP3 encoder delay makes the container duration longer than the audio
|
||||
silencedetect timestamps. Without this correction the trailing-region test
|
||||
needs a slack epsilon big enough to clip real speech.
|
||||
The recorder deliberately over-captures the tail (it closes only after the
|
||||
silence timeout has actually elapsed in the audio), so `tail` is how the
|
||||
real silence run reaches the logs.
|
||||
"""
|
||||
assert _parse_duration(FFMPEG_STDERR) == pytest.approx(4.70 - 0.050113)
|
||||
audio = silence(0.5) + speech(2.0) + silence(3.0)
|
||||
_, result = trim_pcm(audio, threshold_db=-40.0, guard=GUARD)
|
||||
assert result.tail == pytest.approx(3.0 - GUARD, abs=0.05)
|
||||
assert result.trimmed_seconds == pytest.approx(result.lead + result.tail)
|
||||
|
||||
|
||||
def test_duration_is_none_when_absent():
|
||||
assert _parse_duration("no duration here") is None
|
||||
def test_an_all_silence_buffer_is_reported_not_truncated_to_nothing():
|
||||
audio = silence(4.0)
|
||||
kept, result = trim_pcm(audio, threshold_db=-40.0, guard=GUARD)
|
||||
|
||||
assert result.all_silence
|
||||
assert not result.applied
|
||||
assert kept == audio, "an all-silence recording must not become zero-length"
|
||||
|
||||
|
||||
def test_silence_regions_are_parsed():
|
||||
regions = _parse_silences(FFMPEG_STDERR)
|
||||
|
||||
assert len(regions) == 2
|
||||
assert regions[0] == (pytest.approx(0.0), pytest.approx(2.05361))
|
||||
assert regions[1] == (pytest.approx(3.56367), pytest.approx(4.63102))
|
||||
def test_digital_silence_at_the_measured_field_floor_is_detected():
|
||||
"""
|
||||
The -91 dBFS floor is the whole reason this needs no field calibration.
|
||||
Detection must not depend on the threshold being tuned to a noise floor.
|
||||
"""
|
||||
audio = silence(1.0, level=1) + speech(1.0) + silence(1.0, level=1)
|
||||
for threshold in (-70.0, -60.0, -50.0, -40.0):
|
||||
_, result = trim_pcm(audio, threshold_db=threshold, guard=GUARD)
|
||||
assert result.applied, f"threshold {threshold} should still find the speech"
|
||||
assert result.lead == pytest.approx(0.75, abs=0.05)
|
||||
|
||||
|
||||
def test_a_region_with_no_silence_end_is_still_parsed():
|
||||
"""Older FFmpeg simply stopped reporting at EOF — keep handling that."""
|
||||
regions = _parse_silences(FFMPEG_STDERR_OPEN_TAIL)
|
||||
def test_audio_with_no_silence_at_either_end_is_left_alone():
|
||||
audio = speech(3.0)
|
||||
kept, result = trim_pcm(audio, threshold_db=-40.0, guard=GUARD)
|
||||
|
||||
assert regions[-1][1] is None
|
||||
assert not result.applied
|
||||
assert kept == audio
|
||||
assert result.duration_before == pytest.approx(result.duration_after)
|
||||
|
||||
|
||||
def test_an_empty_buffer_is_handled():
|
||||
kept, result = trim_pcm(b"", threshold_db=-40.0, guard=GUARD)
|
||||
assert kept == b"" and not result.applied and not result.all_silence
|
||||
|
||||
|
||||
def test_thresholds_default_to_settings():
|
||||
audio = silence(1.0) + speech(1.0) + silence(1.0)
|
||||
_, result = trim_pcm(audio)
|
||||
assert result.applied
|
||||
assert settings.trim_silence_threshold_db == -40.0
|
||||
assert settings.trim_silence_guard_seconds == 0.25
|
||||
assert result.lead == pytest.approx(1.0 - settings.trim_silence_guard_seconds, abs=0.05)
|
||||
|
||||
|
||||
def test_a_scan_that_gives_up_leaves_the_audio_untouched_and_says_so():
|
||||
"""
|
||||
Refusing to guess is the point: an untrimmed upload is always better than a
|
||||
wrongly-truncated one, and better than dropping a call as "all silence"
|
||||
without having actually looked at all of it.
|
||||
"""
|
||||
long_silence = silence(audio_trim.MAX_SCAN_SECONDS + 5.0)
|
||||
kept, result = trim_pcm(long_silence, threshold_db=-40.0, guard=GUARD)
|
||||
|
||||
assert result.scan_truncated
|
||||
assert not result.all_silence
|
||||
assert not result.applied
|
||||
assert kept == long_silence
|
||||
|
||||
|
||||
def test_trim_result_reports_total_trimmed():
|
||||
result = TrimResult(path=None, lead=1.9, tail=0.35)
|
||||
assert result.trimmed_seconds == pytest.approx(2.25)
|
||||
assert TrimResult(lead=1.9, tail=0.35).trimmed_seconds == pytest.approx(2.25)
|
||||
|
||||
@@ -1,13 +1,17 @@
|
||||
"""
|
||||
Unit tests for the CallRecorder pre-roll ring buffer and per-call accumulator.
|
||||
Unit tests for the CallRecorder: PCM ring buffer, per-call accumulator, the
|
||||
continuous voice-activity signal the segmenter reads, and the single encode.
|
||||
|
||||
No FFmpeg and no PulseAudio: chunks are pushed through _ingest() with a patched
|
||||
clock, which is exactly what the capture loop does at runtime. Silence trimming
|
||||
is disabled by default here and exercised separately with a stubbed trimmer.
|
||||
clock, which is exactly what the capture loop does at runtime, and the MP3
|
||||
encoder is replaced with a stub that writes the raw PCM it was handed. That stub
|
||||
is also how "exactly one encode per call" is asserted — the old design captured
|
||||
MP3 and then re-encoded it to trim, so every upload was double-encoded.
|
||||
"""
|
||||
import asyncio
|
||||
import itertools
|
||||
import time
|
||||
from array import array
|
||||
from typing import List
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -15,7 +19,7 @@ import pytest
|
||||
|
||||
from app.config import settings
|
||||
from app.internal import call_recorder as recorder_mod
|
||||
from app.internal.audio_trim import TrimResult
|
||||
from app.internal import pcm
|
||||
from app.internal.call_recorder import (
|
||||
CallRecorder,
|
||||
MAX_RECORDING_BYTES,
|
||||
@@ -25,11 +29,42 @@ from app.internal.call_recorder import (
|
||||
)
|
||||
|
||||
T0 = 1_700_000_000.0
|
||||
CHUNK_INTERVAL = 0.1 # seconds of audio per synthetic chunk
|
||||
|
||||
# One synthetic chunk carries exactly CHUNK_INTERVAL seconds of audio AND
|
||||
# arrives CHUNK_INTERVAL apart, so arrival-timestamp arithmetic (slicing) and
|
||||
# byte-offset arithmetic (trimming) agree with each other.
|
||||
CHUNK_INTERVAL = 0.1
|
||||
CHUNK_SAMPLES = int(pcm.SAMPLE_RATE * CHUNK_INTERVAL)
|
||||
CHUNK_BYTES = CHUNK_SAMPLES * pcm.FRAME_BYTES
|
||||
|
||||
SPEECH_LEVEL = 4096 # -18 dBFS, the measured field average
|
||||
FLOOR_LEVEL = 1 # -90.3 dBFS, the measured digital-silence floor
|
||||
|
||||
|
||||
def block(level: int, samples: int = CHUNK_SAMPLES) -> bytes:
|
||||
return array("h", [level, -level] * (samples // 2)).tobytes()
|
||||
|
||||
|
||||
VOICE = block(SPEECH_LEVEL)
|
||||
QUIET = block(FLOOR_LEVEL)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorder(tmp_path, monkeypatch):
|
||||
def encodes(monkeypatch):
|
||||
"""Replace the one encode with a stub that writes the PCM it was given."""
|
||||
calls: List[tuple] = []
|
||||
|
||||
async def _encode(audio: bytes, path):
|
||||
calls.append((audio, path))
|
||||
path.write_bytes(audio)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(recorder_mod, "encode_mp3", _encode)
|
||||
return calls
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorder(tmp_path, monkeypatch, encodes):
|
||||
monkeypatch.setattr(settings, "trim_silence", False)
|
||||
r = CallRecorder()
|
||||
r._recordings_dir = tmp_path
|
||||
@@ -37,64 +72,51 @@ def recorder(tmp_path, monkeypatch):
|
||||
return r
|
||||
|
||||
|
||||
def ingest(recorder, start: float, end: float, marker: bytes = b"A", index: int = 0) -> int:
|
||||
"""Feed one chunk every CHUNK_INTERVAL seconds over [start, end) through _ingest."""
|
||||
def ingest(recorder, start: float, end: float, chunk: bytes = VOICE) -> None:
|
||||
"""Feed one chunk every CHUNK_INTERVAL seconds over [start, end)."""
|
||||
stamps: List[float] = []
|
||||
chunks: List[bytes] = []
|
||||
ts = start
|
||||
while ts < end:
|
||||
stamps.append(ts)
|
||||
chunks.append(marker + str(index).encode() + b";")
|
||||
index += 1
|
||||
ts = round(ts + CHUNK_INTERVAL, 6)
|
||||
|
||||
if not stamps:
|
||||
return
|
||||
with patch("app.internal.call_recorder.time.time", side_effect=stamps):
|
||||
for chunk in chunks:
|
||||
for _ in stamps:
|
||||
recorder._ingest(chunk)
|
||||
return index
|
||||
|
||||
|
||||
def fill(recorder, start: float, end: float, marker: bytes = b"A", index: int = 0) -> int:
|
||||
"""Alias kept for readability where the accumulator is not the point."""
|
||||
return ingest(recorder, start, end, marker=marker, index=index)
|
||||
def duration_of(path) -> float:
|
||||
return pcm.seconds(len(path.read_bytes()))
|
||||
|
||||
|
||||
def timestamps(recorder):
|
||||
return [ts for ts, _ in recorder._buffer]
|
||||
|
||||
|
||||
def markers(path) -> List[str]:
|
||||
return path.read_bytes().decode().strip(";").split(";")
|
||||
|
||||
|
||||
def indices(path) -> List[int]:
|
||||
return [int(m[1:]) for m in markers(path) if m[1:].isdigit()]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ring buffer trimming (pre-roll duty only)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_idle_buffer_keeps_only_the_rolling_window(recorder):
|
||||
for offset in range(0, int(RING_BUFFER_SECONDS) + 20):
|
||||
with patch("app.internal.call_recorder.time.time", return_value=T0 + offset):
|
||||
recorder._ingest(b"x" * 16)
|
||||
ingest(recorder, T0, T0 + RING_BUFFER_SECONDS + 20)
|
||||
|
||||
assert len(recorder._buffer) <= RING_BUFFER_SECONDS + 1
|
||||
assert min(timestamps(recorder)) >= (T0 + RING_BUFFER_SECONDS + 19) - RING_BUFFER_SECONDS
|
||||
assert recorder.buffered_seconds <= RING_BUFFER_SECONDS + CHUNK_INTERVAL
|
||||
newest = max(timestamps(recorder))
|
||||
assert min(timestamps(recorder)) >= newest - RING_BUFFER_SECONDS
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ring_buffer_is_trimmed_even_while_recording(recorder):
|
||||
"""
|
||||
The ring buffer serves PRE-ROLL only. An open recording must no longer pin
|
||||
it — that was the mechanism that made call length depend on buffer size.
|
||||
The ring buffer serves PRE-ROLL only. An open recording must not pin it —
|
||||
that was the mechanism that made call length depend on buffer size.
|
||||
"""
|
||||
index = ingest(recorder, T0, T0 + 2.0)
|
||||
ingest(recorder, T0, T0 + 2.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
ingest(recorder, T0 + 2.0, T0 + 2.0 + RING_BUFFER_SECONDS + 10, index=index)
|
||||
ingest(recorder, T0 + 2.0, T0 + 2.0 + RING_BUFFER_SECONDS + 10)
|
||||
|
||||
assert recorder.buffered_seconds <= RING_BUFFER_SECONDS + 1
|
||||
assert recorder.buffered_seconds <= RING_BUFFER_SECONDS + CHUNK_INTERVAL
|
||||
# ...and the audio the ring buffer dropped is safe in the accumulator.
|
||||
assert recorder._active is not None
|
||||
assert recorder._active.chunks[0][0] == pytest.approx(T0 + 1.0 - PRE_ROLL_SECONDS, abs=CHUNK_INTERVAL)
|
||||
@@ -110,19 +132,16 @@ async def test_call_longer_than_the_ring_buffer_is_captured_whole(recorder):
|
||||
grant = T0 + 1.0
|
||||
end = grant + call_length
|
||||
|
||||
index = ingest(recorder, T0, grant)
|
||||
ingest(recorder, T0, grant)
|
||||
await recorder.start_recording("call-long", start_epoch=grant)
|
||||
ingest(recorder, grant, end + 1.0, index=index)
|
||||
ingest(recorder, grant, end + 1.0)
|
||||
|
||||
rec = await recorder.stop_recording(end_epoch=end)
|
||||
assert rec is not None and rec.path is not None
|
||||
|
||||
kept = indices(rec.path)
|
||||
# Contiguous: no hole anywhere in the middle of a 65s call.
|
||||
assert kept == list(range(kept[0], kept[-1] + 1))
|
||||
span = (kept[-1] - kept[0]) * CHUNK_INTERVAL
|
||||
assert span > RING_BUFFER_SECONDS, "call length must not be clamped by the ring buffer"
|
||||
assert span == pytest.approx(call_length + PRE_ROLL_SECONDS, abs=2 * CHUNK_INTERVAL)
|
||||
captured = duration_of(rec.path)
|
||||
assert captured > RING_BUFFER_SECONDS, "call length must not be clamped by the ring buffer"
|
||||
assert captured == pytest.approx(call_length + PRE_ROLL_SECONDS, abs=2 * CHUNK_INTERVAL)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -130,8 +149,12 @@ async def test_accumulator_stops_growing_at_the_memory_ceiling(recorder, caplog)
|
||||
"""A runaway call must not be able to exhaust RAM on a Pi."""
|
||||
await recorder.start_recording("call-runaway", start_epoch=T0)
|
||||
|
||||
big = b"z" * 64_000
|
||||
# Silent blocks on purpose: this test is about bytes, not content, and the
|
||||
# all-zero fast path keeps it from spending seconds in the RMS loop.
|
||||
big = b"\x00" * 64_000
|
||||
needed = (MAX_RECORDING_BYTES // len(big)) + 5
|
||||
# An unbounded clock: patching time.time patches it for everything running
|
||||
# inside the block, not only for our calls.
|
||||
ticks = itertools.count()
|
||||
with caplog.at_level("WARNING", logger="drb-edge-node"):
|
||||
with patch("app.internal.call_recorder.time.time",
|
||||
@@ -145,73 +168,133 @@ async def test_accumulator_stops_growing_at_the_memory_ceiling(recorder, caplog)
|
||||
assert any("memory ceiling" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
def test_the_byte_ceiling_can_never_truncate_a_legal_call():
|
||||
"""
|
||||
PCM costs 44.1 KB/s where MP3 cost 2 KB/s, so this had to be re-derived.
|
||||
The TIME cap must always bite before the BYTE cap, or a long pursuit would
|
||||
be silently cut short by a memory limit.
|
||||
"""
|
||||
assert MAX_RECORDING_BYTES > MAX_RECORDING_SECONDS * pcm.BYTES_PER_SECOND
|
||||
# ...and it still has to be a deliberate, bounded number on a Pi.
|
||||
assert MAX_RECORDING_BYTES <= 48 * 1024 * 1024
|
||||
|
||||
|
||||
def test_the_ring_buffer_memory_cost_is_bounded():
|
||||
assert RING_BUFFER_SECONDS * pcm.BYTES_PER_SECOND < 2 * 1024 * 1024
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Voice activity — the signal the segmenter starts and stops on
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_digital_silence_produces_no_voice_marks(recorder):
|
||||
ingest(recorder, T0, T0 + 5.0, chunk=QUIET)
|
||||
|
||||
activity = recorder.audio_activity()
|
||||
assert activity.last_voice_epoch is None
|
||||
assert activity.voice_onset_epoch is None
|
||||
|
||||
|
||||
def test_voice_onset_is_the_arrival_of_the_first_non_silent_chunk(recorder):
|
||||
ingest(recorder, T0, T0 + 2.0, chunk=QUIET)
|
||||
ingest(recorder, T0 + 2.0, T0 + 3.0, chunk=VOICE)
|
||||
|
||||
activity = recorder.audio_activity()
|
||||
assert activity.voice_onset_epoch == pytest.approx(T0 + 2.0)
|
||||
assert activity.last_voice_epoch == pytest.approx(T0 + 3.0 - CHUNK_INTERVAL)
|
||||
|
||||
|
||||
def test_a_gap_shorter_than_the_silence_timeout_does_not_start_a_new_run(recorder, monkeypatch):
|
||||
"""Back-and-forth inside the window is ONE run, hence one recording."""
|
||||
monkeypatch.setattr(settings, "call_silence_timeout", 3.0)
|
||||
ingest(recorder, T0, T0 + 1.0, chunk=VOICE)
|
||||
ingest(recorder, T0 + 1.0, T0 + 2.5, chunk=QUIET)
|
||||
ingest(recorder, T0 + 2.5, T0 + 3.5, chunk=VOICE)
|
||||
|
||||
assert recorder.audio_activity().voice_onset_epoch == pytest.approx(T0)
|
||||
|
||||
|
||||
def test_a_gap_longer_than_the_silence_timeout_starts_a_new_run(recorder, monkeypatch):
|
||||
monkeypatch.setattr(settings, "call_silence_timeout", 3.0)
|
||||
ingest(recorder, T0, T0 + 1.0, chunk=VOICE)
|
||||
ingest(recorder, T0 + 1.0, T0 + 6.0, chunk=QUIET)
|
||||
ingest(recorder, T0 + 6.0, T0 + 7.0, chunk=VOICE)
|
||||
|
||||
assert recorder.audio_activity().voice_onset_epoch == pytest.approx(T0 + 6.0)
|
||||
|
||||
|
||||
def test_activity_snapshot_reports_capture_and_recording_state(recorder):
|
||||
activity = recorder.audio_activity()
|
||||
assert activity.capturing is True
|
||||
assert activity.recording is False
|
||||
|
||||
recorder._capturing = False
|
||||
assert recorder.audio_activity().capturing is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-roll and slicing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_slice_starts_pre_roll_before_the_op25_timestamp(recorder):
|
||||
fill(recorder, T0, T0 + 10.0)
|
||||
grant_time = T0 + 5.0
|
||||
async def test_slice_starts_pre_roll_before_the_detected_onset(recorder):
|
||||
ingest(recorder, T0, T0 + 10.0)
|
||||
onset = T0 + 5.0
|
||||
|
||||
await recorder.start_recording("call-1", start_epoch=grant_time)
|
||||
assert recorder._active.slice_start == pytest.approx(grant_time - PRE_ROLL_SECONDS)
|
||||
await recorder.start_recording("call-1", start_epoch=onset)
|
||||
assert recorder._active.slice_start == pytest.approx(onset - PRE_ROLL_SECONDS)
|
||||
|
||||
rec = await recorder.stop_recording(end_epoch=grant_time + 2.0)
|
||||
rec = await recorder.stop_recording(end_epoch=onset + 2.0)
|
||||
assert rec is not None and rec.path is not None and rec.path.exists()
|
||||
|
||||
first_ts = T0 + indices(rec.path)[0] * CHUNK_INTERVAL
|
||||
|
||||
# A chunk stamped `ts` holds the audio that arrived over [ts - interval, ts],
|
||||
# so the audio actually covered must begin at or before the requested slice
|
||||
# start — erring early is the safe direction, erring late loses speech.
|
||||
assert first_ts - CHUNK_INTERVAL <= grant_time - PRE_ROLL_SECONDS + 1e-6
|
||||
# ...and no more than one chunk of extra pre-roll is dragged in.
|
||||
assert first_ts >= grant_time - PRE_ROLL_SECONDS - 1e-6
|
||||
first_ts = recorder._active.chunks[0][0] if recorder._active else None
|
||||
assert first_ts is None # recording closed
|
||||
# The audio actually covered must begin at or before the requested slice
|
||||
# start — erring early is safe, erring late loses speech.
|
||||
assert rec.audio_start_epoch <= onset - PRE_ROLL_SECONDS + 1e-6
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tail_chunk_straddling_the_end_is_included(recorder):
|
||||
fill(recorder, T0, T0 + 10.0)
|
||||
ingest(recorder, T0, T0 + 10.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
|
||||
# End halfway through a chunk interval.
|
||||
rec = await recorder.stop_recording(end_epoch=T0 + 3.05)
|
||||
last_ts = T0 + indices(rec.path)[-1] * CHUNK_INTERVAL
|
||||
|
||||
assert last_ts >= T0 + 3.05, "the chunk covering the end instant must be kept"
|
||||
# The chunk covering the end instant must be kept, so the captured audio
|
||||
# reaches past the requested end rather than stopping short of it.
|
||||
assert rec.audio_end_epoch >= T0 + 3.05
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_max_recording_seconds_caps_the_slice(recorder):
|
||||
index = fill(recorder, T0, T0 + 1.0)
|
||||
ingest(recorder, T0, T0 + 1.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
ingest(recorder, T0 + 1.0, T0 + MAX_RECORDING_SECONDS + 60, index=index)
|
||||
ingest(recorder, T0 + 1.0, T0 + MAX_RECORDING_SECONDS + 60)
|
||||
|
||||
rec = await recorder.stop_recording(end_epoch=T0 + MAX_RECORDING_SECONDS + 50)
|
||||
last_ts = T0 + indices(rec.path)[-1] * CHUNK_INTERVAL
|
||||
|
||||
assert last_ts <= T0 + 1.0 + MAX_RECORDING_SECONDS + CHUNK_INTERVAL
|
||||
assert duration_of(rec.path) <= MAX_RECORDING_SECONDS + PRE_ROLL_SECONDS + CHUNK_INTERVAL
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tail wait — the fix for recordings that ended mid-word
|
||||
# Tail wait — still needed for control-channel-derived ends (tgid splits)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stop_waits_for_captured_audio_to_reach_the_call_end(recorder, caplog):
|
||||
"""
|
||||
PulseAudio → FFmpeg → encoder → muxer → our pipe read has latency, so at the
|
||||
instant a call ends the newest captured chunk is OLDER than the end epoch.
|
||||
Slicing immediately cuts the last word off. stop_recording must wait for it.
|
||||
A tgid_change close pads past a control-channel timestamp that is ~now, so
|
||||
the audio it asks for has not been captured yet. Slicing immediately would
|
||||
cut the last word off.
|
||||
"""
|
||||
index = ingest(recorder, T0, T0 + 4.0)
|
||||
ingest(recorder, T0, T0 + 4.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
|
||||
async def late_tail():
|
||||
await asyncio.sleep(0.15)
|
||||
with patch("app.internal.call_recorder.time.time", return_value=T0 + 4.6):
|
||||
recorder._ingest(b"TAIL;")
|
||||
recorder._ingest(block(SPEECH_LEVEL))
|
||||
|
||||
task = asyncio.create_task(late_tail())
|
||||
with caplog.at_level("INFO", logger="drb-edge-node"):
|
||||
@@ -219,10 +302,14 @@ async def test_stop_waits_for_captured_audio_to_reach_the_call_end(recorder, cap
|
||||
await task
|
||||
|
||||
assert rec is not None and rec.path is not None
|
||||
assert b"TAIL" in rec.path.read_bytes(), "the late-arriving tail must be in the file"
|
||||
# The slice covers the chunks stamped T0+0.8 .. T0+3.9 (32 of them) plus the
|
||||
# one that arrived late — and that last one is where the final word of the
|
||||
# transmission lives. Without the wait it would have been cut.
|
||||
chunk_seconds = pcm.seconds(len(VOICE))
|
||||
assert duration_of(rec.path) == pytest.approx(33 * chunk_seconds, abs=0.01)
|
||||
assert duration_of(rec.path) > 32 * chunk_seconds
|
||||
assert any("Waited" in r.message and "tail" in r.message for r in caplog.records), \
|
||||
"a tail wait must be observable in the field logs"
|
||||
assert index # sanity: the pre-roll fill actually ran
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -245,6 +332,10 @@ async def test_tail_wait_is_bounded_and_warns_when_audio_never_arrives(recorder,
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_wait_when_the_buffer_already_covers_the_end(recorder, caplog):
|
||||
"""
|
||||
An audio-driven close derives its end epoch from audio that is already
|
||||
buffered, so the common path must never pay the tail wait at all.
|
||||
"""
|
||||
ingest(recorder, T0, T0 + 10.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
|
||||
@@ -261,7 +352,7 @@ async def test_no_wait_when_the_buffer_already_covers_the_end(recorder, caplog):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pre_roll_earlier_than_buffer_start_is_clamped_and_warned(recorder, caplog):
|
||||
"""A grant older than anything buffered must still produce a file, loudly."""
|
||||
"""An onset older than anything buffered must still produce a file, loudly."""
|
||||
ingest(recorder, T0 + 5.0, T0 + 10.0) # buffer only covers T0+5 onwards
|
||||
|
||||
with caplog.at_level("WARNING", logger="drb-edge-node"):
|
||||
@@ -269,36 +360,87 @@ async def test_pre_roll_earlier_than_buffer_start_is_clamped_and_warned(recorder
|
||||
rec = await recorder.stop_recording(end_epoch=T0 + 8.0)
|
||||
|
||||
assert rec is not None and rec.path is not None and rec.path.stat().st_size > 0
|
||||
assert markers(rec.path)[0] == "A0", "slice should begin at the buffer head, not fail"
|
||||
# Buffer head is T0+5.0, requested slice start is T0-PRE_ROLL: everything in
|
||||
# between is audio we can never recover, and the number must be reported.
|
||||
assert rec.clamped_seconds == pytest.approx(5.0 + PRE_ROLL_SECONDS, abs=CHUNK_INTERVAL)
|
||||
assert any("BUFFER CLAMP" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_buffered_audio_returns_none(recorder):
|
||||
async def test_no_buffered_audio_returns_none(recorder, encodes):
|
||||
await recorder.start_recording("call-1", start_epoch=T0)
|
||||
assert await recorder.stop_recording(end_epoch=T0 + 2.0) is None
|
||||
assert encodes == [], "nothing to encode means no encoder subprocess"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_start_epoch_omitted_falls_back_to_now(recorder):
|
||||
now = time.time()
|
||||
fill(recorder, now - 5.0, now)
|
||||
ingest(recorder, now - 5.0, now)
|
||||
|
||||
await recorder.start_recording("call-1")
|
||||
assert recorder._active.slice_start == pytest.approx(now - PRE_ROLL_SECONDS, abs=1.0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Encode — exactly once, at save time
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_is_encoded_exactly_once_per_call(recorder, encodes, monkeypatch):
|
||||
"""
|
||||
The old pipeline captured MP3 and then re-encoded it to trim, so every
|
||||
upload was double-encoded. Capture is PCM now and MP3 happens once, after
|
||||
trimming, at save time.
|
||||
"""
|
||||
monkeypatch.setattr(settings, "trim_silence", True)
|
||||
ingest(recorder, T0, T0 + 1.0, chunk=QUIET)
|
||||
ingest(recorder, T0 + 1.0, T0 + 3.0, chunk=VOICE)
|
||||
ingest(recorder, T0 + 3.0, T0 + 6.0, chunk=QUIET)
|
||||
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
rec = await recorder.stop_recording(end_epoch=T0 + 6.0)
|
||||
|
||||
assert rec is not None and rec.path is not None
|
||||
assert len(encodes) == 1, "exactly one encode per recording"
|
||||
encoded_audio, encoded_path = encodes[0]
|
||||
assert encoded_path == rec.path
|
||||
# What was encoded is the TRIMMED audio, not the raw slice.
|
||||
assert pcm.seconds(len(encoded_audio)) < 5.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_failed_encode_leaves_no_file_and_no_recording(recorder, monkeypatch):
|
||||
async def _fail(audio, path):
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(recorder_mod, "encode_mp3", _fail)
|
||||
ingest(recorder, T0, T0 + 5.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
|
||||
assert await recorder.stop_recording(end_epoch=T0 + 3.0) is None
|
||||
assert list(recorder._recordings_dir.glob("*.mp3")) == []
|
||||
|
||||
|
||||
def test_encoder_command_contract_matches_what_c2_expects():
|
||||
"""
|
||||
/upload has always received mono MP3 at 22050 Hz / 16 kbps, and Whisper
|
||||
consumes it downstream. The single encode must not quietly change that.
|
||||
"""
|
||||
assert recorder_mod.MP3_SAMPLE_RATE == str(pcm.SAMPLE_RATE) == "22050"
|
||||
assert recorder_mod.MP3_BITRATE == "16k"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Silence trimming and timing metadata
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _recorded(recorder, end_offset: float = 3.0):
|
||||
ingest(recorder, T0, T0 + 10.0)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
return await recorder.stop_recording(end_epoch=T0 + end_offset)
|
||||
async def _recorded(recorder, lead_silence=1.0, voice=2.0, tail_silence=1.0):
|
||||
start = T0
|
||||
ingest(recorder, start, start + lead_silence, chunk=QUIET)
|
||||
ingest(recorder, start + lead_silence, start + lead_silence + voice, chunk=VOICE)
|
||||
ingest(recorder, start + lead_silence + voice,
|
||||
start + lead_silence + voice + tail_silence, chunk=QUIET)
|
||||
await recorder.start_recording("call-1", start_epoch=start + 0.5)
|
||||
return await recorder.stop_recording(end_epoch=start + lead_silence + voice + tail_silence)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -306,12 +448,12 @@ async def test_trimming_is_off_when_the_setting_is_off(recorder, monkeypatch):
|
||||
monkeypatch.setattr(settings, "trim_silence", False)
|
||||
called = False
|
||||
|
||||
async def _never(*args, **kwargs):
|
||||
def _never(*args, **kwargs):
|
||||
nonlocal called
|
||||
called = True
|
||||
return TrimResult(path=None)
|
||||
return b"", None
|
||||
|
||||
monkeypatch.setattr(recorder_mod.audio_trim, "trim_silence", _never)
|
||||
monkeypatch.setattr(recorder_mod.audio_trim, "trim_pcm", _never)
|
||||
rec = await _recorded(recorder)
|
||||
assert rec is not None and not called
|
||||
|
||||
@@ -325,41 +467,31 @@ async def test_trim_shifts_the_audio_bounds_but_not_the_call_bounds(recorder, mo
|
||||
"""
|
||||
monkeypatch.setattr(settings, "trim_silence", True)
|
||||
|
||||
async def _trim(path, **kwargs):
|
||||
return TrimResult(path=path, lead=1.9, tail=0.4, duration_before=3.3,
|
||||
duration_after=1.0, applied=True)
|
||||
rec = await _recorded(recorder, lead_silence=1.0, voice=2.0, tail_silence=1.5)
|
||||
|
||||
monkeypatch.setattr(recorder_mod.audio_trim, "trim_silence", _trim)
|
||||
|
||||
rec = await _recorded(recorder)
|
||||
assert rec is not None and rec.path is not None
|
||||
assert rec.lead_trimmed == pytest.approx(1.9)
|
||||
assert rec.tail_trimmed == pytest.approx(0.4)
|
||||
# Untrimmed slice was [T0+0.75, T0+3.0]; the audio now starts 1.9s later and
|
||||
# ends 0.4s earlier, which is exactly what downstream needs to map an audio
|
||||
# offset back to wall clock.
|
||||
assert rec.audio_start_epoch == pytest.approx(T0 + 1.0 - PRE_ROLL_SECONDS + 1.9, abs=CHUNK_INTERVAL)
|
||||
assert rec.audio_end_epoch == pytest.approx(T0 + 3.0 - 0.4, abs=CHUNK_INTERVAL)
|
||||
assert rec.lead_trimmed > 0.0 and rec.tail_trimmed > 0.0
|
||||
guard = settings.trim_silence_guard_seconds
|
||||
# Slice began at T0+0.25; speech begins at T0+1.0, so the audio now starts
|
||||
# one guard margin before the speech.
|
||||
assert rec.audio_start_epoch == pytest.approx(T0 + 1.0 - guard, abs=3 * CHUNK_INTERVAL)
|
||||
assert rec.audio_end_epoch == pytest.approx(T0 + 3.0 + guard, abs=3 * CHUNK_INTERVAL)
|
||||
assert rec.audio_end_epoch > rec.audio_start_epoch
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_silence_recording_is_dropped_and_logged(recorder, monkeypatch, caplog):
|
||||
async def test_all_silence_recording_is_dropped_and_logged(recorder, monkeypatch, caplog, encodes):
|
||||
monkeypatch.setattr(settings, "trim_silence", True)
|
||||
seen = {}
|
||||
|
||||
async def _trim(path, **kwargs):
|
||||
seen["path"] = path
|
||||
return TrimResult(path=path, duration_before=4.0, duration_after=4.0, all_silence=True)
|
||||
|
||||
monkeypatch.setattr(recorder_mod.audio_trim, "trim_silence", _trim)
|
||||
ingest(recorder, T0, T0 + 5.0, chunk=QUIET)
|
||||
await recorder.start_recording("call-1", start_epoch=T0 + 1.0)
|
||||
|
||||
with caplog.at_level("WARNING", logger="drb-edge-node"):
|
||||
rec = await _recorded(recorder)
|
||||
rec = await recorder.stop_recording(end_epoch=T0 + 4.0)
|
||||
|
||||
assert rec is not None
|
||||
assert rec.all_silence is True
|
||||
assert rec.path is None, "an all-silence recording must not be uploaded"
|
||||
assert not seen["path"].exists(), "the file must be cleaned up, not left on disk"
|
||||
assert encodes == [], "and must not be encoded either"
|
||||
assert any("no speech" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
@@ -369,7 +501,7 @@ async def test_all_silence_recording_is_dropped_and_logged(recorder, monkeypatch
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_second_start_is_rejected_while_recording(recorder):
|
||||
fill(recorder, T0, T0 + 5.0)
|
||||
ingest(recorder, T0, T0 + 5.0)
|
||||
assert await recorder.start_recording("call-1", start_epoch=T0 + 1.0) is True
|
||||
assert await recorder.start_recording("call-2", start_epoch=T0 + 2.0) is False
|
||||
assert recorder.is_recording
|
||||
@@ -381,6 +513,21 @@ async def test_stop_without_start_is_a_noop(recorder):
|
||||
assert not recorder.is_recording
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_discard_drops_the_audio_without_writing_anything(recorder, encodes):
|
||||
"""The orphan-audio path: unattributed audio must never reach a file."""
|
||||
ingest(recorder, T0, T0 + 5.0)
|
||||
await recorder.start_recording("call-orphan", start_epoch=T0 + 1.0)
|
||||
|
||||
await recorder.discard_recording()
|
||||
|
||||
assert not recorder.is_recording
|
||||
assert encodes == []
|
||||
assert list(recorder._recordings_dir.glob("*.mp3")) == []
|
||||
# ...and the recorder is immediately reusable.
|
||||
assert await recorder.start_recording("call-next", start_epoch=T0 + 2.0) is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_split_then_immediate_restart_keeps_both_slices(recorder):
|
||||
"""
|
||||
@@ -405,22 +552,26 @@ async def test_split_then_immediate_restart_keeps_both_slices(recorder):
|
||||
# FFmpeg invocation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_ffmpeg_command_reads_pulse_and_flushes_packets(recorder):
|
||||
def test_capture_command_asks_for_raw_pcm_not_mp3(recorder):
|
||||
cmd = recorder._ffmpeg_command()
|
||||
joined = " ".join(cmd)
|
||||
|
||||
assert "-f pulse" in joined
|
||||
assert "drb_sink.monitor" in joined, "must address the monitor explicitly, not 'default'"
|
||||
# Without -flush_packets the mp3 muxer buffers 32 KB (~16 s at 16 kbps) before
|
||||
# writing, which would destroy the ring buffer's timestamp resolution.
|
||||
assert "-flush_packets" in cmd
|
||||
assert cmd[-1] == "-" and cmd[-2] == "mp3", "must emit MP3 on stdout for /upload"
|
||||
assert cmd[-1] == "-" and cmd[-2] == "s16le", "capture must emit raw PCM on stdout"
|
||||
assert "mp3" not in joined, "MP3 now happens once at save time, not in the capture"
|
||||
assert "-ar" in cmd and str(pcm.SAMPLE_RATE) in cmd
|
||||
assert "-ac" in cmd and str(pcm.CHANNELS) in cmd
|
||||
|
||||
|
||||
def test_memory_ceiling_covers_the_longest_allowed_call():
|
||||
"""The cap must bound RAM without ever being able to truncate a legal call."""
|
||||
bytes_per_second = 16_000 // 8
|
||||
assert MAX_RECORDING_BYTES >= MAX_RECORDING_SECONDS * bytes_per_second
|
||||
def test_read_chunk_is_finer_than_the_pre_roll(recorder):
|
||||
"""
|
||||
Chunk size is both the ring buffer's timestamp resolution and the window
|
||||
silence detection runs over, so it has to stay well under the pre-roll.
|
||||
"""
|
||||
chunk_seconds = pcm.seconds(recorder_mod.READ_CHUNK_BYTES)
|
||||
assert chunk_seconds < PRE_ROLL_SECONDS / 4
|
||||
assert recorder_mod.READ_CHUNK_BYTES % pcm.FRAME_BYTES == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -481,4 +632,3 @@ def test_capture_exit_with_no_stderr_captured_is_generic_warning(recorder, caplo
|
||||
recorder._log_capture_exit()
|
||||
|
||||
assert _log_levels(caplog) == ["WARNING"]
|
||||
assert MAX_RECORDING_BYTES <= 8 * 1024 * 1024, "must stay small enough for a Pi"
|
||||
|
||||
@@ -1,15 +1,30 @@
|
||||
"""
|
||||
Unit tests for the event-driven MetadataWatcher state machine.
|
||||
Unit tests for the MetadataWatcher segmentation state machine.
|
||||
|
||||
Call START comes from OP25 `call_log` entries (stamped with OP25's own
|
||||
time.time()); call END comes from the srcaddr != 0 -> srcaddr == 0 transition in
|
||||
`channel_update`. All OP25 HTTP calls are mocked — no running services required.
|
||||
TWO MODES, both covered here:
|
||||
|
||||
AUDIO MODE (production) — a recording STARTS at voice onset heard in the
|
||||
captured audio and STOPS after settings.call_silence_timeout seconds of
|
||||
silence heard in the same audio. The OP25 console supplies only the LABEL
|
||||
(talkgroup/alias/rid), resolved at CLOSE time from a rolling history, and the
|
||||
forced SPLIT when the talkgroup changes with no silence between calls.
|
||||
|
||||
CONSOLE FALLBACK (capture down) — the older state machine: `call_log` grants
|
||||
start segments, the srcaddr != 0 -> 0 edge plus call_idle_timeout ends them.
|
||||
Tests that wire no audio provider exercise this path, which is exactly the
|
||||
behaviour a node falls back to when PulseAudio is not producing audio.
|
||||
|
||||
All OP25 HTTP calls are mocked — no running services required.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from app.config import settings
|
||||
from app.internal.call_recorder import AudioActivity
|
||||
from app.internal.metadata_watcher import (
|
||||
ATTRIBUTION_LOOKAHEAD_SECONDS,
|
||||
ATTRIBUTION_LOOKBACK_SECONDS,
|
||||
MAX_SEGMENT_SECONDS,
|
||||
MetadataWatcher,
|
||||
OP25_OFFLINE_GRACE,
|
||||
)
|
||||
@@ -37,6 +52,7 @@ def clock():
|
||||
|
||||
@pytest.fixture
|
||||
def watcher(clock):
|
||||
"""Console fallback mode: no audio provider wired, capture assumed down."""
|
||||
w = MetadataWatcher()
|
||||
w._clock = clock
|
||||
w.on_call_start = AsyncMock()
|
||||
@@ -44,6 +60,58 @@ def watcher(clock):
|
||||
return w
|
||||
|
||||
|
||||
class FakeAudio:
|
||||
"""
|
||||
Stands in for CallRecorder.audio_activity.
|
||||
|
||||
Mirrors the recorder's own rule for what starts a new voice RUN: a
|
||||
non-silent chunk more than settings.call_silence_timeout after the previous
|
||||
one. Tests drive it with speak()/quiet() instead of synthesising PCM, so the
|
||||
segmentation logic is tested independently of the detector.
|
||||
"""
|
||||
|
||||
def __init__(self, clock):
|
||||
self.clock = clock
|
||||
self.capturing = True
|
||||
self.recording = False
|
||||
self.last_voice = None
|
||||
self.onset = None
|
||||
|
||||
def speak(self, at=None):
|
||||
"""Mark voice heard now (or at `at`)."""
|
||||
moment = self.clock.now if at is None else at
|
||||
if self.last_voice is None or (moment - self.last_voice) >= settings.call_silence_timeout:
|
||||
self.onset = moment
|
||||
self.last_voice = moment
|
||||
return moment
|
||||
|
||||
def __call__(self) -> AudioActivity:
|
||||
silence = 0.0 if self.last_voice is None else max(0.0, self.clock.now - self.last_voice)
|
||||
return AudioActivity(
|
||||
capturing=self.capturing,
|
||||
recording=self.recording,
|
||||
last_voice_epoch=self.last_voice,
|
||||
voice_onset_epoch=self.onset,
|
||||
silence_seconds=silence,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def audio(clock):
|
||||
return FakeAudio(clock)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def hearing(clock, audio):
|
||||
"""Audio mode: the production wiring, with a controllable audio stream."""
|
||||
w = MetadataWatcher()
|
||||
w._clock = clock
|
||||
w.on_call_start = AsyncMock()
|
||||
w.on_call_end = AsyncMock()
|
||||
w.audio_activity = audio
|
||||
return w
|
||||
|
||||
|
||||
def grant(tgid: int, time_: float, tgtag: str = "", rid: int = 101, freq: int = 851_000_000):
|
||||
"""One OP25 call_log entry (see tk_p25.log_call)."""
|
||||
return {
|
||||
@@ -195,7 +263,12 @@ async def test_op25_unreachable_does_not_start_call(watcher):
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Call end — srcaddr edge + idle timeout
|
||||
# Console fallback: call end — srcaddr edge + idle timeout
|
||||
#
|
||||
# This is the path a node uses ONLY when PulseAudio capture is not producing
|
||||
# audio. It is measurably wrong (the srcaddr edge fires mid-word) but it is all
|
||||
# there is when there is no audio to segment on, and it keeps the node
|
||||
# reporting radio activity to C2 while the audio path is broken.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -578,3 +651,499 @@ async def test_end_never_precedes_start(watcher, clock):
|
||||
|
||||
payload = watcher.on_call_end.call_args[0][0]
|
||||
assert payload["ended_at_epoch"] >= payload["started_at_epoch"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# AUDIO MODE — boundaries from the audio, label from the console
|
||||
#
|
||||
# This is the production path. Everything above this line is the fallback that
|
||||
# only runs when PulseAudio capture is down.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_onset_starts_the_recording(hearing, clock, audio):
|
||||
"""Voice onset opens the segment, and the recorder is told to slice to it."""
|
||||
onset = audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now, tgtag="Police Dispatch")],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
|
||||
assert hearing.is_active
|
||||
hearing.on_call_start.assert_called_once()
|
||||
payload = hearing.on_call_start.call_args[0][0]
|
||||
assert payload["driver"] == "audio"
|
||||
# The recorder slices the ring buffer back to THIS epoch, so it has to be
|
||||
# the audio onset, not the grant and not our detection time.
|
||||
assert payload["started_at_epoch"] == onset
|
||||
assert payload["tgid"] == 1234
|
||||
assert payload["tgid_name"] == "Police Dispatch"
|
||||
assert payload["attributed"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_grant_with_no_audio_does_not_start_a_recording(hearing, clock):
|
||||
"""
|
||||
The grant fires 0.84-1.62s before anyone speaks. Opening on it is what put
|
||||
seconds of dead air at the head of every recording.
|
||||
"""
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now, tgtag="Fire")],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
|
||||
assert not hearing.is_active
|
||||
hearing.on_call_start.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recording_closes_after_the_configured_silence(hearing, clock, audio):
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now)],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
|
||||
clock.advance(0.5)
|
||||
last_voice = audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=555)]))
|
||||
|
||||
# Quiet, but not for long enough yet.
|
||||
clock.advance(settings.call_silence_timeout - 0.5)
|
||||
await tick(hearing, update(channels=[channel()]))
|
||||
assert hearing.is_active
|
||||
hearing.on_call_end.assert_not_called()
|
||||
|
||||
clock.advance(0.6)
|
||||
await tick(hearing, update(channels=[channel()]))
|
||||
|
||||
assert not hearing.is_active
|
||||
payload = hearing.on_call_end.call_args[0][0]
|
||||
assert payload["end_reason"] == "audio_silence"
|
||||
assert payload["tgid"] == 1234
|
||||
# The audio ends where the silence run began plus the threshold, so the
|
||||
# trim can measure and strip exactly that run.
|
||||
assert payload["ended_at_epoch"] == pytest.approx(last_voice + settings.call_silence_timeout)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_false_srcaddr_drop_mid_speech_does_not_end_the_recording(hearing, clock, audio):
|
||||
"""
|
||||
THE BUG THIS REARCHITECTURE EXISTS FOR. srcaddr can reset to 0 while someone
|
||||
is still talking; under the old design that started the idle timer and the
|
||||
window closed on top of live speech (recording 0ff35b20: "-1.61s lead,
|
||||
-0.00s tail" — nothing left to trim because the cut landed mid-word).
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now)],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
call_id = hearing.active_call_id
|
||||
|
||||
# The control channel now says the call is over. It is wrong; the audio
|
||||
# keeps arriving. This runs well past call_idle_timeout, which is what
|
||||
# would have closed the segment before.
|
||||
for _ in range(10):
|
||||
clock.advance(0.5)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=0, hold_tgid=1234)]))
|
||||
assert hearing.is_active, "a false srcaddr drop must never end a recording"
|
||||
|
||||
assert (clock.now - hearing._started_at) > settings.call_idle_timeout
|
||||
assert hearing.active_call_id == call_id
|
||||
hearing.on_call_end.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_silence_close_logs_the_measured_trailing_silence(hearing, clock, audio, caplog):
|
||||
"""
|
||||
call_silence_timeout can only be tuned from the real trailing silence, so
|
||||
the measured number has to reach the log — the audio-mode counterpart of
|
||||
the "measured control-channel idle" line.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
|
||||
with caplog.at_level("INFO", logger="drb-edge-node"):
|
||||
clock.advance(settings.call_silence_timeout + 0.25)
|
||||
await tick(hearing, update(channels=[channel()]))
|
||||
|
||||
lines = [r.message for r in caplog.records if "measured trailing silence" in r.message]
|
||||
assert lines, "audio closes must log the measured silence for later tuning"
|
||||
assert "3.25s" in lines[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_same_tgid_grant_continues_one_audio_recording(hearing, clock, audio):
|
||||
"""Back-and-forth on one talkgroup must stay a single call/recording."""
|
||||
onset = audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now)],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
call_id = hearing.active_call_id
|
||||
|
||||
# The other party keys up on the SAME tgid while audio is still flowing.
|
||||
clock.advance(1.0)
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now)],
|
||||
channels=[channel(tgid=1234, srcaddr=777)],
|
||||
))
|
||||
|
||||
assert hearing.active_call_id == call_id, "same tgid must not open a new call"
|
||||
hearing.on_call_start.assert_called_once()
|
||||
hearing.on_call_end.assert_not_called()
|
||||
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update(channels=[channel()]))
|
||||
|
||||
hearing.on_call_end.assert_called_once()
|
||||
payload = hearing.on_call_end.call_args[0][0]
|
||||
assert payload["call_id"] == call_id
|
||||
assert payload["started_at_epoch"] == onset
|
||||
assert payload["transmissions"] == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_different_tgid_splits_even_with_no_silence_between(hearing, clock, audio):
|
||||
"""
|
||||
Back-to-back calls on two talkgroups with no gap. Pure audio segmentation
|
||||
would merge them into ONE file under ONE label, which corrupts correlation.
|
||||
The console talkgroup change has to force the cut.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1111, clock.now, tgtag="Fire")],
|
||||
channels=[channel(tgid=1111, srcaddr=1)],
|
||||
))
|
||||
first_id = hearing.active_call_id
|
||||
|
||||
clock.advance(2.0)
|
||||
audio.speak() # still talking — no silence anywhere in this test
|
||||
split = clock.now
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(2222, split, tgtag="EMS")],
|
||||
channels=[channel(tgid=2222, srcaddr=2)],
|
||||
))
|
||||
|
||||
hearing.on_call_end.assert_called_once()
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["call_id"] == first_id
|
||||
assert ended["tgid"] == 1111
|
||||
assert ended["end_reason"] == "tgid_change"
|
||||
# Padded past the split: buffered audio lags the control channel, so cutting
|
||||
# at the exact grant timestamp clipped the outgoing call's last words. The
|
||||
# overlap between the two slices is correct.
|
||||
assert ended["ended_at_epoch"] == split + settings.call_tail_pad_seconds
|
||||
|
||||
assert hearing.is_active and hearing.current_tgid == 2222
|
||||
assert hearing.active_call_id != first_id
|
||||
assert hearing.on_call_start.call_count == 2
|
||||
assert hearing.on_call_start.call_args[0][0]["started_at_epoch"] == split
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_an_unlogged_tgid_change_also_splits_and_reopens(hearing, clock, audio):
|
||||
"""
|
||||
OP25's call_log deque is capped at 10, so grants get dropped. If our only
|
||||
receiver is plainly on another talkgroup the segment is over — and in audio
|
||||
mode a new one must open immediately or the audio would be dropped on the
|
||||
floor until the next voice run.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1111, clock.now)],
|
||||
channels=[channel(tgid=1111, srcaddr=1)],
|
||||
))
|
||||
first_id = hearing.active_call_id
|
||||
|
||||
clock.advance(1.0)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=3333, srcaddr=7)])) # no call_log
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["call_id"] == first_id
|
||||
assert ended["end_reason"] == "tgid_change_unlogged"
|
||||
assert hearing.is_active and hearing.current_tgid == 3333
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_max_length_backstop_closes_and_reopens(hearing, clock, audio):
|
||||
"""
|
||||
A talkgroup that never goes quiet must not produce an unbounded recording —
|
||||
but the audio must not be dropped either, so the segment is immediately
|
||||
reopened rather than simply abandoned.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
first_id = hearing.active_call_id
|
||||
|
||||
clock.advance(MAX_SEGMENT_SECONDS / 2)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=5)]))
|
||||
assert hearing.active_call_id == first_id
|
||||
|
||||
clock.advance(MAX_SEGMENT_SECONDS / 2 + 1)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=5)]))
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["call_id"] == first_id
|
||||
assert ended["end_reason"] == "max_length"
|
||||
assert hearing.is_active, "a still-live transmission must not be dropped at the cap"
|
||||
assert hearing.active_call_id != first_id
|
||||
assert hearing.on_call_start.call_count == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Attribution: resolved at close, from a bounded rolling console history
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_orphan_audio_is_discarded_flagged_and_counted(hearing, clock, audio, caplog):
|
||||
"""
|
||||
Audio with no console talkgroup anywhere near it — Liquidsoap fallback, a
|
||||
test tone, stray noise, a dropped call_log. It must be impossible to miss
|
||||
and must never be uploaded: an untagged call poisons correlation.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update()) # console says nothing at all
|
||||
|
||||
assert hearing.is_active
|
||||
started = hearing.on_call_start.call_args[0][0]
|
||||
assert started["tgid"] is None
|
||||
assert started["attributed"] is False
|
||||
|
||||
with caplog.at_level("ERROR", logger="drb-edge-node"):
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update())
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["attributed"] is False
|
||||
assert ended["tgid"] is None
|
||||
assert hearing.unattributed_segments == 1
|
||||
assert any("ORPHAN AUDIO" in r.message for r in caplog.records), \
|
||||
"unattributed audio must be loud in the logs, not silent"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_grant_before_audio_onset_still_attributes_the_recording(hearing, clock, audio):
|
||||
"""
|
||||
The common ordering: the console grants the channel, then 0.84-1.62s later
|
||||
(plus pipeline lag) the audio shows up. The lookback has to cover it.
|
||||
"""
|
||||
grant_time = clock.now
|
||||
await tick(hearing, update(call_log=[grant(1234, grant_time, tgtag="Fire")]))
|
||||
assert not hearing.is_active
|
||||
|
||||
clock.advance(2.0)
|
||||
audio.speak()
|
||||
await tick(hearing, update()) # console says nothing NOW
|
||||
|
||||
assert hearing.is_active
|
||||
payload = hearing.on_call_start.call_args[0][0]
|
||||
assert payload["tgid"] == 1234
|
||||
assert payload["tgid_name"] == "Fire"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_grant_after_audio_onset_attributes_the_recording_late(hearing, clock, audio):
|
||||
"""
|
||||
There is no guaranteed ordering: the console is polled every 500ms, so the
|
||||
grant can land after voice onset. The segment starts unattributed and picks
|
||||
the talkgroup up part-way through — expected, and fine.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update())
|
||||
|
||||
assert hearing.is_active
|
||||
assert hearing.on_call_start.call_args[0][0]["attributed"] is False
|
||||
|
||||
clock.advance(0.5)
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now, tgtag="EMS")]))
|
||||
assert hearing.current_tgid == 1234
|
||||
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update())
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["attributed"] is True
|
||||
assert ended["tgid"] == 1234
|
||||
assert ended["tgid_name"] == "EMS"
|
||||
assert hearing.unattributed_segments == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_attribution_resolves_from_a_channel_row_when_the_grant_was_dropped(hearing, clock, audio):
|
||||
"""A dropped grant is survivable: an active channel row names the talkgroup."""
|
||||
audio.speak()
|
||||
await tick(hearing, update())
|
||||
assert hearing.on_call_start.call_args[0][0]["tgid"] is None
|
||||
|
||||
clock.advance(0.5)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[channel(tgid=4321, srcaddr=99, tag="Sheriff")]))
|
||||
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update())
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["tgid"] == 4321
|
||||
assert ended["attributed"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_console_activity_outside_the_tolerance_does_not_attribute(hearing, clock, audio):
|
||||
"""
|
||||
The tolerance is deliberately bounded. A grant from long before the audio is
|
||||
not evidence about this audio, and borrowing it would be worse than
|
||||
admitting the audio is unattributed.
|
||||
"""
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
|
||||
clock.advance(ATTRIBUTION_LOOKBACK_SECONDS + 5.0)
|
||||
audio.speak()
|
||||
await tick(hearing, update())
|
||||
|
||||
assert hearing.is_active
|
||||
assert hearing.on_call_start.call_args[0][0]["tgid"] is None
|
||||
assert ATTRIBUTION_LOOKAHEAD_SECONDS > 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_directly_stated_talkgroup_is_not_overruled_at_close(hearing, clock, audio, caplog):
|
||||
"""
|
||||
The attribution window extends past the audio on purpose, so a neighbouring
|
||||
call's console activity can fall inside it. An inference over that window
|
||||
must never overrule a talkgroup the console stated outright — but the
|
||||
overlap does mean the split logic missed something, so it is logged.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1111, clock.now, tgtag="Fire")],
|
||||
channels=[channel(tgid=1111, srcaddr=1)],
|
||||
))
|
||||
|
||||
# A second talkgroup is busy on ANOTHER receiver, so no split fires, and it
|
||||
# produces more console observations than ours did.
|
||||
for _ in range(4):
|
||||
clock.advance(0.5)
|
||||
audio.speak()
|
||||
await tick(hearing, update(channels=[
|
||||
channel(tgid=1111, srcaddr=0, hold_tgid=1111),
|
||||
channel(tgid=2222, srcaddr=7),
|
||||
]))
|
||||
|
||||
with caplog.at_level("WARNING", logger="drb-edge-node"):
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update())
|
||||
|
||||
ended = hearing.on_call_end.call_args[0][0]
|
||||
assert ended["tgid"] == 1111, "the console said 1111 directly; inference must not overrule it"
|
||||
assert any("split logic should have fired" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_same_voice_run_does_not_reopen_a_second_recording(hearing, clock, audio):
|
||||
"""A closed run must stay closed; only NEW audio opens the next segment."""
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
|
||||
clock.advance(settings.call_silence_timeout + 0.5)
|
||||
await tick(hearing, update())
|
||||
assert not hearing.is_active
|
||||
|
||||
# Several more quiet polls must not resurrect it.
|
||||
for _ in range(3):
|
||||
clock.advance(0.5)
|
||||
await tick(hearing, update())
|
||||
assert not hearing.is_active
|
||||
assert hearing.on_call_start.call_count == 1
|
||||
|
||||
# ...but the next voice run does open a new segment.
|
||||
clock.advance(1.0)
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
assert hearing.is_active
|
||||
assert hearing.on_call_start.call_count == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mode changes: capture loss, console loss
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_losing_capture_closes_an_audio_segment(hearing, clock, audio):
|
||||
"""
|
||||
An audio-driven segment must never hang open when the audio stops arriving:
|
||||
with no chunks there is no silence to detect, so the mode change is the
|
||||
thing that has to close it.
|
||||
"""
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
assert hearing.is_active
|
||||
|
||||
audio.capturing = False
|
||||
clock.advance(0.5)
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=555)]))
|
||||
|
||||
assert not hearing.is_active
|
||||
assert hearing.on_call_end.call_args[0][0]["end_reason"] == "capture_lost"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_console_segmentation_takes_over_while_capture_is_down(hearing, clock, audio):
|
||||
"""
|
||||
No audio means no recordings, but the node must still report real radio
|
||||
activity to C2 rather than going silent about it.
|
||||
"""
|
||||
audio.capturing = False
|
||||
|
||||
await tick(hearing, update(
|
||||
call_log=[grant(1234, clock.now, tgtag="Police")],
|
||||
channels=[channel(tgid=1234, srcaddr=555)],
|
||||
))
|
||||
assert hearing.is_active
|
||||
assert hearing.on_call_start.call_args[0][0]["driver"] == "console"
|
||||
|
||||
clock.advance(0.5)
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=0, hold_tgid=1234)]))
|
||||
clock.advance(settings.call_idle_timeout + 0.5)
|
||||
await tick(hearing, update(channels=[channel()]))
|
||||
|
||||
assert not hearing.is_active
|
||||
assert hearing.on_call_end.call_args[0][0]["end_reason"] == "idle_timeout"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_op25_unreachable_still_closes_an_audio_segment(hearing, clock, audio):
|
||||
"""Without the console there is no attribution, so there is nothing to keep open."""
|
||||
audio.speak()
|
||||
await tick(hearing, update(call_log=[grant(1234, clock.now)]))
|
||||
|
||||
clock.advance(OP25_OFFLINE_GRACE + 0.5)
|
||||
audio.speak()
|
||||
await tick(hearing, None)
|
||||
|
||||
assert not hearing.is_active
|
||||
assert hearing.on_call_end.call_args[0][0]["end_reason"] == "op25_unreachable"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_console_history_is_bounded(hearing, clock, audio):
|
||||
"""A rolling history that grows without limit would be a slow memory leak."""
|
||||
from app.internal.metadata_watcher import CONSOLE_HISTORY_MAX, CONSOLE_HISTORY_SECONDS
|
||||
|
||||
for _ in range(CONSOLE_HISTORY_MAX + 200):
|
||||
clock.advance(0.05)
|
||||
await tick(hearing, update(channels=[channel(tgid=1234, srcaddr=5)]))
|
||||
|
||||
assert len(hearing._console) <= CONSOLE_HISTORY_MAX
|
||||
|
||||
# ...and old entries age out even when the count is low.
|
||||
clock.advance(CONSOLE_HISTORY_SECONDS + 1)
|
||||
await tick(hearing, update())
|
||||
assert all(e.epoch >= clock.now - CONSOLE_HISTORY_SECONDS for e in hearing._console)
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
"""
|
||||
Unit tests for the raw-PCM primitives that silence detection rests on.
|
||||
|
||||
The whole audio-driven design depends on one field observation: between
|
||||
transmissions the captured stream is DIGITAL silence (a PulseAudio null-sink
|
||||
monitor), measured at about -91 dBFS — one least-significant bit — not an analog
|
||||
noise floor. These tests pin that assumption down in code: a 1-LSB "silent"
|
||||
buffer must read as silence at every sane threshold, and speech-level audio must
|
||||
never read as silence.
|
||||
"""
|
||||
from array import array
|
||||
|
||||
import pytest
|
||||
|
||||
from app.internal import pcm
|
||||
|
||||
|
||||
def tone(level: int, samples: int = 1024) -> bytes:
|
||||
"""A square wave at +/-level, so RMS == level exactly."""
|
||||
return array("h", [level, -level] * (samples // 2)).tobytes()
|
||||
|
||||
|
||||
def zeros(samples: int = 1024) -> bytes:
|
||||
return b"\x00\x00" * samples
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Format arithmetic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_capture_format_is_22050_mono_16bit():
|
||||
"""MP3_SAMPLE_RATE in call_recorder must match, or the encode resamples."""
|
||||
assert (pcm.SAMPLE_RATE, pcm.CHANNELS, pcm.SAMPLE_WIDTH) == (22050, 1, 2)
|
||||
assert pcm.BYTES_PER_SECOND == 44100
|
||||
|
||||
|
||||
def test_seconds_and_byte_offset_round_trip():
|
||||
assert pcm.seconds(pcm.BYTES_PER_SECOND) == pytest.approx(1.0)
|
||||
assert pcm.byte_offset(1.0) == pcm.BYTES_PER_SECOND
|
||||
assert pcm.byte_offset(0.5) == 22050
|
||||
|
||||
|
||||
def test_byte_offset_is_always_sample_aligned():
|
||||
"""A byte offset that splits a sample would shift every later sample."""
|
||||
for seconds in (0.001, 0.0137, 0.25, 1.7):
|
||||
assert pcm.byte_offset(seconds) % pcm.FRAME_BYTES == 0
|
||||
|
||||
|
||||
def test_align_drops_a_trailing_half_sample():
|
||||
assert pcm.align(9) == 8
|
||||
assert pcm.align(0) == 0
|
||||
assert pcm.align(-4) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Silence detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_exact_digital_zero_is_silence():
|
||||
assert pcm.is_all_zero(zeros())
|
||||
assert pcm.rms_dbfs(zeros()) == pcm.SILENT_DBFS
|
||||
assert pcm.is_silent(zeros(), -50.0)
|
||||
assert pcm.is_silent(zeros(), -90.0)
|
||||
|
||||
|
||||
def test_one_lsb_of_dither_is_the_measured_field_floor():
|
||||
"""
|
||||
The gap between transmissions measures ~-91 dBFS on a live node, which is
|
||||
exactly 20*log10(1/32768) — a single LSB. It must read as silence at any
|
||||
threshold we would ever configure.
|
||||
"""
|
||||
floor = tone(1)
|
||||
assert pcm.rms_dbfs(floor) == pytest.approx(-90.3, abs=0.2)
|
||||
assert pcm.is_silent(floor, -50.0)
|
||||
assert pcm.is_silent(floor, -70.0)
|
||||
assert not pcm.is_silent(floor, -95.0), "an absurd threshold should still be honoured"
|
||||
|
||||
|
||||
def test_speech_level_audio_is_never_silence():
|
||||
"""Speech on the live node averages about -18 dBFS."""
|
||||
speech = tone(4096) # -18.06 dBFS
|
||||
assert pcm.rms_dbfs(speech) == pytest.approx(-18.06, abs=0.1)
|
||||
assert not pcm.is_silent(speech, -50.0)
|
||||
assert not pcm.is_silent(speech, -40.0)
|
||||
|
||||
|
||||
def test_threshold_is_honoured_exactly_at_the_boundary():
|
||||
# RMS 104 -> -49.96 dBFS, just above a -50 threshold.
|
||||
assert not pcm.is_silent(tone(104), -50.0)
|
||||
# RMS 100 -> -50.30 dBFS, just below it.
|
||||
assert pcm.is_silent(tone(100), -50.0)
|
||||
|
||||
|
||||
def test_empty_buffer_counts_as_silence():
|
||||
"""
|
||||
"No audio arrived" must never read as "someone is talking" — otherwise a
|
||||
stalled capture would hold a segment open forever.
|
||||
"""
|
||||
assert pcm.is_silent(b"", -50.0)
|
||||
assert pcm.rms_dbfs(b"") == pcm.SILENT_DBFS
|
||||
|
||||
|
||||
def test_full_scale_is_zero_dbfs():
|
||||
assert pcm.rms_dbfs(tone(32767)) == pytest.approx(0.0, abs=0.001)
|
||||
|
||||
|
||||
def test_a_trailing_odd_byte_does_not_break_detection():
|
||||
"""Short reads at EOF can leave half a sample; it must be dropped, not skew."""
|
||||
assert not pcm.is_silent(tone(8000) + b"\x00", -50.0)
|
||||
assert pcm.samples(zeros(4) + b"\x01").itemsize == 2
|
||||
assert len(pcm.samples(zeros(4) + b"\x01")) == 4
|
||||
|
||||
|
||||
def test_rms_attenuates_an_isolated_click_the_way_peak_would_not():
|
||||
"""
|
||||
Why RMS and not peak. A single stray sample in an otherwise silent window is
|
||||
a decoder click, not speech. Peak would score it at its full amplitude and
|
||||
hold a recording open; RMS spreads it over the window and divides it down by
|
||||
sqrt(N) — 30 dB for a 1024-sample window.
|
||||
|
||||
A full-scale click still reads as signal even after that attenuation, which
|
||||
is deliberate: at worst it extends a recording by the silence timeout, and
|
||||
the trim strips the result before upload. Under-detecting speech is the
|
||||
failure that loses words permanently.
|
||||
"""
|
||||
moderate = bytearray(zeros(1024))
|
||||
moderate[0:2] = array("h", [1000]).tobytes() # -30 dBFS peak
|
||||
assert pcm.rms_dbfs(bytes(moderate)) == pytest.approx(-60.4, abs=0.2)
|
||||
assert pcm.is_silent(bytes(moderate), -50.0)
|
||||
|
||||
full_scale = bytearray(zeros(1024))
|
||||
full_scale[0:2] = array("h", [32767]).tobytes()
|
||||
assert pcm.rms_dbfs(bytes(full_scale)) == pytest.approx(-30.1, abs=0.2)
|
||||
assert not pcm.is_silent(bytes(full_scale), -50.0)
|
||||
Reference in New Issue
Block a user