call_recorder: spool failed uploads to disk and retry instead of deleting (node-26#44)
upload_recording deleted the file in `finally` even when the POST failed, so
any C2 outage, 5xx or timeout lost the call's audio for good.
- Delete only after a 2xx that carries a URL (C2 answers 200 {url: null}
when its GCS write fails).
- Retryable failures (network, timeout, 5xx, 401/408/429, 2xx-no-url) move
the file plus a JSON sidecar (call_id, talkgroup, system_id, audio epochs)
into <recordings>/spool. A background task retries with exponential
backoff and drains the spool on startup. Other 4xx are deleted with an
ERROR naming the status.
- Bounded: UPLOAD_SPOOL_MAX_BYTES (512 MiB) and UPLOAD_SPOOL_MAX_AGE_HOURS
(72), oldest evicted first, since a full host disk stops recording (#41).
- Crash-safe: the sidecar is fsynced and written before the audio moves in;
a lost or corrupt sidecar is rebuilt from the filename instead of
deleting the audio.
- The inline attempt uses UPLOAD_INLINE_TIMEOUT_SECONDS (30), so a
black-holed C2 no longer stalls the segmenter for 120 s per call.
- call_end carries audio_upload=spooled|rejected|lost; a new API key wakes
the spool; /api/status and the checkin report spool depth and the last
upload outcome (toward #40).
Reviewed by drb-audio-review. Its two server-side findings (late uploads
correlated against now; no /upload idempotency) are filed separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1b4fcefbf2
commit
6b884d5433
@@ -99,6 +99,15 @@ TRIM_SILENCE_THRESHOLD_DB=-40
|
||||
# Seconds of audio kept either side of detected speech.
|
||||
TRIM_SILENCE_GUARD_SECONDS=0.25
|
||||
|
||||
# Upload spool: a recording whose upload to C2 fails is kept on disk and
|
||||
# retried, never deleted (default dir: <recordings>/spool). Bounded so a long
|
||||
# C2 outage cannot fill the host disk; the oldest recordings go first.
|
||||
# UPLOAD_SPOOL_DIR=/recordings/spool
|
||||
UPLOAD_SPOOL_MAX_BYTES=536870912
|
||||
UPLOAD_SPOOL_MAX_AGE_HOURS=72
|
||||
UPLOAD_RETRY_MIN_SECONDS=15
|
||||
UPLOAD_RETRY_MAX_SECONDS=600
|
||||
|
||||
# OP25 container (usually no need to change)
|
||||
OP25_API_URL=http://localhost:8001
|
||||
OP25_TERMINAL_URL=http://localhost:8081
|
||||
|
||||
Reference in New Issue
Block a user