call_recorder: spool failed uploads to disk and retry instead of deleting (node-26#44)

upload_recording deleted the file in `finally` even when the POST failed, so
any C2 outage, 5xx or timeout lost the call's audio for good.

- Delete only after a 2xx that carries a URL (C2 answers 200 {url: null}
  when its GCS write fails).
- Retryable failures (network, timeout, 5xx, 401/408/429, 2xx-no-url) move
  the file plus a JSON sidecar (call_id, talkgroup, system_id, audio epochs)
  into <recordings>/spool. A background task retries with exponential
  backoff and drains the spool on startup. Other 4xx are deleted with an
  ERROR naming the status.
- Bounded: UPLOAD_SPOOL_MAX_BYTES (512 MiB) and UPLOAD_SPOOL_MAX_AGE_HOURS
  (72), oldest evicted first, since a full host disk stops recording (#41).
- Crash-safe: the sidecar is fsynced and written before the audio moves in;
  a lost or corrupt sidecar is rebuilt from the filename instead of
  deleting the audio.
- The inline attempt uses UPLOAD_INLINE_TIMEOUT_SECONDS (30), so a
  black-holed C2 no longer stalls the segmenter for 120 s per call.
- call_end carries audio_upload=spooled|rejected|lost; a new API key wakes
  the spool; /api/status and the checkin report spool depth and the last
  upload outcome (toward #40).

Reviewed by drb-audio-review. Its two server-side findings (late uploads
correlated against now; no /upload idempotency) are filed separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-27 20:14:36 -04:00
co-authored by Claude Opus 5.5
parent 1b4fcefbf2
commit 6b884d5433
8 changed files with 752 additions and 43 deletions
+9
View File
@@ -99,6 +99,15 @@ TRIM_SILENCE_THRESHOLD_DB=-40
# Seconds of audio kept either side of detected speech.
TRIM_SILENCE_GUARD_SECONDS=0.25
# Upload spool: a recording whose upload to C2 fails is kept on disk and
# retried, never deleted (default dir: <recordings>/spool). Bounded so a long
# C2 outage cannot fill the host disk; the oldest recordings go first.
# UPLOAD_SPOOL_DIR=/recordings/spool
UPLOAD_SPOOL_MAX_BYTES=536870912
UPLOAD_SPOOL_MAX_AGE_HOURS=72
UPLOAD_RETRY_MIN_SECONDS=15
UPLOAD_RETRY_MAX_SECONDS=600
# OP25 container (usually no need to change)
OP25_API_URL=http://localhost:8001
OP25_TERMINAL_URL=http://localhost:8081