call_recorder: spool failed uploads to disk and retry instead of deleting (node-26#44)

upload_recording deleted the file in `finally` even when the POST failed, so
any C2 outage, 5xx or timeout lost the call's audio for good.

- Delete only after a 2xx that carries a URL (C2 answers 200 {url: null}
  when its GCS write fails).
- Retryable failures (network, timeout, 5xx, 401/408/429, 2xx-no-url) move
  the file plus a JSON sidecar (call_id, talkgroup, system_id, audio epochs)
  into <recordings>/spool. A background task retries with exponential
  backoff and drains the spool on startup. Other 4xx are deleted with an
  ERROR naming the status.
- Bounded: UPLOAD_SPOOL_MAX_BYTES (512 MiB) and UPLOAD_SPOOL_MAX_AGE_HOURS
  (72), oldest evicted first, since a full host disk stops recording (#41).
- Crash-safe: the sidecar is fsynced and written before the audio moves in;
  a lost or corrupt sidecar is rebuilt from the filename instead of
  deleting the audio.
- The inline attempt uses UPLOAD_INLINE_TIMEOUT_SECONDS (30), so a
  black-holed C2 no longer stalls the segmenter for 120 s per call.
- call_end carries audio_upload=spooled|rejected|lost; a new API key wakes
  the spool; /api/status and the checkin report spool depth and the last
  upload outcome (toward #40).

Reviewed by drb-audio-review. Its two server-side findings (late uploads
correlated against now; no /upload idempotency) are filed separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-27 20:14:36 -04:00
co-authored by Claude Opus 5.5
parent 1b4fcefbf2
commit 6b884d5433
8 changed files with 752 additions and 43 deletions
+16 -2
View File
@@ -111,8 +111,20 @@ async def on_call_end(data: dict):
)
if audio_url:
data["audio_url"] = audio_url
else:
logger.error(f"Audio upload failed for call {data['call_id']}. Verify C2_URL and Node API Key.")
elif settings.c2_url:
last = call_recorder.spool.last_result or {}
if last.get("kind") == "lost":
data["audio_upload"] = "lost"
elif last.get("kind") == "retry":
# The file is in the upload spool and will be retried; C2 attaches
# it to this call_id whenever it lands (node-26#44).
data["audio_upload"] = "spooled"
else:
data["audio_upload"] = "rejected"
logger.error(
f"Audio upload for call {data['call_id']} was rejected ({last.get('detail')}). "
"Verify the node is approved and its API key is current."
)
elif recording is not None and recording.all_silence:
# Explicit policy: an all-silence recording is not uploaded. It has no
# transcript value and silence is what makes Whisper invent text.
@@ -179,6 +191,8 @@ async def on_api_key(payload: dict):
if key:
credentials.save_api_key(key)
mqtt_manager.update_api_key(key)
# Uploads spooled on a 401 can go now rather than at their next backoff.
call_recorder.spool.wake()
logger.info("Node API key received and saved.")