call_recorder: spool failed uploads to disk and retry instead of deleting (node-26#44)
upload_recording deleted the file in `finally` even when the POST failed, so
any C2 outage, 5xx or timeout lost the call's audio for good.
- Delete only after a 2xx that carries a URL (C2 answers 200 {url: null}
when its GCS write fails).
- Retryable failures (network, timeout, 5xx, 401/408/429, 2xx-no-url) move
the file plus a JSON sidecar (call_id, talkgroup, system_id, audio epochs)
into <recordings>/spool. A background task retries with exponential
backoff and drains the spool on startup. Other 4xx are deleted with an
ERROR naming the status.
- Bounded: UPLOAD_SPOOL_MAX_BYTES (512 MiB) and UPLOAD_SPOOL_MAX_AGE_HOURS
(72), oldest evicted first, since a full host disk stops recording (#41).
- Crash-safe: the sidecar is fsynced and written before the audio moves in;
a lost or corrupt sidecar is rebuilt from the filename instead of
deleting the audio.
- The inline attempt uses UPLOAD_INLINE_TIMEOUT_SECONDS (30), so a
black-holed C2 no longer stalls the segmenter for 120 s per call.
- call_end carries audio_upload=spooled|rejected|lost; a new API key wakes
the spool; /api/status and the checkin report spool depth and the last
upload outcome (toward #40).
Reviewed by drb-audio-review. Its two server-side findings (late uploads
correlated against now; no /upload idempotency) are filed separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1b4fcefbf2
commit
6b884d5433
@@ -111,8 +111,20 @@ async def on_call_end(data: dict):
|
||||
)
|
||||
if audio_url:
|
||||
data["audio_url"] = audio_url
|
||||
else:
|
||||
logger.error(f"Audio upload failed for call {data['call_id']}. Verify C2_URL and Node API Key.")
|
||||
elif settings.c2_url:
|
||||
last = call_recorder.spool.last_result or {}
|
||||
if last.get("kind") == "lost":
|
||||
data["audio_upload"] = "lost"
|
||||
elif last.get("kind") == "retry":
|
||||
# The file is in the upload spool and will be retried; C2 attaches
|
||||
# it to this call_id whenever it lands (node-26#44).
|
||||
data["audio_upload"] = "spooled"
|
||||
else:
|
||||
data["audio_upload"] = "rejected"
|
||||
logger.error(
|
||||
f"Audio upload for call {data['call_id']} was rejected ({last.get('detail')}). "
|
||||
"Verify the node is approved and its API key is current."
|
||||
)
|
||||
elif recording is not None and recording.all_silence:
|
||||
# Explicit policy: an all-silence recording is not uploaded. It has no
|
||||
# transcript value and silence is what makes Whisper invent text.
|
||||
@@ -179,6 +191,8 @@ async def on_api_key(payload: dict):
|
||||
if key:
|
||||
credentials.save_api_key(key)
|
||||
mqtt_manager.update_api_key(key)
|
||||
# Uploads spooled on a 401 can go now rather than at their next backoff.
|
||||
call_recorder.spool.wake()
|
||||
logger.info("Node API key received and saved.")
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user