/recordings is a bind mount of the host checkout, so shutil.disk_usage on it
is the disk the recorder writes to and Docker fills. Explicit nulls on error
so C2 never keeps a stale reading. Server half: server-26 feat/41-node-disk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
node-002's root disk hit 100% (39.5 GB build cache + 16 GB unused images)
and the node silently stopped recording for 17 days: BUILD=1 runs
`docker compose build` on every update and nothing ever pruned the cache;
the existing prunes only drop dangling images and old src- pins.
After `up -d`: `builder prune -af --filter until=72h` and
`image prune -af --filter until=168h`. Neither touches anything a running
container uses; the age filters keep the next build's cache warm.
Disk-free reporting to the portal (the other half of #41) is not in this
change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Permissions are pre-checked now, so a connect timeout is the voice handshake
(pre-2.7 discord.py: DAVE close 4017 retried to the 30s timeout) or UDP egress,
not permissions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live symptom: the radio bot shows "Listening to <system>" but is not in the
voice channel.
- Presence was set once on the first successful join and never cleared, so a
bot whose voice dropped (or whose rejoin kept failing) still looked healthy.
Presence is now synced at the end of every reconcile: set only while a voice
connection for that bot is up, cleared otherwise.
- Pre-check the bot's permissions (View Channel, Connect, Speak) and the
channel's user limit before joining. Discord silently ignores a join the bot
may not make; discord.py then times out after 30s and the admin page read
"Could not connect to the voice channel (TimeoutError: )". Now: code
missing_permissions / channel_full with the missing permission named.
- A connect timeout gets its own code (connect_timeout) and readable text.
- requirements: discord.py[voice]>=2.7 — Discord has required DAVE E2EE voice
since 2026-03-01 and 2.7 is the first discord.py with it (the extra pulls
davey). Today's resolution was already 2.7.1 + davey 0.1.6; this stops an
older resolution from ever being accepted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_host_updater: stub git/docker now model cat-file/show/rev-parse
--verify and compose config/image inspect. The two update-path tests
were failing on Linux CI (skipped locally on macOS, no flock). Adds
tests for pull-before-checkout + .env pins, a missing pinned image
leaving the node unchanged, and a pre-pinning target still updating.
- drb-update.sh: load image-tags.sh via a variable + eval and confirm the
functions exist; `source <(git show ...)` "succeeded" with nothing
defined when git show failed, wedging every later update.
- install.sh: skip pinning (warn) when the checked-out ref predates
scripts/image-tags.sh instead of dying on the missing file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rewritten as RadioManager reconciling C2's retained
nodes/{id}/discord/desired (token ids) and publishing
nodes/{id}/discord/actual {applied_seq, sessions, streams}.
- One gateway session per leased token, one voice connection per
(token, guild); a token no stream needs is logged OUT (client.close()),
which is what lets C2 free it safely (item 1).
- Watchdog is a separate reconcile loop; recovering a dead session can no
longer cancel it (item 2). Presence kept across rejoins.
- No message-content intent and no @bot leave/join mention controls (item 3).
- Local dashboard: /api/discord/join returns 409 with an explanation instead
of a TypeError 500; /api/discord/leave pauses locally; GET /api/discord
shows state (item 4). Join/leave results are reported to C2 (item 5).
- Token strings fetched over HTTPS from C2 (lease holder only), memory only.
- MQTT-loss fail-safe: logs every bot out after
DISCORD_MQTT_LOSS_STOP_SECONDS (300, below C2's 600s lease grace).
- Legacy discord_join/discord_leave MQTT commands are ignored.
- Audio path unchanged: FFmpeg -f pulse on settings.pulse_source per
connection, never Icecast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
op25-container
- generate-config takes p25_mode "conventional" + conv_channels [{frequency,
name, nac}]. It builds fixed fsk4 channels (up to 2 on one non-tunable SDR,
DC spike kept off the channels) with no control channel to hunt. Two
channels send audio to UDP 23456/23457, and audio.py -2 + amplify(2) keeps
levels.
- tk_drb_conv.py (a tk_p25.rx_ctl subclass, copied into OP25 apps) reports
voice / NAC / encryption (LC service options + LDU2 ESS algid/keyid) for
conventional channels and applies the per-channel NAC filter, which stock
tk_p25 does not. Analog uses it too.
- Trunked P25 config is unchanged when p25_mode is absent.
drb-edge-node
- radio_config: system kind from type + config.mode (missing = trunked).
- conventional.py normalizes the console: a distinctive LC TGID, else the
channel's last TGID (30 s), else a synthetic per-channel id (freq/100 Hz).
Encrypted rows are never talkers. With exactly one conventional channel,
orphan audio is attributed to it at close, which fixes analog recording
(#20). The synthetic id and the real TGID on one channel never split.
- Health/dashboard: P25 conventional listen tab (/api/listen p25conv), and
conventional decode + encryption checks. The checkin carries radio_channels.
- Tests: edge 296 pass; op25-container 18 (new CI job). Reviewed by
drb-audio-review (findings fixed; simultaneous two-channel mixdown filed
as #26) and drb-qa-review.
Not verified on SDR hardware.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /api/health: ordered checklist (password, C2 approval, MQTT, SDR, source,
OP25, decoding, audio, Icecast) with what's wrong and what to do; pure
build_checks() so every state (incl. Toughny's unapproved node) is tested
- /api/listen: pick P25 control channels / one analog freq / a cached C2
system, validated in MHz, stored as a local override; OP25 restarted with
the start failure reason surfaced
- /api/config/system no longer writes a raw blob OP25 can't read
- OP25 resumes after reboot on a locally chosen system, not only C2-assigned
- op25-container: analog (NBFM) config generation fixed (AttributeError +
wrong field names), /op25/log tail, start refuses without a config,
status detects a dead multi_rx ("crashed") -- node-26#12
- index.html rebuilt on UI_REDESIGN tokens; scanner drops DMR, shows System key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install.sh (#14): after enrolling it now BLOCKS until an admin approves,
printing "Please have an admin approve node '<id>' at <app>/settings/nodes
to continue", polling GET /nodes/{id}/credentials with the pickup secret
(no default timeout; Ctrl-C safe; re-run resumes from configs/pickup_secret).
A re-run with a key on disk re-checks it against C2 and replaces a stale
(reissued) key. credentials.json is merged atomically so the dashboard auth
salt/secret survive. edge-node is ALWAYS force-recreated at the end, so a key
written after the container started is loaded. --wait-approval N caps the
wait; 0 = don't wait. git fetch uses --force so a re-cut v1 tag is accepted.
edge-node (#14): with no key at startup, or on a broker auth refusal
(CONNACK 4/5/134/135), mqtt_manager re-reads credentials.json, else fetches
the key over HTTPS (key_pickup.py, pickup secret from /configs), persists it
and rebuilds the paho client in place. Single-flight, 30s..15min backoff.
A key pushed over MQTT while connected is applied for the next reconnect.
OTA (#15): node_update no longer os._exit()s. It drops
configs/update-requested; install.sh installs scripts/install-updater.sh's
drb-update.{service,timer,path}. The path unit fires on the sentinel, the
timer (~15 min) does an HTTPS check-in (POST /nodes/{id}/update-check,
server-26 PR) so offline/locked-out nodes still update. drb-update.sh only
installs the repo/ref/registry in root-owned /etc/drb/updater.conf; C2's
answer is just a timestamp; origin and registry mismatches abort; creds go
to curl via a 0600 header file.
Tests: test_credential_recovery.py, test_host_updater.py (runs the real
updater script with stub git/docker/curl).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes node-26#11. OP25's generated config said "rtl" (= whichever dongle
enumerates first), so on 2-SDR nodes a decoder could take OP25's dongle
and stop recording. Now:
- sdr_pins: {op25|adsb|ais: serial}, absent = automatic. Every OP25
config generation (op25_client.generate_config) rewrites the device to
rtl=<serial>: the pin, else the first dongle's serial, which is what
"rtl" always opened. Left as "rtl" only for unknown/shared serials.
- secondary-sdr: /secondary/devices lists dongles + serials via librtlsdr
(works while claimed). apply(priority, pins, reserved) never touches
OP25's dongle, gives a pinned service only its own dongle, lets a
higher-priority service take a spare from a lower one, and still runs a
pinned lower-priority service when the top pick has no dongle.
- sdr_settings.py replaces secondary_priority.py: one apply path for the
local dashboard, the new set_sdr_config C2 command (set_secondary_priority
kept as an alias) and config pushes. OP25 restarts only when its own
dongle changes. Checkin reports sdr_devices, sdr_pins, op25_sdr_serial.
- Local dashboard: 'SDRs' card with an OP25 SDR dropdown and a per-service
dongle dropdown, duplicate-serial and double-pin warnings.
Verified: edge-node pytest 194 passed; secondary-sdr tests 7 passed;
flake8 clean; page JS passes node --check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the single secondary_sdr_mode with secondary_sdr_priority, e.g.
["adsb", "ais"]. OP25 always keeps its own dongle; the secondary-sdr
container starts decoders top-down until it runs out of free SDRs, so a
3-SDR node runs ADS-B and AIS at once and a 2-SDR node runs the top pick.
- secondary-sdr: one decoder per mode, POST /secondary/apply(priority)
(no-op when the right prefix is already running), orphaned decoders
from a uvicorn reload are reaped on start, /status reports sdr_count
via lsusb (op25's :stable image has none).
- edge-node: one apply path (set_secondary_priority) for the local
dashboard, a new C2 'set_secondary_priority' MQTT command, and config
pushes; it never restarts op25. Legacy mode migrates on load. Checkin
reports priority, what's running, and sdr_count. Uplink forwards
aircraft and vessels whenever either is present.
- Local dashboard: 'Secondary SDRs' card to enable/reorder/save.
Verified: edge-node pytest 190 passed, flake8 clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The single encode at save time was %mp3(bitrate=16), and the comment said why:
it matched what Liquidsoap pushes to Icecast. That was the wrong thing to
match. Icecast is the LISTENING path and 16 kbps is a bandwidth budget for a
live stream; this file is the ACCURACY path -- it is what Whisper transcribes,
and CLAUDE.md is explicit that everything downstream is hostage to it. P25 has
already been through a vocoder, so 16 kbps MP3 stacked a second lossy stage on
the one copy that had to stay faithful.
FLAC instead. Lossless, so the bytes Whisper receives are the bytes PulseAudio
captured. ~1.3 MB/min against 120 KB/min, which keeps a 600 s call (the time
cap) around 13 MB -- inside Whisper's 25 MB request cap and well inside
upload_max_bytes. Icecast's own 16 kbps stream is untouched; nothing about
live listening changes.
Capture, buffering, silence detection and the byte-offset trim are all
unchanged: they operate on raw PCM and never saw the encode. The sample rate
stays pinned to pcm.SAMPLE_RATE so the encode remains a straight pass -- the
trim arithmetic depends on that, and Whisper resamples to 16 kHz itself.
encode_mp3 is now encode_recording, the upload sends audio/flac, and the test
that pinned the old contract now pins losslessness instead, including an
assertion that no bitrate constant comes back.
This is the before/after boundary for STT quality. Last night's window is the
16 kbps baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two unauthenticated surfaces closed on the edge node.
Dashboard and API: the local dashboard and every /api/* route were open to
anything on the node's LAN. Adds a login page plus session-cookie auth for
the browser, and cookie-or-Basic for the API so scripted callers stay
possible. Passwords are hashed with stdlib scrypt (no new dependency, this
runs on a Pi) and compared in constant time; the salt and session-signing
secret persist in credentials.json. Startup warns while the default password
is still in place. No non-browser callers of the node API exist today
(C2 talks to nodes over MQTT and nodes call C2 outbound), so nothing breaks.
Adds python-multipart, which FastAPI's Form() needs for the login POST and
which was missing from requirements entirely.
MQTT: nodes authenticated with a shared drb-node password, and the broker
ACL keyed off %c — the client-supplied client id — so any holder of that one
password could claim another node's topic namespace. Nodes now connect as
username=<node_id>, password=<their C2-issued api_key>, which mosquitto's
dynamic-security plugin checks, with the ACL keyed off the authenticated %u.
TLS is gated on MQTT_TLS and uses default CA verification.
The old key_request MQTT path stays in place behind TODO(mqtt-cutover)
markers as the fallback until the cutover is proven; a node with no api_key
on disk logs a clear repeated refusal rather than spinning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The control channel was wrong in both directions. Grants fire 0.84-1.62s
before anyone speaks, and srcaddr can drop to 0 while someone is still
talking - one recording came back "-1.61s lead, -0.00s tail", the trim
finding nothing to remove because the window had closed on live speech.
Confirmed by ear: the cut lands at a word boundary on an unfinished word.
Audio is ground truth for WHEN. The console remains the only source of
WHO, so it still supplies talkgroup, alias and rid.
START voice onset in the captured audio, with a 0.25s pre-roll that
now covers only chunk quantisation and threshold ramp-up rather
than a variable control-channel offset.
STOP call_silence_timeout seconds of silence heard in the audio.
LABEL resolved AT CLOSE from a bounded rolling history of console
observations overlapping the window, +4s/-2s, because there is
no guaranteed ordering between a grant and its audio.
SPLIT a console talkgroup change still forces a cut, since two calls
with no silence between them would otherwise merge into one.
Capture now emits raw PCM instead of MP3. Silence detection becomes
integer arithmetic per chunk with no decode, trimming becomes a byte
offset slice rather than a second ffmpeg pass, and MP3 encoding happens
exactly once at save - uploads are no longer double-encoded.
Audio with no talkgroup anywhere in its window is discarded rather than
uploaded: an untagged call silently poisons incident correlation, which
is worse than losing the audio. Logged at ERROR and counted on
/api/status.
When capture produces no audio at all the old console state machine
still runs, so a node with a broken audio path keeps reporting radio
activity. That is now the only consumer of call_idle_timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three unrelated failures from `make test` on the node:
test_metadata_watcher: two tests asserted the old zero-pad tgid_change
contract. Both were off by exactly call_tail_pad_seconds, i.e. the code
was doing what f1de157 intended and the tests encoded the behaviour we
deliberately changed. Updated to expect the pad.
test_pulse: four async tests errored with "async def functions are not
natively supported". Root cause is not the tests - pytest.ini sets
asyncio_mode = auto but the Dockerfile only copies app/ and tests/, so
the container had no pytest config at all and fell back to strict mode.
That is also why these passed in a local venv and failed on the node.
Copy pytest.ini into the image, and add explicit @pytest.mark.asyncio so
the tests hold up regardless of how pytest is configured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recording window is anchored to OP25 control-channel timestamps, but
the buffered audio lags those by roughly 1.5s. Trim logs across seven
calls measured the offset at 0.84-1.62s, consistently present.
At a 1.0s pad a short call closed its window before the voice arrived:
a 0.97s control-channel call closed at T+1.97 while voice started around
T+1.5, capturing ~0.4s of speech and cutting mid-word. Confirmed by a
0.57s file whose final 0.10s measured -12.2dB against its own -18.2dB
average - clipped speech, not a tail - and by two short calls that
logged no trim at all because no trailing silence remained.
Being generous is free here: trim_silence already strips trailing
silence back to the guard margin before upload, so long calls are
unaffected while short ones gain the window they need. Over-capture
costs nothing; under-capture loses words permanently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pulse_socket named volume survives container recreation, so after a
compose recreate the previous container's /run/pulse/pid and native
socket were still present. PulseAudio read the stale pid file, decided a
daemon was already running, and refused to start:
E: [pulseaudio] pid.c: Daemon already running.
The entrypoint still reported "PulseAudio socket ready" because it only
checked that the socket file existed - and a stale one did. Capture then
failed in a restart loop against a dead daemon.
Readiness in both the op25 entrypoint and drb-edge-node now means a
pactl probe actually succeeds. Stale pid/socket are removed only when
that probe fails, so a live daemon's socket is never deleted.
pulseaudio-utils was missing from the edge-node image (only libpulse0
was installed), so no pactl binary existed there at all - added.
Capture exits are now classified: a missing source logs at ERROR and
names the configured PULSE_SOURCE, rather than looking identical to
"daemon not up yet". Retrying forever against a wrong source name is how
the April PulseAudio failure stayed hidden.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured six recordings off a live P25 node and found two independent
defects causing lost audio at the end of calls:
- stop_recording() sliced the ring buffer immediately, so if the MP3
muxer had not yet delivered the tail the file was silently short.
Now waits (bounded, 2s) until buffered audio covers the end epoch.
- TGID-change closes used the new grant's epoch as the end with no pad
at all, guaranteeing truncation on every split. Tail pad is now a
setting, default raised 0.5s -> 1.0s.
The ring buffer also capped maximum call length: a call longer than the
buffer had its front silently clamped. The ring now serves the pre-roll
only, with a per-call accumulator for the rest, bounded at 4.8MB.
Clamping is loudly warned rather than silent.
Uploads averaged 63% silence, which inflates STT cost and is a known
Whisper hallucination trigger. Leading/trailing silence is now trimmed
conservatively (-40dB, 0.25s guard, internal pauses untouched).
started_at/ended_at still describe the call; new audio_* fields carry
the trimmed audio bounds so playback can map back to wall clock.
All-silence recordings are skipped and logged instead of uploaded.
Also: log measured control-channel idle on idle-timeout closes so
CALL_IDLE_TIMEOUT can be tuned from data, and quiet the httpx logger
which emitted ~170k lines/day of poll noise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
