logan/node-26
1
0
Fork 0
Code Issues 14 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Updater never prunes Docker build cache: node-002 disk hit 100% and silently stopped recording for 17 days #41

New Issue
Closed
opened 2026-10-26 00:59:47 -04:00 by logan · 3 comments
logan commented 2026-10-26 00:59:47 -04:00
Owner
Copy Link
Copy Source

Found 2026-10-26 on node-002 (radio-box, 172.16.100.58): root disk 100% full (98G/98G). The node stopped recording around 2026-10-08 16:15 UTC. The edge-node log has no lines after that apart from one MQTT disconnect, and the portal still showed it Online. A call doc created after a Restart OP25 on 10-25 had no audio, because the recording could not be written.

docker system df on the node:

  • Build Cache 39.53 GB (37.71 GB reclaimable, 158 entries, 0 active)
  • Images 52.84 GB (16.26 GB reclaimable)
  • recordings/ 616 KB, node.log 40 MB. The disk is all Docker.

Cause: scripts/drb-update.sh runs docker compose build on every update when BUILD=1, and nothing ever prunes the BuildKit cache. Old images are removed only on the pin=1 path; docker image prune -f removes only dangling images.

Fix shape:

  1. After a successful up -d: docker builder prune -af --filter until=72h (or a size cap, --keep-storage).
  2. Also prune unused images when pin != 1, or always use docker image prune -af --filter until=..., which never touches running containers.
  3. Have the checkin report free disk %. The portal should go amber below 15% and red below 5% (ties into #40: "online" hid this for 17 days).

Server hit the same failure and fixed it in fix/129-deploy-disk-prune; reuse that approach.

**Found 2026-10-26 on node-002 (radio-box, 172.16.100.58): root disk 100% full (98G/98G).** The node stopped recording around 2026-10-08 16:15 UTC. The edge-node log has no lines after that apart from one MQTT disconnect, and the portal still showed it Online. A call doc created after a Restart OP25 on 10-25 had no audio, because the recording could not be written. `docker system df` on the node: - Build Cache 39.53 GB (37.71 GB reclaimable, 158 entries, 0 active) - Images 52.84 GB (16.26 GB reclaimable) - recordings/ 616 KB, node.log 40 MB. The disk is all Docker. Cause: `scripts/drb-update.sh` runs `docker compose build` on every update when `BUILD=1`, and nothing ever prunes the BuildKit cache. Old images are removed only on the `pin=1` path; `docker image prune -f` removes only dangling images. Fix shape: 1. After a successful `up -d`: `docker builder prune -af --filter until=72h` (or a size cap, `--keep-storage`). 2. Also prune unused images when `pin != 1`, or always use `docker image prune -af --filter until=...`, which never touches running containers. 3. Have the checkin report free disk %. The portal should go amber below 15% and red below 5% (ties into #40: "online" hid this for 17 days). Server hit the same failure and fixed it in fix/129-deploy-disk-prune; reuse that approach.
logan added the
type
bug
area/recording-audioarea/node-ota
priority
p1
 labels 2026-10-26 00:59:47 -04:00
logan commented 2026-10-26 01:15:03 -04:00
Author
Owner
Copy Link
Copy Source

node-002 remediated 2026-10-26 ~05:00 UTC: docker builder prune -af + image prune -af freed the disk from 100% to 55% (43 GB free); restarted edge-node; owner confirms it is recording again. Updater fix in PR #42 (not yet merged). Disk-free reporting to the portal still open.

node-002 remediated 2026-10-26 ~05:00 UTC: docker builder prune -af + image prune -af freed the disk from 100% to 55% (43 GB free); restarted edge-node; owner confirms it is recording again. Updater fix in PR #42 (not yet merged). Disk-free reporting to the portal still open.
logan commented 2026-10-26 02:18:27 -04:00
Author
Owner
Copy Link
Copy Source

Correction to the stated cause: node-002 runs the updater with BUILD=0 (REF=v1), so the 39.5 GB build cache did not come from the updater building on each update. It came from earlier local builds (setup.sh / make up / manual compose builds), which nothing ever pruned. PR #42 still fixes this: the prunes run on every update regardless of BUILD. v1 re-cut onto 04ba545 (was 5770eca) on 2026-10-26 to ship it, together with #36.

Correction to the stated cause: node-002 runs the updater with BUILD=0 (REF=v1), so the 39.5 GB build cache did not come from the updater building on each update. It came from earlier local builds (setup.sh / make up / manual compose builds), which nothing ever pruned. PR #42 still fixes this: the prunes run on every update regardless of BUILD. v1 re-cut onto 04ba545 (was 5770eca) on 2026-10-26 to ship it, together with #36.
logan commented 2026-10-26 03:01:48 -04:00
Author
Owner
Copy Link
Copy Source

Shipped. Updater prune: a26a77d (PR #42). Disk reporting: node ab10626 (PR #43) + server-26 f4d78ab (PR 295). v1 re-cut to ab10626; all 3 nodes updated 2026-10-26 ~03:00 and report disk in the portal: node-002 77 GB free of 98 (79%), RPI-Node-01 14/28 GB (52%), Toughny 14/29 GB (48%). Cards warn amber <15%, red <5% or 2 GB.

Shipped. Updater prune: a26a77d (PR #42). Disk reporting: node ab10626 (PR #43) + server-26 f4d78ab (PR 295). v1 re-cut to ab10626; all 3 nodes updated 2026-10-26 ~03:00 and report disk in the portal: node-002 77 GB free of 98 (79%), RPI-Node-01 14/28 GB (52%), Toughny 14/29 GB (48%). Cards warn amber <15%, red <5% or 2 GB.
logan closed this issue 2026-10-26 03:01:48 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/node-ota area/recording-audio
priority
p1
type
bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/node-26#41