From e97dab22ce8f7e1c44a0e31ddfa8c06ee258a2eb Mon Sep 17 00:00:00 2001 From: Logan Cusano Date: Sun, 13 Sep 2026 12:27:31 -0400 Subject: [PATCH] ci: prune docker images before AND after deploy, not only on pull failure (#129) The 2026-09-12 deploy of #126 failed instantly -- git pull on the VM hit 'No space left on device' before the deploy script could even capture a rollback target. Every deploy leaves 3 freshly SHA-tagged images that only got cleaned up by a prune gated on a failed compose pull; a failure earlier than that (like this one) never reached it. 96 of 100 local images were unreferenced, 23.76GB reclaimable, disk at 100%. Move an unconditional docker image prune -af to the top of the deploy, before git pull, and upgrade the post-up -d prune from -f (dangling only) to -af (all unused) so stale tagged images stop re-accumulating. Both are safe: prune -a never touches an image a running container references. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01Tbknwttzou4s46PAykmtix --- .gitea/workflows/deploy.yml | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index 6ddea9d..b4b6d19 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -97,6 +97,21 @@ jobs: set -e cd /opt/drb + # server-26#129: every deploy pushes 3 freshly SHA-tagged images and + # nothing ever removed the old ones except a prune that only ran + # AFTER a successful `compose pull` -- so a run that never got that + # far (this one) left the leak unaddressed forever. That silently + # filled the disk to 100% over ~week of deploys (2026-09-12: 29G/29G + # used, 96 of 100 local images unreferenced, 23.76GB reclaimable) and + # took `git pull` itself down with "No space left on device" before + # the deploy could even determine a rollback target. Prune BEFORE + # doing anything else, not after: `docker image prune -af` only + # removes images with no container referencing them, so it can never + # touch what's currently running -- there is nothing here for a + # mid-flight deploy to lose. Warn-not-fail: a prune failure must not + # block a deploy that doesn't actually need the space this time. + docker image prune -af || echo "WARNING: pre-deploy image prune failed (server-26#129) -- disk pressure may persist" + # Update compose files + mosquitto config git pull origin main @@ -159,7 +174,12 @@ jobs: $COMPOSE pull fi $COMPOSE up -d --remove-orphans - docker image prune -f + # server-26#129: -f alone only removes dangling (untagged) images -- + # the SHA-tagged image from every PAST deploy is not dangling, just + # unreferenced once `up -d` swaps the running container to the new + # tag, so it survived this indefinitely. -a catches those too; see + # the pre-pull prune above for why this can't touch anything live. + docker image prune -af ENDSSH ) echo "$OUTPUT" -- 2.54.0