Deploy of PR #126 (2026-09-12) failed instantly: git pull origin main on the VM died with fatal: unable to write loose object file: No space left on device / fatal: unpack-objects failed. The rollback path never ran because the failure happened before PREV_TAG was even captured — prod was left on the previous good build, not broken, but no further deploy (including the pending chatter-classifier PR) could land until this cleared.
Root cause: docker image prune -f (dangling-only) only ran AFTER a successful docker compose pull, and only ran -af (all-unused) as a retry-on-pull-failure. A run that fails earlier than that — like this one, at git pull — never reaches either prune. Every deploy pushes 3 freshly SHA-tagged images that nothing else ever removes; docker system df on the VM showed 100 local images, 4 active, 23.76GB reclaimable (96 unreferenced tagged images) — the disk was 29G/29G (100%) full.
Manual fix applied same day: docker image prune -af on the VM, reclaimed 23.76GB, df -h / back to 23% used. Re-ran the stuck deploy via the Gitea API (POST .../actions/runs/{id}/rerun) rather than a new commit.
Durable fix (this PR): move an unconditional docker image prune -af to the very start of the deploy's SSH block, before git pull — so a full disk can never block the pull in the first place — and change the post-up -d prune from -f to -af so stale SHA-tagged images from past deploys don't re-accumulate between now and the next incident. Both prunes are safe by construction: docker image prune -a never removes an image a running container references, so nothing currently serving traffic can be pruned mid-deploy.
Deploy of PR #126 (2026-09-12) failed instantly: `git pull origin main` on the VM died with `fatal: unable to write loose object file: No space left on device` / `fatal: unpack-objects failed`. The rollback path never ran because the failure happened before `PREV_TAG` was even captured — prod was left on the previous good build, not broken, but no further deploy (including the pending chatter-classifier PR) could land until this cleared.
Root cause: `docker image prune -f` (dangling-only) only ran AFTER a successful `docker compose pull`, and only ran `-af` (all-unused) as a retry-on-pull-failure. A run that fails earlier than that — like this one, at `git pull` — never reaches either prune. Every deploy pushes 3 freshly SHA-tagged images that nothing else ever removes; `docker system df` on the VM showed 100 local images, 4 active, 23.76GB reclaimable (96 unreferenced tagged images) — the disk was 29G/29G (100%) full.
Manual fix applied same day: `docker image prune -af` on the VM, reclaimed 23.76GB, `df -h /` back to 23% used. Re-ran the stuck deploy via the Gitea API (`POST .../actions/runs/{id}/rerun`) rather than a new commit.
Durable fix (this PR): move an unconditional `docker image prune -af` to the very start of the deploy's SSH block, before `git pull` — so a full disk can never block the pull in the first place — and change the post-`up -d` prune from `-f` to `-af` so stale SHA-tagged images from past deploys don't re-accumulate between now and the next incident. Both prunes are safe by construction: `docker image prune -a` never removes an image a running container references, so nothing currently serving traffic can be pruned mid-deploy.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Deploy of PR #126 (2026-09-12) failed instantly:
git pull origin mainon the VM died withfatal: unable to write loose object file: No space left on device/fatal: unpack-objects failed. The rollback path never ran because the failure happened beforePREV_TAGwas even captured — prod was left on the previous good build, not broken, but no further deploy (including the pending chatter-classifier PR) could land until this cleared.Root cause:
docker image prune -f(dangling-only) only ran AFTER a successfuldocker compose pull, and only ran-af(all-unused) as a retry-on-pull-failure. A run that fails earlier than that — like this one, atgit pull— never reaches either prune. Every deploy pushes 3 freshly SHA-tagged images that nothing else ever removes;docker system dfon the VM showed 100 local images, 4 active, 23.76GB reclaimable (96 unreferenced tagged images) — the disk was 29G/29G (100%) full.Manual fix applied same day:
docker image prune -afon the VM, reclaimed 23.76GB,df -h /back to 23% used. Re-ran the stuck deploy via the Gitea API (POST .../actions/runs/{id}/rerun) rather than a new commit.Durable fix (this PR): move an unconditional
docker image prune -afto the very start of the deploy's SSH block, beforegit pull— so a full disk can never block the pull in the first place — and change the post-up -dprune from-fto-afso stale SHA-tagged images from past deploys don't re-accumulate between now and the next incident. Both prunes are safe by construction:docker image prune -anever removes an image a running container references, so nothing currently serving traffic can be pruned mid-deploy.