From 861ea41cece1fe47fd937ea3fd0183a8192912b9 Mon Sep 17 00:00:00 2001 From: Logan Cusano Date: Sun, 23 Aug 2026 01:38:09 -0400 Subject: [PATCH] Deploy the commit's own images instead of :latest The build-stamp health check added in 8fbfe7d worked on its first run, and what it caught was not a stale container -- it was a race. Runs 544 and 545 overlapped; both deployed :latest, 545's images won, and 544's health check correctly reported that the build serving traffic was not the one it had just deployed. That is a real hazard, not a false positive: with :latest, two pushes landing close together means whichever finishes last silently wins for BOTH, and neither run's log tells you which code is actually live. Pushes land close together constantly here. docker-compose.yml already resolved images as ${TAG:-latest}, so the fix is to export TAG= for the deploy. Each run now pulls and starts exactly the images it built, rollback becomes "deploy a different tag", and the health check's assertion becomes meaningful rather than order-dependent. A manual `docker compose up -d` on the VM with no TAG set still falls back to :latest, which is the intended escape hatch. Also replaces the health check's single `sleep 20` with a poll of up to 100s that stops as soon as the expected SHA appears. A fixed sleep is either too short -- flaky red runs -- or wastes time on every deploy, and a check that cries wolf gets ignored, which is exactly the failure this job exists to stop. Refs logan/server-26#21 --- .gitea/workflows/deploy.yml | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index aca5c40..01f8de9 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -93,6 +93,15 @@ jobs: # Update compose files + mosquitto config git pull origin main + # Deploy THIS commit's images, not :latest. Overlapping runs are + # normal here, and with :latest whichever finishes last wins for + # both -- run 544 asserted its own SHA and found run 545's build + # already serving. compose already supports ${TAG:-latest}, so + # pinning makes each deploy deterministic and a rollback just a + # different tag. A later manual `up -d` on the VM without TAG set + # still falls back to :latest, which is the intended escape hatch. + export TAG=${{ gitea.sha }} + # Pull pre-built images and restart (no build on the VM). # # The retry is not defensive padding: this exact step failed fifteen @@ -114,9 +123,19 @@ jobs: - name: Health check run: | - sleep 20 - BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || { - echo "Health check failed: /health did not respond"; exit 1; } + # Poll rather than sleep-once: the container has to finish starting, + # and a fixed sleep is either too short (flaky red) or wastes time on + # every deploy. A health check that cries wolf gets ignored, which is + # the failure mode this whole job exists to prevent. + BODY="" + for _ in $(seq 1 20); do + sleep 5 + BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || continue + case "$BODY" in *"${{ gitea.sha }}"*) break ;; esac + done + if [ -z "$BODY" ]; then + echo "Health check failed: /health never responded"; exit 1 + fi echo "$BODY" # Liveness alone is not enough. A deploy can report success while the