Deploy the commit's own images instead of :latest
Build & Deploy / Build & push images (push) Successful in 4m4s
Build & Deploy / Deploy to VM (push) Successful in 1m26s
Build & Deploy / Report a failed deploy (push) Skipped

The build-stamp health check added in 8fbfe7d worked on its first run, and
what it caught was not a stale container -- it was a race. Runs 544 and 545
overlapped; both deployed :latest, 545's images won, and 544's health check
correctly reported that the build serving traffic was not the one it had just
deployed.

That is a real hazard, not a false positive: with :latest, two pushes landing
close together means whichever finishes last silently wins for BOTH, and
neither run's log tells you which code is actually live. Pushes land close
together constantly here.

docker-compose.yml already resolved images as ${TAG:-latest}, so the fix is to
export TAG=<commit sha> for the deploy. Each run now pulls and starts exactly
the images it built, rollback becomes "deploy a different tag", and the health
check's assertion becomes meaningful rather than order-dependent. A manual
`docker compose up -d` on the VM with no TAG set still falls back to :latest,
which is the intended escape hatch.

Also replaces the health check's single `sleep 20` with a poll of up to 100s
that stops as soon as the expected SHA appears. A fixed sleep is either too
short -- flaky red runs -- or wastes time on every deploy, and a check that
cries wolf gets ignored, which is exactly the failure this job exists to stop.

Refs logan/server-26#21
This commit is contained in:
Logan Cusano
2026-08-23 01:38:09 -04:00
parent 82c88379d4
commit 861ea41cec
+22 -3
View File
@@ -93,6 +93,15 @@ jobs:
# Update compose files + mosquitto config # Update compose files + mosquitto config
git pull origin main git pull origin main
# Deploy THIS commit's images, not :latest. Overlapping runs are
# normal here, and with :latest whichever finishes last wins for
# both -- run 544 asserted its own SHA and found run 545's build
# already serving. compose already supports ${TAG:-latest}, so
# pinning makes each deploy deterministic and a rollback just a
# different tag. A later manual `up -d` on the VM without TAG set
# still falls back to :latest, which is the intended escape hatch.
export TAG=${{ gitea.sha }}
# Pull pre-built images and restart (no build on the VM). # Pull pre-built images and restart (no build on the VM).
# #
# The retry is not defensive padding: this exact step failed fifteen # The retry is not defensive padding: this exact step failed fifteen
@@ -114,9 +123,19 @@ jobs:
- name: Health check - name: Health check
run: | run: |
sleep 20 # Poll rather than sleep-once: the container has to finish starting,
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || { # and a fixed sleep is either too short (flaky red) or wastes time on
echo "Health check failed: /health did not respond"; exit 1; } # every deploy. A health check that cries wolf gets ignored, which is
# the failure mode this whole job exists to prevent.
BODY=""
for _ in $(seq 1 20); do
sleep 5
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || continue
case "$BODY" in *"${{ gitea.sha }}"*) break ;; esac
done
if [ -z "$BODY" ]; then
echo "Health check failed: /health never responded"; exit 1
fi
echo "$BODY" echo "$BODY"
# Liveness alone is not enough. A deploy can report success while the # Liveness alone is not enough. A deploy can report success while the