deploy.yml: serialise prod deploys and skip superseded runs (#167) #210

Merged
logan merged 3 commits from fix/167-deploy-concurrency into main 2026-10-04 18:29:32 -04:00
Owner

Fixes #167.

What changed (.gitea/workflows/deploy.yml only)

  • deploy job: concurrency: { group: drb-prod-deploy, cancel-in-progress: false }. Builds still overlap; only the VM part (deploy, health check, rollback) is serialised. Queue, never kill mid-compose.
  • Syntax support checked: Gitea added concurrency in 1.26.0 (go-gitea/gitea#32751, which also removed the old implicit auto-cancel); this instance reports 1.27.3. No VM-side flock needed.
  • Deploy step now git fetches on the VM and exits 0 before the prune or any compose call if gitea.sha is no longer origin/main's tip (prints SUPERSEDED=<tip>, sets superseded=true). Health check is skipped for a superseded run, so it can't fail on the newer SHA and trigger a rollback. This covers an older run going after a newer one (later build finish, or a manual re-run). Deliberately putting an older build back is now a manual TAG=<sha> on the VM, not a re-run.
  • deploy-firestore-rules: its own group (drb-prod-firestore-rules) plus the same tip check via git ls-remote. If the remote can't be read it still deploys, as before.

Verification

  • YAML parses; job concurrency and step if: read back as intended. Output-parsing logic simulated under bash -eo pipefail for both the superseded and normal paths.
  • No actionlint/shellcheck on this box. The real test is the next two merges landing close together: the second should show "waiting" until the first's Deploy to VM finishes.
  • Trade-off: if the newest run's build fails, the older superseded commit is not deployed either. Its changes are in the newer commit, so they ship when that's fixed.

🤖 Generated with Claude Code

Fixes #167. **What changed** (`.gitea/workflows/deploy.yml` only) - `deploy` job: `concurrency: { group: drb-prod-deploy, cancel-in-progress: false }`. Builds still overlap; only the VM part (deploy, health check, rollback) is serialised. Queue, never kill mid-compose. - Syntax support checked: Gitea added `concurrency` in **1.26.0** (go-gitea/gitea#32751, which also removed the old implicit auto-cancel); this instance reports **1.27.3**. No VM-side flock needed. - Deploy step now `git fetch`es on the VM and exits 0 **before the prune or any compose call** if `gitea.sha` is no longer `origin/main`'s tip (prints `SUPERSEDED=<tip>`, sets `superseded=true`). Health check is skipped for a superseded run, so it can't fail on the newer SHA and trigger a rollback. This covers an older run going after a newer one (later build finish, or a manual re-run). Deliberately putting an older build back is now a manual `TAG=<sha>` on the VM, not a re-run. - `deploy-firestore-rules`: its own group (`drb-prod-firestore-rules`) plus the same tip check via `git ls-remote`. If the remote can't be read it still deploys, as before. **Verification** - YAML parses; job `concurrency` and step `if:` read back as intended. Output-parsing logic simulated under `bash -eo pipefail` for both the superseded and normal paths. - No actionlint/shellcheck on this box. The real test is the next two merges landing close together: the second should show "waiting" until the first's Deploy to VM finishes. - Trade-off: if the newest run's build fails, the older superseded commit is not deployed either. Its changes are in the newer commit, so they ship when that's fixed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
logan added 1 commit 2026-10-04 18:02:16 -04:00
Two merges minutes apart ran two "Deploy to VM" jobs on the VM at once
(616/617, 652/656): racing compose pull/up, one run's prune breaking the
other's retry, health checks judging the wrong containers.

- deploy job: concurrency group drb-prod-deploy, cancel-in-progress false
  (Gitea >= 1.26 honours it, go-gitea/gitea#32751; server is 1.27.3).
- Deploy step exits clean, before touching the VM, when its SHA is no
  longer origin/main's tip; Health check is skipped for a superseded run.
  Stops an older run landing after a newer one and rolling prod back.
- deploy-firestore-rules: its own group plus the same tip check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
logan added 1 commit 2026-10-04 18:07:21 -04:00
The tip check's git fetch ran before the disk-relief prune that #129 says
must be the first thing a deploy does; on a full disk the fetch could fail
and skip the prune. Prune only removes unreferenced images, so running it
on a run that then turns out superseded is harmless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
logan added 1 commit 2026-10-04 18:29:24 -04:00
logan merged commit 8869d51557 into main 2026-10-04 18:29:32 -04:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#210