deploy.yml has no concurrency guard — two pushes to main deploy to the VM at once, last-finisher wins #167

Open
opened 2026-09-24 01:10:46 -04:00 by logan · 0 comments
Owner

Observed 2026-09-24: runs 616 (6c0e7a4) and 617 (02b5b7b, newer) were both in "Deploy to VM" simultaneously after two merges a few minutes apart (#165, #166).

No concurrency: block in .gitea/workflows/deploy.yml, so:

  • two docker compose pull && up -d race on the VM
  • whichever finishes last sets the running SHA — an OLDER run can land after a newer one and silently roll prod back
  • the health-check/rollback step of one run can judge the other run's containers

Fix shape: concurrency: { group: deploy-prod, cancel-in-progress: false } on the deploy job (queue, don't cancel mid-compose), plus have the deploy step skip if github.sha is no longer the tip of main. Verify Gitea Actions (act_runner) honours concurrency — support was partial in older versions.

Workaround until then: space merges to main until the previous Deploy to VM finishes, and check /health SHA against main HEAD after (deploy-check skill).

Observed 2026-09-24: runs 616 (6c0e7a4) and 617 (02b5b7b, newer) were both in "Deploy to VM" simultaneously after two merges a few minutes apart (#165, #166). No `concurrency:` block in `.gitea/workflows/deploy.yml`, so: - two `docker compose pull && up -d` race on the VM - whichever finishes last sets the running SHA — an OLDER run can land after a newer one and silently roll prod back - the health-check/rollback step of one run can judge the other run's containers Fix shape: `concurrency: { group: deploy-prod, cancel-in-progress: false }` on the deploy job (queue, don't cancel mid-compose), plus have the deploy step skip if `github.sha` is no longer the tip of main. Verify Gitea Actions (act_runner) honours `concurrency` — support was partial in older versions. Workaround until then: space merges to main until the previous Deploy to VM finishes, and check /health SHA against main HEAD after (deploy-check skill).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#167