deploy.yml has no concurrency guard — two pushes to main deploy to the VM at once, last-finisher wins #167

New Issue

2026-09-24T01:10:46-04:00

logan commented

2026-09-24 01:10:46 -04:00

Observed 2026-09-24: runs 616 (6c0e7a4) and 617 (02b5b7b, newer) were both in "Deploy to VM" simultaneously after two merges a few minutes apart (#165, #166).

No concurrency: block in .gitea/workflows/deploy.yml, so:

two docker compose pull && up -d race on the VM
whichever finishes last sets the running SHA — an OLDER run can land after a newer one and silently roll prod back
the health-check/rollback step of one run can judge the other run's containers

Fix shape: concurrency: { group: deploy-prod, cancel-in-progress: false } on the deploy job (queue, don't cancel mid-compose), plus have the deploy step skip if github.sha is no longer the tip of main. Verify Gitea Actions (act_runner) honours concurrency — support was partial in older versions.

Workaround until then: space merges to main until the previous Deploy to VM finishes, and check /health SHA against main HEAD after (deploy-check skill).

Observed 2026-09-24: runs 616 (6c0e7a4) and 617 (02b5b7b, newer) were both in "Deploy to VM" simultaneously after two merges a few minutes apart (#165, #166). No `concurrency:` block in `.gitea/workflows/deploy.yml`, so: - two `docker compose pull && up -d` race on the VM - whichever finishes last sets the running SHA — an OLDER run can land after a newer one and silently roll prod back - the health-check/rollback step of one run can judge the other run's containers Fix shape: `concurrency: { group: deploy-prod, cancel-in-progress: false }` on the deploy job (queue, don't cancel mid-compose), plus have the deploy step skip if `github.sha` is no longer the tip of main. Verify Gitea Actions (act_runner) honours `concurrency` — support was partial in older versions. Workaround until then: space merges to main until the previous Deploy to VM finishes, and check /health SHA against main HEAD after (deploy-check skill).

logan commented

2026-09-27 14:51:49 -04:00

Hit again 2026-09-27: runs 652 (000a5f1) and 656 (49ef914) overlapped on the VM. 656's pull failed on a layer extract, its prune-and-retry got "Error response from daemon: a prune operation is already running" (652's prune), and Main Deploy exited 1. There was no prev tag, so there was no rollback. Prod /health returned 502 while 652 restarted containers, then recovered on 000a5f1. 656 was re-run by hand. So the overlap breaks the retry path too, not just last-finisher-wins.

Hit again 2026-09-27: runs 652 (000a5f1) and 656 (49ef914) overlapped on the VM. 656's pull failed on a layer extract, its prune-and-retry got "Error response from daemon: a prune operation is already running" (652's prune), and Main Deploy exited 1. There was no prev tag, so there was no rollback. Prod /health returned 502 while 652 restarted containers, then recovered on 000a5f1. 656 was re-run by hand. So the overlap breaks the retry path too, not just last-finisher-wins.

logan referenced a pull request that will close this issue

2026-10-04 18:02:15 -04:00

deploy.yml: serialise prod deploys and skip superseded runs (#167) #210

logan commented

2026-10-04 18:02:21 -04:00

PR #210 (#210): job-level concurrency group on deploy (Gitea 1.26+ supports it, this instance is 1.27.3, so no flock), plus the deploy step skips a run whose SHA is no longer main tip, and health check is skipped for it. Firestore rules job gets its own group + the same tip check. First real proof: two merges close together, the second should wait.

PR #210 (https://git.vpn.cusano.net/logan/server-26/pulls/210): job-level concurrency group on deploy (Gitea 1.26+ supports it, this instance is 1.27.3, so no flock), plus the deploy step skips a run whose SHA is no longer main tip, and health check is skipped for it. Firestore rules job gets its own group + the same tip check. First real proof: two merges close together, the second should wait.

logan closed this issue

2026-10-04 18:29:32 -04:00

logan referenced this issue from a commit

2026-10-04 18:29:33 -04:00

Merge pull request 'deploy.yml: serialise prod deploys and skip superseded runs (#167)' (#210) from fix/167-deploy-concurrency into main

logan referenced this issue

2026-10-04 18:40:40 -04:00

deploy.yml: a superseded run is a skip, not a failure (#167) #217

logan referenced this issue from a commit

2026-10-04 18:41:34 -04:00

Merge pull request 'deploy.yml: a superseded run is a skip, not a failure (#167)' (#217) from fix/167-superseded-exit into main

logan added the area/infra-deploy