deploy.yml: /opt/drb/.last_good_tag reads empty (not missing), so PREV_TAG is blank and every deploy's Main Deploy step fails — rollback safety net has been dead since at least run #602 #156

New Issue

2026-09-20T14:38:54-04:00

logan commented

2026-09-20 14:38:54 -04:00

Found while deploy-checking commit fb0bb15 (server-26#155's incident-clearance fix). Prod IS correctly running the new code (/health confirms git_sha: fb0bb15..., healthy) — this is a CI/rollback-tooling bug, not a prod outage.

Evidence (job 820, run 606, full log pulled): line ~122 of the Main Deploy step:

PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null || echo latest)
echo "PREV_TAG=$PREV_TAG"

Logged output: PREV_TAG= — completely empty, not even the latest fallback. git pull, the image pulls, and compose up -d all ran and succeeded afterward (confirmed: new images pulled, containers came up, /health now serves the new SHA). But because PREV_TAG came back empty, the step at the end (if [ -z "$PREV_TAG" ]; then ... exit 1; fi) fails the whole job with "Could not determine the previous tag from deploy output - rollback target unknown," which in turn triggers "Rollback on failed health check" — which itself immediately no-ops ("No previous tag was captured ... cannot roll back automatically") since it also has no PREV_TAG to roll back to.

Root cause: cat /opt/drb/.last_good_tag 2>/dev/null || echo latest only falls back to latest when cat fails (file missing, permission denied, etc). If the file EXISTS but is EMPTY (zero bytes), cat succeeds with exit 0 and empty stdout — the || never fires, and PREV_TAG is silently set to the empty string. The health-check step (deploy.yml ~line 228-234) is the only writer of this file, via echo '${{ gitea.sha }}' > /opt/drb/.last_good_tag — if that SSH write was ever interrupted mid-flight (network hiccup, runner timeout) after truncating but before writing the SHA, the file is left permanently empty, and every subsequent deploy reads the same empty value forever, since nothing ever rewrites it back to non-empty (the read happens before the write in each run).

Impact, ongoing: checked runs 602 through 606 (5 most recent, 2026-09-13 through 2026-09-20) — every single one shows Deploy to VM: failure at the job level. The underlying docker compose pull && up -d appears to succeed regardless (spot-confirmed for 605 and 606 via direct /health SHA match), so prod itself is not stuck — but:

The rollback safety net (server-26#65's whole reason for existing) has not been able to function for at least a week — if a deploy DID actually break prod during this window, there was no automatic rollback available.
CI status is a false negative on every push — deploy-check skill users have to manually verify /health every time rather than trusting the Actions conclusion, which defeats part of the point of having it.
.last_good_tag needs to be manually repaired on the VM (echo '<known-good-sha>' > /opt/drb/.last_good_tag, e.g. the current live SHA) before the fallback will read latest again on any FUTURE empty-file case — that alone doesn't fix the underlying bug, since a future interrupted write reproduces the same empty-file state.

Suggested fix: treat an empty (not just missing) file as "no previous tag" — e.g. PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null); [ -n "$PREV_TAG" ] || PREV_TAG=latest. Separately, consider writing the new SHA to a temp file and mving it into place atomically, so an interrupted write can never leave a truncated/empty file in .last_good_tag's place.

Not fixed this session — out of scope for the incident-clearance work server-26#155 was tracking; filing so it doesn't get lost. cio territory per CLAUDE.md (infra/deploy truth).

Filed by Claude Code, 2026-09-20, while deploy-checking #155.

Found while deploy-checking commit `fb0bb15` (server-26#155's incident-clearance fix). Prod IS correctly running the new code (`/health` confirms `git_sha: fb0bb15...`, healthy) — **this is a CI/rollback-tooling bug, not a prod outage.** **Evidence** (job 820, run 606, full log pulled): line ~122 of the Main Deploy step: ``` PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null || echo latest) echo "PREV_TAG=$PREV_TAG" ``` Logged output: `PREV_TAG=` — completely empty, not even the `latest` fallback. `git pull`, the image pulls, and `compose up -d` all ran and succeeded afterward (confirmed: new images pulled, containers came up, `/health` now serves the new SHA). But because `PREV_TAG` came back empty, the step at the end (`if [ -z "$PREV_TAG" ]; then ... exit 1; fi`) fails the whole job with "Could not determine the previous tag from deploy output - rollback target unknown," which in turn triggers "Rollback on failed health check" — which itself immediately no-ops ("No previous tag was captured ... cannot roll back automatically") since it also has no PREV_TAG to roll back to. **Root cause:** `cat /opt/drb/.last_good_tag 2>/dev/null || echo latest` only falls back to `latest` when `cat` fails (file missing, permission denied, etc). If the file EXISTS but is EMPTY (zero bytes), `cat` succeeds with exit 0 and empty stdout — the `||` never fires, and `PREV_TAG` is silently set to the empty string. The health-check step (deploy.yml ~line 228-234) is the only writer of this file, via `echo '${{ gitea.sha }}' > /opt/drb/.last_good_tag` — if that SSH write was ever interrupted mid-flight (network hiccup, runner timeout) after truncating but before writing the SHA, the file is left permanently empty, and every subsequent deploy reads the same empty value forever, since nothing ever rewrites it back to non-empty (the read happens before the write in each run). **Impact, ongoing:** checked runs 602 through 606 (5 most recent, 2026-09-13 through 2026-09-20) — every single one shows `Deploy to VM: failure` at the job level. The underlying `docker compose pull && up -d` appears to succeed regardless (spot-confirmed for 605 and 606 via direct `/health` SHA match), so **prod itself is not stuck** — but: 1. The rollback safety net (server-26#65's whole reason for existing) has not been able to function for at least a week — if a deploy DID actually break prod during this window, there was no automatic rollback available. 2. CI status is a false negative on every push — `deploy-check` skill users have to manually verify `/health` every time rather than trusting the Actions conclusion, which defeats part of the point of having it. 3. `.last_good_tag` needs to be manually repaired on the VM (`echo '<known-good-sha>' > /opt/drb/.last_good_tag`, e.g. the current live SHA) before the fallback will read `latest` again on any FUTURE empty-file case — that alone doesn't fix the underlying bug, since a future interrupted write reproduces the same empty-file state. **Suggested fix:** treat an empty (not just missing) file as "no previous tag" — e.g. `PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null); [ -n "$PREV_TAG" ] || PREV_TAG=latest`. Separately, consider writing the new SHA to a temp file and `mv`ing it into place atomically, so an interrupted write can never leave a truncated/empty file in `.last_good_tag`'s place. Not fixed this session — out of scope for the incident-clearance work server-26#155 was tracking; filing so it doesn't get lost. `cio` territory per CLAUDE.md (infra/deploy truth). Filed by Claude Code, 2026-09-20, while deploy-checking #155.

logan referenced this issue from a commit

2026-09-20 20:36:20 -04:00

deploy: treat an empty .last_good_tag the same as a missing one (#156)

logan commented

2026-09-20 20:43:06 -04:00

Fixed in f91d455, pushed to main. Verified on run 612: "Deploy to VM" reports success (first green deploy since the lock started), and /health confirms git_sha f91d4559 matches HEAD. The self-perpetuating lock is broken -- .last_good_tag is now correctly populated for future deploys.

Fixed in f91d455, pushed to main. Verified on run 612: "Deploy to VM" reports success (first green deploy since the lock started), and /health confirms git_sha f91d4559 matches HEAD. The self-perpetuating lock is broken -- .last_good_tag is now correctly populated for future deploys.

logan closed this issue

2026-09-20 20:43:07 -04:00

logan added the area/infra-deploy

type

bug

labels 2026-10-05 02:04:54 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#156