Registry credential expires mid-build, so a deploy fails after some images have already pushed — cost us a night's deploy on 2026-09-01 #100

New Issue

2026-09-01T02:58:58-04:00

logan commented

2026-09-01 02:58:58 -04:00

Hit live on the unattended run of 2026-09-01. Detected, diagnosed and worked around in the same run — production is now correct — but the defect is intermittent and will recur.

What happened

Run 570 (8b6c170, the #81 spend-leak fix) failed:

#11 pushing manifest for ***/c2-core:latest ... 1.1s done        06:46:50   <- succeeded
#10 ERROR: failed to push ***/discord-bot:latest:
    failed to authorize: failed to fetch oauth token: unauthorized  06:47:09  <- failed

c2-core pushed successfully at 06:46:50 and discord-bot failed authorization 19 seconds later on the same credential. That is not an invalid or revoked credential — it is a short-lived registry token expiring part-way through a multi-image build. The frontend image never got its turn.

Job outcomes for run 570:

Job	Result
Build & push images	failure
Deploy to VM	skipped
Report a failed deploy	success

Why this is worse than an ordinary flake

Deploy to VM is gated on the build job, so a late-stage push failure silently skips the deploy entirely. Production stayed on b722223 with the viewer-triggerable OpenAI spend leak (#81) still open, while the fix sat on main looking shipped. This is server-26#21's failure mode arriving through a new door: the tracker and main say shipped, prod disagrees.

It is also partially destructive to the registry state: c2-core:latest and c2-core:<sha> were pushed and are now newer than discord-bot:latest and frontend:latest. A docker compose pull between the failed build and the retry would have produced a mixed-version deployment — a c2-core from one commit beside a bot and frontend from an older one. Nothing detected that; it was luck that no deploy ran in the gap.

Workaround applied this run

Empty commit 85393bd pushed to re-trigger. Run 571 went green on both jobs and /health now reports 85393bd, so 8b6c170 is live and #81 is genuinely closed in production. This is a workaround, not a fix — the next multi-image build can lose the token at the same point.

Note the Gitea API token available to unattended runs has scope write:issue,read:repository only, so POST /actions/runs/{id}/rerun returns required=[write:repository]. An unattended run cannot re-run a workflow and must push an empty commit instead. That is worth deciding on deliberately rather than leaving as an accident of scope.

Fix shape

Ranked, cheapest first:

Re-authenticate before each push, or move docker/login-action so the credential is refreshed per image rather than once per job. Directly addresses the observed 19-second-later expiry.
Retry the push step on unauthorized (the workflow already has a prune-and-retry for the containerd Lchown failure — this is the same shape of problem and deserves the same treatment).
Push all three images atomically, or none — build all, then push all, so a mid-sequence failure cannot leave the registry with mismatched :latest tags.
Longer-lived or non-expiring registry credential for CI, if the Gitea package registry supports one.

Also worth having

Nothing alerts on "build succeeded partially". The Report a failed deploy job ran and the failure was still only discovered because an unattended run checked /health against origin/main by hand. Related: #21.

Hit live on the unattended run of 2026-09-01. **Detected, diagnosed and worked around in the same run** — production is now correct — but the defect is intermittent and will recur. ## What happened Run **570** (`8b6c170`, the #81 spend-leak fix) failed: ``` #11 pushing manifest for ***/c2-core:latest ... 1.1s done 06:46:50 <- succeeded #10 ERROR: failed to push ***/discord-bot:latest: failed to authorize: failed to fetch oauth token: unauthorized 06:47:09 <- failed ``` **`c2-core` pushed successfully at 06:46:50 and `discord-bot` failed authorization 19 seconds later on the same credential.** That is not an invalid or revoked credential — it is a **short-lived registry token expiring part-way through a multi-image build**. The `frontend` image never got its turn. Job outcomes for run 570: | Job | Result | |---|---| | Build & push images | **failure** | | Deploy to VM | **skipped** | | Report a failed deploy | success | ## Why this is worse than an ordinary flake `Deploy to VM` is gated on the build job, so a late-stage push failure **silently skips the deploy entirely**. Production stayed on `b722223` with the viewer-triggerable OpenAI spend leak (#81) still open, while the fix sat on `main` looking shipped. This is server-26#21's failure mode arriving through a new door: the tracker and `main` say shipped, prod disagrees. It is also **partially destructive to the registry state**: `c2-core:latest` and `c2-core:<sha>` were pushed and are now newer than `discord-bot:latest` and `frontend:latest`. A `docker compose pull` between the failed build and the retry would have produced a **mixed-version deployment** — a c2-core from one commit beside a bot and frontend from an older one. Nothing detected that; it was luck that no deploy ran in the gap. ## Workaround applied this run Empty commit `85393bd` pushed to re-trigger. Run **571** went green on both jobs and `/health` now reports `85393bd`, so `8b6c170` is live and #81 is genuinely closed in production. **This is a workaround, not a fix** — the next multi-image build can lose the token at the same point. Note the Gitea API token available to unattended runs has scope `write:issue,read:repository` only, so `POST /actions/runs/{id}/rerun` returns `required=[write:repository]`. An unattended run **cannot re-run a workflow** and must push an empty commit instead. That is worth deciding on deliberately rather than leaving as an accident of scope. ## Fix shape Ranked, cheapest first: 1. **Re-authenticate before each push**, or move `docker/login-action` so the credential is refreshed per image rather than once per job. Directly addresses the observed 19-second-later expiry. 2. **Retry the push step** on `unauthorized` (the workflow already has a prune-and-retry for the containerd Lchown failure — this is the same shape of problem and deserves the same treatment). 3. **Push all three images atomically, or none** — build all, then push all, so a mid-sequence failure cannot leave the registry with mismatched `:latest` tags. 4. Longer-lived or non-expiring registry credential for CI, if the Gitea package registry supports one. ## Also worth having Nothing alerts on "build succeeded partially". The `Report a failed deploy` job ran and the failure was still only discovered because an unattended run checked `/health` against `origin/main` by hand. Related: #21.

logan added the area/infra-deploy

labels 2026-10-05 02:04:44 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#100