Hit live on the unattended run of 2026-09-01. Detected, diagnosed and worked around in the same run — production is now correct — but the defect is intermittent and will recur.
#11 pushing manifest for ***/c2-core:latest ... 1.1s done 06:46:50 <- succeeded
#10 ERROR: failed to push ***/discord-bot:latest:
failed to authorize: failed to fetch oauth token: unauthorized 06:47:09 <- failed
c2-core pushed successfully at 06:46:50 and discord-bot failed authorization 19 seconds later on the same credential. That is not an invalid or revoked credential — it is a short-lived registry token expiring part-way through a multi-image build. The frontend image never got its turn.
Job outcomes for run 570:
Job
Result
Build & push images
failure
Deploy to VM
skipped
Report a failed deploy
success
Why this is worse than an ordinary flake
Deploy to VM is gated on the build job, so a late-stage push failure silently skips the deploy entirely. Production stayed on b722223 with the viewer-triggerable OpenAI spend leak (#81) still open, while the fix sat on main looking shipped. This is server-26#21's failure mode arriving through a new door: the tracker and main say shipped, prod disagrees.
It is also partially destructive to the registry state: c2-core:latest and c2-core:<sha> were pushed and are now newer than discord-bot:latest and frontend:latest. A docker compose pull between the failed build and the retry would have produced a mixed-version deployment — a c2-core from one commit beside a bot and frontend from an older one. Nothing detected that; it was luck that no deploy ran in the gap.
Workaround applied this run
Empty commit 85393bd pushed to re-trigger. Run 571 went green on both jobs and /health now reports 85393bd, so 8b6c170 is live and #81 is genuinely closed in production. This is a workaround, not a fix — the next multi-image build can lose the token at the same point.
Note the Gitea API token available to unattended runs has scope write:issue,read:repository only, so POST /actions/runs/{id}/rerun returns required=[write:repository]. An unattended run cannot re-run a workflow and must push an empty commit instead. That is worth deciding on deliberately rather than leaving as an accident of scope.
Fix shape
Ranked, cheapest first:
Re-authenticate before each push, or move docker/login-action so the credential is refreshed per image rather than once per job. Directly addresses the observed 19-second-later expiry.
Retry the push step on unauthorized (the workflow already has a prune-and-retry for the containerd Lchown failure — this is the same shape of problem and deserves the same treatment).
Push all three images atomically, or none — build all, then push all, so a mid-sequence failure cannot leave the registry with mismatched :latest tags.
Longer-lived or non-expiring registry credential for CI, if the Gitea package registry supports one.
Also worth having
Nothing alerts on "build succeeded partially". The Report a failed deploy job ran and the failure was still only discovered because an unattended run checked /health against origin/main by hand. Related: #21.
Hit live on the unattended run of 2026-09-01. **Detected, diagnosed and worked around in the same run** — production is now correct — but the defect is intermittent and will recur.
## What happened
Run **570** (`8b6c170`, the #81 spend-leak fix) failed:
```
#11 pushing manifest for ***/c2-core:latest ... 1.1s done 06:46:50 <- succeeded
#10 ERROR: failed to push ***/discord-bot:latest:
failed to authorize: failed to fetch oauth token: unauthorized 06:47:09 <- failed
```
**`c2-core` pushed successfully at 06:46:50 and `discord-bot` failed authorization 19 seconds later on the same credential.** That is not an invalid or revoked credential — it is a **short-lived registry token expiring part-way through a multi-image build**. The `frontend` image never got its turn.
Job outcomes for run 570:
| Job | Result |
|---|---|
| Build & push images | **failure** |
| Deploy to VM | **skipped** |
| Report a failed deploy | success |
## Why this is worse than an ordinary flake
`Deploy to VM` is gated on the build job, so a late-stage push failure **silently skips the deploy entirely**. Production stayed on `b722223` with the viewer-triggerable OpenAI spend leak (#81) still open, while the fix sat on `main` looking shipped. This is server-26#21's failure mode arriving through a new door: the tracker and `main` say shipped, prod disagrees.
It is also **partially destructive to the registry state**: `c2-core:latest` and `c2-core:<sha>` were pushed and are now newer than `discord-bot:latest` and `frontend:latest`. A `docker compose pull` between the failed build and the retry would have produced a **mixed-version deployment** — a c2-core from one commit beside a bot and frontend from an older one. Nothing detected that; it was luck that no deploy ran in the gap.
## Workaround applied this run
Empty commit `85393bd` pushed to re-trigger. Run **571** went green on both jobs and `/health` now reports `85393bd`, so `8b6c170` is live and #81 is genuinely closed in production. **This is a workaround, not a fix** — the next multi-image build can lose the token at the same point.
Note the Gitea API token available to unattended runs has scope `write:issue,read:repository` only, so `POST /actions/runs/{id}/rerun` returns `required=[write:repository]`. An unattended run **cannot re-run a workflow** and must push an empty commit instead. That is worth deciding on deliberately rather than leaving as an accident of scope.
## Fix shape
Ranked, cheapest first:
1. **Re-authenticate before each push**, or move `docker/login-action` so the credential is refreshed per image rather than once per job. Directly addresses the observed 19-second-later expiry.
2. **Retry the push step** on `unauthorized` (the workflow already has a prune-and-retry for the containerd Lchown failure — this is the same shape of problem and deserves the same treatment).
3. **Push all three images atomically, or none** — build all, then push all, so a mid-sequence failure cannot leave the registry with mismatched `:latest` tags.
4. Longer-lived or non-expiring registry credential for CI, if the Gitea package registry supports one.
## Also worth having
Nothing alerts on "build succeeded partially". The `Report a failed deploy` job ran and the failure was still only discovered because an unattended run checked `/health` against `origin/main` by hand. Related: #21.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Hit live on the unattended run of 2026-09-01. Detected, diagnosed and worked around in the same run — production is now correct — but the defect is intermittent and will recur.
What happened
Run 570 (
8b6c170, the #81 spend-leak fix) failed:c2-corepushed successfully at 06:46:50 anddiscord-botfailed authorization 19 seconds later on the same credential. That is not an invalid or revoked credential — it is a short-lived registry token expiring part-way through a multi-image build. Thefrontendimage never got its turn.Job outcomes for run 570:
Why this is worse than an ordinary flake
Deploy to VMis gated on the build job, so a late-stage push failure silently skips the deploy entirely. Production stayed onb722223with the viewer-triggerable OpenAI spend leak (#81) still open, while the fix sat onmainlooking shipped. This is server-26#21's failure mode arriving through a new door: the tracker andmainsay shipped, prod disagrees.It is also partially destructive to the registry state:
c2-core:latestandc2-core:<sha>were pushed and are now newer thandiscord-bot:latestandfrontend:latest. Adocker compose pullbetween the failed build and the retry would have produced a mixed-version deployment — a c2-core from one commit beside a bot and frontend from an older one. Nothing detected that; it was luck that no deploy ran in the gap.Workaround applied this run
Empty commit
85393bdpushed to re-trigger. Run 571 went green on both jobs and/healthnow reports85393bd, so8b6c170is live and #81 is genuinely closed in production. This is a workaround, not a fix — the next multi-image build can lose the token at the same point.Note the Gitea API token available to unattended runs has scope
write:issue,read:repositoryonly, soPOST /actions/runs/{id}/rerunreturnsrequired=[write:repository]. An unattended run cannot re-run a workflow and must push an empty commit instead. That is worth deciding on deliberately rather than leaving as an accident of scope.Fix shape
Ranked, cheapest first:
docker/login-actionso the credential is refreshed per image rather than once per job. Directly addresses the observed 19-second-later expiry.unauthorized(the workflow already has a prune-and-retry for the containerd Lchown failure — this is the same shape of problem and deserves the same treatment).:latesttags.Also worth having
Nothing alerts on "build succeeded partially". The
Report a failed deployjob ran and the failure was still only discovered because an unattended run checked/healthagainstorigin/mainby hand. Related: #21.