Every push between 427d2a9f (run 517) and 4919b022 (run 531) built successfully and then FAILED to deploy. Production kept serving the image from run 516 (4842725a, 2026-08-18) for two days while fifteen "successful looking" pushes went by.
What was NOT running in production during that window:
The entire frontend redesign, chunks 2 through 8 — tokens, primitives, nav/IA, the MapView rewrite that draws incident paths, the Live view, incident detail, incidents browse. This is why the UI still looked unchanged after "the update": only chunk 1 was ever deployed.
427d2a9f — the loud OpenAI billing failure.
157be0c0 — the Firebase auth handler proxy.
90a04120 / bc191fb5 — the correlation debug view fixes.
c3fe2a34 / 1a563c99 — the org backfill fixes.
Root cause (from job 624's log, run 531): the Deploy step's docker compose pull on the VM died in containerd:
failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:67e4f550…)
to overlayfs …: failed to Lchown "/var/lib/containerd/…/site-packages/google/ai/
generativelanguage_v1beta3/services/model_service/transports/rest.py" for UID 0, GID 0:
… no such file or directory
That is a corrupted or partially-extracted layer in the VM's containerd snapshot store, not a code problem — the build job passed every time. Worth checking the VM's disk usage; a full or flaky disk produces exactly this. It has since started working again on its own (runs 534 and 535 both deployed cleanly), which is its own reason to distrust it.
The deeper problem is not the layer error, it is that nobody noticed for two days. A red run in Gitea Actions is only visible to someone who opens Gitea. Same shape as server-26#14 (AI provider dies, logs an ERROR, nobody reads it) and the same shape as the retired Gemini model IDs surviving for days.
Wanted:
Notify on deploy failure — Discord webhook from the workflow's failure path is the cheapest thing that works.
Make the failure recoverable rather than fatal: on a layer-extraction error, docker image prune and retry the pull once before giving up.
Consider having the health check assert the deployed commit SHA, so a succeeded deploy that did not actually replace the running container still fails loudly.
Found during the 2026-08-20 03:00 review, while investigating why a correlation dump showed no LLM activity.
**Every push between `427d2a9f` (run 517) and `4919b022` (run 531) built successfully and then FAILED to deploy.** Production kept serving the image from run 516 (`4842725a`, 2026-08-18) for two days while fifteen "successful looking" pushes went by.
What was NOT running in production during that window:
- The entire frontend redesign, chunks 2 through 8 — tokens, primitives, nav/IA, the MapView rewrite that draws incident paths, the Live view, incident detail, incidents browse. **This is why the UI still looked unchanged after "the update": only chunk 1 was ever deployed.**
- `427d2a9f` — the loud OpenAI billing failure.
- `157be0c0` — the Firebase auth handler proxy.
- `90a04120` / `bc191fb5` — the correlation debug view fixes.
- `c3fe2a34` / `1a563c99` — the org backfill fixes.
**Root cause** (from job 624's log, run 531): the `Deploy` step's `docker compose pull` on the VM died in containerd:
```
failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:67e4f550…)
to overlayfs …: failed to Lchown "/var/lib/containerd/…/site-packages/google/ai/
generativelanguage_v1beta3/services/model_service/transports/rest.py" for UID 0, GID 0:
… no such file or directory
```
That is a corrupted or partially-extracted layer in the VM's containerd snapshot store, not a code problem — the build job passed every time. Worth checking the VM's disk usage; a full or flaky disk produces exactly this. It has since started working again on its own (runs 534 and 535 both deployed cleanly), which is its own reason to distrust it.
**The deeper problem is not the layer error, it is that nobody noticed for two days.** A red run in Gitea Actions is only visible to someone who opens Gitea. Same shape as server-26#14 (AI provider dies, logs an ERROR, nobody reads it) and the same shape as the retired Gemini model IDs surviving for days.
Wanted:
1. Notify on deploy failure — Discord webhook from the workflow's failure path is the cheapest thing that works.
2. Make the failure recoverable rather than fatal: on a layer-extraction error, `docker image prune` and retry the pull once before giving up.
3. Consider having the health check assert the deployed commit SHA, so a *succeeded* deploy that did not actually replace the running container still fails loudly.
Found during the 2026-08-20 03:00 review, while investigating why a correlation dump showed no LLM activity.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Every push between
427d2a9f(run 517) and4919b022(run 531) built successfully and then FAILED to deploy. Production kept serving the image from run 516 (4842725a, 2026-08-18) for two days while fifteen "successful looking" pushes went by.What was NOT running in production during that window:
427d2a9f— the loud OpenAI billing failure.157be0c0— the Firebase auth handler proxy.90a04120/bc191fb5— the correlation debug view fixes.c3fe2a34/1a563c99— the org backfill fixes.Root cause (from job 624's log, run 531): the
Deploystep'sdocker compose pullon the VM died in containerd:That is a corrupted or partially-extracted layer in the VM's containerd snapshot store, not a code problem — the build job passed every time. Worth checking the VM's disk usage; a full or flaky disk produces exactly this. It has since started working again on its own (runs 534 and 535 both deployed cleanly), which is its own reason to distrust it.
The deeper problem is not the layer error, it is that nobody noticed for two days. A red run in Gitea Actions is only visible to someone who opens Gitea. Same shape as server-26#14 (AI provider dies, logs an ERROR, nobody reads it) and the same shape as the retired Gemini model IDs surviving for days.
Wanted:
docker image pruneand retry the pull once before giving up.Found during the 2026-08-20 03:00 review, while investigating why a correlation dump showed no LLM activity.