Deploy failed silently on 15 consecutive pushes — production ran 08-18 code until 08-20 #21

New Issue

2026-08-20T03:17:18-04:00

logan commented

2026-08-20 03:17:18 -04:00

Every push between 427d2a9f (run 517) and 4919b022 (run 531) built successfully and then FAILED to deploy. Production kept serving the image from run 516 (4842725a, 2026-08-18) for two days while fifteen "successful looking" pushes went by.

What was NOT running in production during that window:

The entire frontend redesign, chunks 2 through 8 — tokens, primitives, nav/IA, the MapView rewrite that draws incident paths, the Live view, incident detail, incidents browse. This is why the UI still looked unchanged after "the update": only chunk 1 was ever deployed.
427d2a9f — the loud OpenAI billing failure.
157be0c0 — the Firebase auth handler proxy.
90a04120 / bc191fb5 — the correlation debug view fixes.
c3fe2a34 / 1a563c99 — the org backfill fixes.

Root cause (from job 624's log, run 531): the Deploy step's docker compose pull on the VM died in containerd:

failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:67e4f550…)
to overlayfs …: failed to Lchown "/var/lib/containerd/…/site-packages/google/ai/
generativelanguage_v1beta3/services/model_service/transports/rest.py" for UID 0, GID 0:
… no such file or directory

That is a corrupted or partially-extracted layer in the VM's containerd snapshot store, not a code problem — the build job passed every time. Worth checking the VM's disk usage; a full or flaky disk produces exactly this. It has since started working again on its own (runs 534 and 535 both deployed cleanly), which is its own reason to distrust it.

The deeper problem is not the layer error, it is that nobody noticed for two days. A red run in Gitea Actions is only visible to someone who opens Gitea. Same shape as server-26#14 (AI provider dies, logs an ERROR, nobody reads it) and the same shape as the retired Gemini model IDs surviving for days.

Wanted:

Notify on deploy failure — Discord webhook from the workflow's failure path is the cheapest thing that works.
Make the failure recoverable rather than fatal: on a layer-extraction error, docker image prune and retry the pull once before giving up.
Consider having the health check assert the deployed commit SHA, so a succeeded deploy that did not actually replace the running container still fails loudly.

Found during the 2026-08-20 03:00 review, while investigating why a correlation dump showed no LLM activity.

**Every push between `427d2a9f` (run 517) and `4919b022` (run 531) built successfully and then FAILED to deploy.** Production kept serving the image from run 516 (`4842725a`, 2026-08-18) for two days while fifteen "successful looking" pushes went by. What was NOT running in production during that window: - The entire frontend redesign, chunks 2 through 8 — tokens, primitives, nav/IA, the MapView rewrite that draws incident paths, the Live view, incident detail, incidents browse. **This is why the UI still looked unchanged after "the update": only chunk 1 was ever deployed.** - `427d2a9f` — the loud OpenAI billing failure. - `157be0c0` — the Firebase auth handler proxy. - `90a04120` / `bc191fb5` — the correlation debug view fixes. - `c3fe2a34` / `1a563c99` — the org backfill fixes. **Root cause** (from job 624's log, run 531): the `Deploy` step's `docker compose pull` on the VM died in containerd: ``` failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:67e4f550…) to overlayfs …: failed to Lchown "/var/lib/containerd/…/site-packages/google/ai/ generativelanguage_v1beta3/services/model_service/transports/rest.py" for UID 0, GID 0: … no such file or directory ``` That is a corrupted or partially-extracted layer in the VM's containerd snapshot store, not a code problem — the build job passed every time. Worth checking the VM's disk usage; a full or flaky disk produces exactly this. It has since started working again on its own (runs 534 and 535 both deployed cleanly), which is its own reason to distrust it. **The deeper problem is not the layer error, it is that nobody noticed for two days.** A red run in Gitea Actions is only visible to someone who opens Gitea. Same shape as server-26#14 (AI provider dies, logs an ERROR, nobody reads it) and the same shape as the retired Gemini model IDs surviving for days. Wanted: 1. Notify on deploy failure — Discord webhook from the workflow's failure path is the cheapest thing that works. 2. Make the failure recoverable rather than fatal: on a layer-extraction error, `docker image prune` and retry the pull once before giving up. 3. Consider having the health check assert the deployed commit SHA, so a *succeeded* deploy that did not actually replace the running container still fails loudly. Found during the 2026-08-20 03:00 review, while investigating why a correlation dump showed no LLM activity.

logan referenced this issue from a commit

2026-08-23 01:26:34 -04:00

Make a failed deploy impossible to miss, and a wildcard CORS harmless

logan closed this issue

2026-08-23 01:26:34 -04:00

logan referenced this issue from a commit

2026-08-23 01:42:41 -04:00

Deploy the commit's own images instead of :latest

logan referenced this issue

2026-08-23 02:21:49 -04:00

React error #310 white-screens /systems and /nodes #32

logan referenced this issue