deploy.yml runs compose up -d before the health check with no rollback - blocks unattended pushes to the AI pipeline #65

New Issue

2026-08-24T23:41:26-04:00

logan commented

2026-08-24 23:41:26 -04:00

Decision 3, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31. From CTO draft #60 finding 3.

Server/.gitea/workflows/deploy.yml:120 runs docker compose ... up -d before the health check. If /health returns the right git_sha but the new code has a live logic bug - exactly the class of bug the correlator instrumentation exists to catch - nothing reverts. notify-failure (lines 152-181) only posts to Discord, and its own message text, that production is still running the previous build, is false in this scenario; it is true only if the pull or restart itself failed.

This is the precise failure mode an autonomously-pushing agent introduces: a commit that passes tests, returns 200, and silently corrupts incidents on live public-safety traffic for 12+ hours before a human looks.

Work

Gate the deploy: capture the previous image tags and SHA, up -d, health-check with retries, and on failure bring the previous tags back up automatically.
Fix the false notify-failure text so it states what actually happened.
Emit the rolled-back-to SHA in the notification.

In force until this lands (#62 Decision 3)

The unattended runner may push to main autonomously in general, but may not push, unattended, any change touching incident_correlator.py, llm_correlator.py, intelligence.py, or routers/upload.py. That work is staged on a branch and reported instead.

Interim mechanism, authorised by #62: after any pipeline-touching deploy the runner tails c2-core logs for about 10 minutes and does git revert plus push on a clear exception spike, rather than only reporting to Telegram.

Refs #62, #60, #57.

**Decision 3, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31.** From CTO draft #60 finding 3. `Server/.gitea/workflows/deploy.yml:120` runs `docker compose ... up -d` **before** the health check. If `/health` returns the right `git_sha` but the new code has a live logic bug - exactly the class of bug the correlator instrumentation exists to catch - nothing reverts. `notify-failure` (lines 152-181) only posts to Discord, and its own message text, that production is still running the previous build, is **false** in this scenario; it is true only if the pull or restart itself failed. This is the precise failure mode an autonomously-pushing agent introduces: a commit that passes tests, returns 200, and silently corrupts incidents on live public-safety traffic for 12+ hours before a human looks. ## Work 1. Gate the deploy: capture the previous image tags and SHA, `up -d`, health-check with retries, and on failure bring the previous tags back up automatically. 2. Fix the false `notify-failure` text so it states what actually happened. 3. Emit the rolled-back-to SHA in the notification. ## In force until this lands (#62 Decision 3) The unattended runner **may** push to `main` autonomously in general, but **may not** push, unattended, any change touching `incident_correlator.py`, `llm_correlator.py`, `intelligence.py`, or `routers/upload.py`. That work is staged on a branch and reported instead. Interim mechanism, authorised by #62: after any pipeline-touching deploy the runner tails `c2-core` logs for about 10 minutes and does `git revert` plus push on a clear exception spike, rather than only reporting to Telegram. Refs #62, #60, #57.

logan referenced this issue

2026-08-24 23:42:52 -04:00

Board 2026-08-24 - Unattended operating cadence and GOALS tracking - FINAL MINUTES #62

logan referenced this issue

2026-08-24 23:42:52 -04:00

Board 2026-08-24 — Unattended operating cadence and GOALS tracking — CTO draft #60

logan referenced this issue

2026-08-24 23:42:53 -04:00

Board 2026-08-24 — Unattended operating cadence and GOALS tracking — CISO draft #61

logan referenced this issue

2026-08-25 21:30:50 -04:00

_call_fits_incident still uses signed idle, so the sweep path skips both over-merge vetoes on back-dated calls #74

logan referenced this issue

2026-08-25 21:30:50 -04:00

Per-system correlation_enabled=false is ignored: the call still correlates, with empty tags, down the thin/recency path #75

logan referenced this issue from a commit

2026-08-25 21:34:16 -04:00

Roll back a bad deploy instead of leaving it live (server-26#65)

logan commented

2026-08-25 21:42:35 -04:00

Fixed in commit 5fc4e2c57b on main.

deploy.yml now captures the previously-verified tag from /opt/drb/.last_good_tag before switching (written only after a prior health check confirmed its SHA live), health-checks the new deploy with the existing bounded 20x5s retry, and on any failure (pull, restart, or SHA mismatch) runs a new Rollback step that redeploys the previous tag and re-verifies via the same git_sha check rather than trusting mere liveness. The job still fails loudly either way. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional "still running the previous build" claim, which was false whenever a bad build passed the SHA check but had a live logic bug.

Verified: c2-core pytest 250 passed, edge-node pytest 185 passed, edge-node flake8 clean, frontend tsc clean, YAML parses. Pushed to main, watched Gitea Actions run 561 (Build/Deploy/notify-failure all completed success/success/skipped), and confirmed prod /health now reports git_sha 5fc4e2c57b -- the deploy actually landed, not just green CI.

This unblocks #62 decision 9 (autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py, routers/upload.py were frozen pending this).

Fixed in commit 5fc4e2c57bdb536cd5dd5af357552d2fc6ae3ad6 on main. deploy.yml now captures the previously-verified tag from /opt/drb/.last_good_tag before switching (written only after a prior health check confirmed its SHA live), health-checks the new deploy with the existing bounded 20x5s retry, and on any failure (pull, restart, or SHA mismatch) runs a new Rollback step that redeploys the previous tag and re-verifies via the same git_sha check rather than trusting mere liveness. The job still fails loudly either way. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional "still running the previous build" claim, which was false whenever a bad build passed the SHA check but had a live logic bug. Verified: c2-core pytest 250 passed, edge-node pytest 185 passed, edge-node flake8 clean, frontend tsc clean, YAML parses. Pushed to main, watched Gitea Actions run 561 (Build/Deploy/notify-failure all completed success/success/skipped), and confirmed prod /health now reports git_sha 5fc4e2c57bdb536cd5dd5af357552d2fc6ae3ad6 -- the deploy actually landed, not just green CI. This unblocks #62 decision 9 (autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py, routers/upload.py were frozen pending this).

logan closed this issue

2026-08-25 21:42:35 -04:00

logan referenced this issue

2026-08-30 02:40:41 -04:00

Encode the runner authorisation boundary and credential-escalation rule into drb-worksession.md #68

logan added the area/infra-deploy

type

bug

labels 2026-10-05 02:04:34 -04:00

Sign in to join this conversation.