Decision 3, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31. From CTO draft #60 finding 3.
Server/.gitea/workflows/deploy.yml:120 runs docker compose ... up -dbefore the health check. If /health returns the right git_sha but the new code has a live logic bug - exactly the class of bug the correlator instrumentation exists to catch - nothing reverts. notify-failure (lines 152-181) only posts to Discord, and its own message text, that production is still running the previous build, is false in this scenario; it is true only if the pull or restart itself failed.
This is the precise failure mode an autonomously-pushing agent introduces: a commit that passes tests, returns 200, and silently corrupts incidents on live public-safety traffic for 12+ hours before a human looks.
Work
Gate the deploy: capture the previous image tags and SHA, up -d, health-check with retries, and on failure bring the previous tags back up automatically.
Fix the false notify-failure text so it states what actually happened.
The unattended runner may push to main autonomously in general, but may not push, unattended, any change touching incident_correlator.py, llm_correlator.py, intelligence.py, or routers/upload.py. That work is staged on a branch and reported instead.
Interim mechanism, authorised by #62: after any pipeline-touching deploy the runner tails c2-core logs for about 10 minutes and does git revert plus push on a clear exception spike, rather than only reporting to Telegram.
**Decision 3, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31.** From CTO draft #60 finding 3.
`Server/.gitea/workflows/deploy.yml:120` runs `docker compose ... up -d` **before** the health check. If `/health` returns the right `git_sha` but the new code has a live logic bug - exactly the class of bug the correlator instrumentation exists to catch - nothing reverts. `notify-failure` (lines 152-181) only posts to Discord, and its own message text, that production is still running the previous build, is **false** in this scenario; it is true only if the pull or restart itself failed.
This is the precise failure mode an autonomously-pushing agent introduces: a commit that passes tests, returns 200, and silently corrupts incidents on live public-safety traffic for 12+ hours before a human looks.
## Work
1. Gate the deploy: capture the previous image tags and SHA, `up -d`, health-check with retries, and on failure bring the previous tags back up automatically.
2. Fix the false `notify-failure` text so it states what actually happened.
3. Emit the rolled-back-to SHA in the notification.
## In force until this lands (#62 Decision 3)
The unattended runner **may** push to `main` autonomously in general, but **may not** push, unattended, any change touching `incident_correlator.py`, `llm_correlator.py`, `intelligence.py`, or `routers/upload.py`. That work is staged on a branch and reported instead.
Interim mechanism, authorised by #62: after any pipeline-touching deploy the runner tails `c2-core` logs for about 10 minutes and does `git revert` plus push on a clear exception spike, rather than only reporting to Telegram.
Refs #62, #60, #57.
deploy.yml now captures the previously-verified tag from /opt/drb/.last_good_tag before switching (written only after a prior health check confirmed its SHA live), health-checks the new deploy with the existing bounded 20x5s retry, and on any failure (pull, restart, or SHA mismatch) runs a new Rollback step that redeploys the previous tag and re-verifies via the same git_sha check rather than trusting mere liveness. The job still fails loudly either way. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional "still running the previous build" claim, which was false whenever a bad build passed the SHA check but had a live logic bug.
Verified: c2-core pytest 250 passed, edge-node pytest 185 passed, edge-node flake8 clean, frontend tsc clean, YAML parses. Pushed to main, watched Gitea Actions run 561 (Build/Deploy/notify-failure all completed success/success/skipped), and confirmed prod /health now reports git_sha 5fc4e2c57b -- the deploy actually landed, not just green CI.
This unblocks #62 decision 9 (autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py, routers/upload.py were frozen pending this).
Fixed in commit 5fc4e2c57bdb536cd5dd5af357552d2fc6ae3ad6 on main.
deploy.yml now captures the previously-verified tag from /opt/drb/.last_good_tag before switching (written only after a prior health check confirmed its SHA live), health-checks the new deploy with the existing bounded 20x5s retry, and on any failure (pull, restart, or SHA mismatch) runs a new Rollback step that redeploys the previous tag and re-verifies via the same git_sha check rather than trusting mere liveness. The job still fails loudly either way. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional "still running the previous build" claim, which was false whenever a bad build passed the SHA check but had a live logic bug.
Verified: c2-core pytest 250 passed, edge-node pytest 185 passed, edge-node flake8 clean, frontend tsc clean, YAML parses. Pushed to main, watched Gitea Actions run 561 (Build/Deploy/notify-failure all completed success/success/skipped), and confirmed prod /health now reports git_sha 5fc4e2c57bdb536cd5dd5af357552d2fc6ae3ad6 -- the deploy actually landed, not just green CI.
This unblocks #62 decision 9 (autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py, routers/upload.py were frozen pending this).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Decision 3, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31. From CTO draft #60 finding 3.
Server/.gitea/workflows/deploy.yml:120runsdocker compose ... up -dbefore the health check. If/healthreturns the rightgit_shabut the new code has a live logic bug - exactly the class of bug the correlator instrumentation exists to catch - nothing reverts.notify-failure(lines 152-181) only posts to Discord, and its own message text, that production is still running the previous build, is false in this scenario; it is true only if the pull or restart itself failed.This is the precise failure mode an autonomously-pushing agent introduces: a commit that passes tests, returns 200, and silently corrupts incidents on live public-safety traffic for 12+ hours before a human looks.
Work
up -d, health-check with retries, and on failure bring the previous tags back up automatically.notify-failuretext so it states what actually happened.In force until this lands (#62 Decision 3)
The unattended runner may push to
mainautonomously in general, but may not push, unattended, any change touchingincident_correlator.py,llm_correlator.py,intelligence.py, orrouters/upload.py. That work is staged on a branch and reported instead.Interim mechanism, authorised by #62: after any pipeline-touching deploy the runner tails
c2-corelogs for about 10 minutes and doesgit revertplus push on a clear exception spike, rather than only reporting to Telegram.Refs #62, #60, #57.
Fixed in commit
5fc4e2c57bon main.deploy.yml now captures the previously-verified tag from /opt/drb/.last_good_tag before switching (written only after a prior health check confirmed its SHA live), health-checks the new deploy with the existing bounded 20x5s retry, and on any failure (pull, restart, or SHA mismatch) runs a new Rollback step that redeploys the previous tag and re-verifies via the same git_sha check rather than trusting mere liveness. The job still fails loudly either way. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional "still running the previous build" claim, which was false whenever a bad build passed the SHA check but had a live logic bug.
Verified: c2-core pytest 250 passed, edge-node pytest 185 passed, edge-node flake8 clean, frontend tsc clean, YAML parses. Pushed to main, watched Gitea Actions run 561 (Build/Deploy/notify-failure all completed success/success/skipped), and confirmed prod /health now reports git_sha
5fc4e2c57b-- the deploy actually landed, not just green CI.This unblocks #62 decision 9 (autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py, routers/upload.py were frozen pending this).