Roll back a bad deploy instead of leaving it live (server-26#65)
deploy.yml ran `compose up -d` before the health check and never reverted on failure. A build that passes tests, returns 200 on /health with the right git_sha, but has a live logic bug (exactly the class of bug the correlator instrumentation exists to catch) would stay live indefinitely - notify-failure would even claim production was "still running the previous build", which is false in that scenario. Deploy step now reads /opt/drb/.last_good_tag (written only after a prior deploy's own health check confirmed its SHA) to capture the previously- verified tag before switching, and emits it as a step output. Health check is unchanged in shape (bounded 20x5s retry, still requires the polled git_sha to match) but now persists the new SHA as the rollback target only once confirmed live. A new Rollback step runs on any failure above, re-deploys the previous tag, and re-verifies via the same git_sha check rather than trusting mere liveness - then fails the job loudly either way, since the push itself was still bad. notify-failure now reports what actually happened (rollback succeeded/failed/skipped and to which SHA) instead of the old unconditional claim. This unblocks #62 decision 9: autonomous pushes to incident_correlator.py, llm_correlator.py, intelligence.py and routers/upload.py were frozen until this rollback path landed. Refs #65, #62, #60, #57.
This commit is contained in:
Logan Cusano
+126
-4
@@ -68,6 +68,10 @@ jobs:
|
||||
name: Deploy to VM
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
prev_sha: ${{ steps.deploy.outputs.prev_sha }}
|
||||
rollback_status: ${{ steps.rollback.outputs.status }}
|
||||
rollback_sha: ${{ steps.rollback.outputs.rolled_back_to }}
|
||||
|
||||
steps:
|
||||
- name: Check runner outbound IP
|
||||
@@ -80,19 +84,33 @@ jobs:
|
||||
ssh-keygen -l -f /tmp/deploy_key
|
||||
|
||||
- name: Deploy
|
||||
id: deploy
|
||||
run: |
|
||||
ssh -o StrictHostKeyChecking=no \
|
||||
set -o pipefail
|
||||
OUTPUT=$(ssh -o StrictHostKeyChecking=no \
|
||||
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
|
||||
-o ConnectTimeout=15 \
|
||||
-v \
|
||||
-i /tmp/deploy_key \
|
||||
drb@${{ secrets.SERVER_IP }} << 'ENDSSH'
|
||||
drb@${{ secrets.SERVER_IP }} << 'ENDSSH' | tee /dev/stderr
|
||||
set -e
|
||||
cd /opt/drb
|
||||
|
||||
# Update compose files + mosquitto config
|
||||
git pull origin main
|
||||
|
||||
# server-26#65: capture what is actually live BEFORE switching, so
|
||||
# a bad deploy has something concrete to fall back to. This reads
|
||||
# from a state file rather than re-deriving it from git log,
|
||||
# because a PRIOR deploy could itself have failed and already
|
||||
# rolled back to something older than HEAD~1 -- the file is only
|
||||
# ever written by the Health check step below, after that step
|
||||
# has confirmed the tag it names actually answered /health. A
|
||||
# fresh VM with no file yet falls back to :latest, same escape
|
||||
# hatch as a manual `up -d` with no TAG set.
|
||||
PREV_TAG=$(cat /opt/drb/.last_good_tag 2>/dev/null || echo latest)
|
||||
echo "PREV_TAG=$PREV_TAG"
|
||||
|
||||
# Deploy THIS commit's images, not :latest. Overlapping runs are
|
||||
# normal here, and with :latest whichever finishes last wins for
|
||||
# both -- run 544 asserted its own SHA and found run 545's build
|
||||
@@ -120,8 +138,17 @@ jobs:
|
||||
$COMPOSE up -d --remove-orphans
|
||||
docker image prune -f
|
||||
ENDSSH
|
||||
)
|
||||
echo "$OUTPUT"
|
||||
PREV_TAG=$(printf '%s\n' "$OUTPUT" | grep '^PREV_TAG=' | tail -n1 | cut -d'=' -f2)
|
||||
if [ -z "$PREV_TAG" ]; then
|
||||
echo "Could not determine the previous tag from deploy output - rollback target unknown."
|
||||
exit 1
|
||||
fi
|
||||
echo "prev_sha=$PREV_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Health check
|
||||
id: health
|
||||
run: |
|
||||
# Poll rather than sleep-once: the container has to finish starting,
|
||||
# and a fixed sleep is either too short (flaky red) or wastes time on
|
||||
@@ -149,6 +176,78 @@ jobs:
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# server-26#65: only now -- confirmed by /health, not by "up -d
|
||||
# returned 0" -- record this as the rollback target for the NEXT
|
||||
# deploy. A failure to write this is a bookkeeping problem, not a
|
||||
# deploy problem, so it warns instead of failing the job (a hard
|
||||
# failure here would trigger the Rollback step below against a
|
||||
# perfectly good deploy).
|
||||
ssh -o StrictHostKeyChecking=no \
|
||||
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
|
||||
-o ConnectTimeout=15 \
|
||||
-i /tmp/deploy_key \
|
||||
drb@${{ secrets.SERVER_IP }} \
|
||||
"echo '${{ gitea.sha }}' > /opt/drb/.last_good_tag" \
|
||||
|| echo "warning: failed to persist .last_good_tag - next deploy's rollback target may be stale"
|
||||
|
||||
- name: Rollback on failed health check
|
||||
id: rollback
|
||||
if: failure()
|
||||
run: |
|
||||
# server-26#65 decision 3 / board minutes #62: up -d used to be the
|
||||
# last word -- a build that passes tests, returns 200, and still
|
||||
# corrupts incidents on live traffic would stay live for 12+ hours
|
||||
# before a human noticed. This step is what makes that impossible:
|
||||
# any failure above (pull, restart, or the health/SHA check) lands
|
||||
# here and puts the previously-verified tag back.
|
||||
PREV_TAG="${{ steps.deploy.outputs.prev_sha }}"
|
||||
if [ -z "$PREV_TAG" ]; then
|
||||
echo "No previous tag was captured (Deploy step itself failed before recording one) - cannot roll back automatically."
|
||||
echo "status=skipped" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "Rolling back to $PREV_TAG"
|
||||
|
||||
ssh -o StrictHostKeyChecking=no \
|
||||
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
|
||||
-o ConnectTimeout=15 \
|
||||
-i /tmp/deploy_key \
|
||||
drb@${{ secrets.SERVER_IP }} << ENDSSH
|
||||
set -e
|
||||
cd /opt/drb
|
||||
export TAG=$PREV_TAG
|
||||
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
|
||||
if ! \$COMPOSE pull; then
|
||||
echo "rollback image pull failed - pruning and retrying once"
|
||||
docker image prune -af
|
||||
\$COMPOSE pull
|
||||
fi
|
||||
\$COMPOSE up -d --remove-orphans
|
||||
ENDSSH
|
||||
|
||||
# Re-verify exactly like the forward health check does: liveness
|
||||
# alone doesn't prove the rollback took, the SHA has to match the
|
||||
# tag we just switched back to.
|
||||
BODY=""
|
||||
for _ in $(seq 1 12); do
|
||||
sleep 5
|
||||
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || continue
|
||||
case "$BODY" in *"$PREV_TAG"*) break ;; esac
|
||||
done
|
||||
|
||||
RUNNING=$(printf '%s' "$BODY" | tr ',' '\n' | grep git_sha | cut -d'"' -f4)
|
||||
if [ "$RUNNING" != "$PREV_TAG" ]; then
|
||||
echo "ROLLBACK FAILED: expected git_sha '$PREV_TAG', got '$RUNNING'."
|
||||
echo "Production state is UNKNOWN - check the VM by hand immediately."
|
||||
echo "status=failed" >> "$GITHUB_OUTPUT"
|
||||
echo "rolled_back_to=$PREV_TAG" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Rolled back successfully to $PREV_TAG"
|
||||
echo "status=success" >> "$GITHUB_OUTPUT"
|
||||
echo "rolled_back_to=$PREV_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
notify-failure:
|
||||
name: Report a failed deploy
|
||||
needs: [build, deploy]
|
||||
@@ -166,6 +265,8 @@ jobs:
|
||||
WEBHOOK: ${{ secrets.DEPLOY_ALERT_WEBHOOK }}
|
||||
RUN_URL: ${{ gitea.server_url }}/${{ gitea.repository }}/actions/runs/${{ gitea.run_number }}
|
||||
SHA: ${{ gitea.sha }}
|
||||
ROLLBACK_STATUS: ${{ needs.deploy.outputs.rollback_status }}
|
||||
ROLLBACK_SHA: ${{ needs.deploy.outputs.rollback_sha }}
|
||||
run: |
|
||||
if [ -z "$WEBHOOK" ]; then
|
||||
echo "DEPLOY_ALERT_WEBHOOK is not set - skipping notification."
|
||||
@@ -173,9 +274,30 @@ jobs:
|
||||
fi
|
||||
python3 - <<'PY' > /tmp/payload.json
|
||||
import json, os
|
||||
sha = os.environ["SHA"][:8]
|
||||
run_url = os.environ["RUN_URL"]
|
||||
status = os.environ.get("ROLLBACK_STATUS", "")
|
||||
rollback_sha = os.environ.get("ROLLBACK_SHA", "")
|
||||
|
||||
# server-26#65: the old text here unconditionally claimed
|
||||
# "production is still running the previous build" -- true only
|
||||
# when the pull/restart itself failed. It's false the moment a
|
||||
# build passes the SHA check but has a live logic bug (exactly the
|
||||
# class of bug the correlator instrumentation exists to catch), or
|
||||
# once the deploy job's own rollback path has run. Say what
|
||||
# actually happened instead.
|
||||
if status == "success":
|
||||
detail = "Automatic rollback to `%s` succeeded. Production is back on the previous good build." % rollback_sha[:8]
|
||||
elif status == "failed":
|
||||
detail = ("Automatic rollback to `%s` FAILED. Production state is UNKNOWN -- "
|
||||
"check the VM by hand immediately.") % rollback_sha[:8]
|
||||
elif status == "skipped":
|
||||
detail = "No rollback was attempted (no previous tag captured, or build/push failed before any deploy). Check the VM by hand."
|
||||
else:
|
||||
detail = "Build failed before any deploy was attempted. Production is unchanged."
|
||||
|
||||
print(json.dumps({"content":
|
||||
"**DRB deploy failed** on `%s`\n%s\nProduction is still running the previous build."
|
||||
% (os.environ["SHA"][:8], os.environ["RUN_URL"])}))
|
||||
"**DRB deploy failed** on `%s`\n%s\n%s" % (sha, run_url, detail)}))
|
||||
PY
|
||||
curl -sS -X POST -H "Content-Type: application/json" \
|
||||
--data @/tmp/payload.json "$WEBHOOK" || echo "notification POST failed"
|
||||
|
||||
Reference in New Issue
Block a user