Two unrelated-looking problems with the same shape: a dangerous state that
looked fine from the outside.
DEPLOY (server-26#21). The Deploy job failed on fifteen consecutive pushes
between 2026-08-18 and 08-20 and nobody noticed for two days, because the
build job was green and a red run is only visible to someone who opens Gitea.
Production served 08-18 code the whole time -- including the entire frontend
redesign, chunks 2 through 8. Three changes:
* The health check now asserts WHICH build answered, not just that something
did. CI bakes the commit into the image (Dockerfile ARG/ENV GIT_SHA) and
/health reports it, so a deploy that "succeeds" while the previous
container keeps running now fails. Liveness alone could never have caught
this.
* The image pull retries once after a prune. The actual failure was
containerd unable to extract a layer -- "failed to Lchown ... no such file
or directory" -- a corrupted entry in the snapshot store, which a prune
clears. A second failure after pruning is a real problem (check the VM's
disk) and still stops the deploy.
* A notify-failure job POSTs to DEPLOY_ALERT_WEBHOOK when anything in the
workflow fails. Unset means skip quietly, not fail.
CORS (server-26#20). allow_origins=["*"] with allow_credentials=True is not
the permissive-but-harmless setting it reads as. Starlette does not reject the
pair -- it reflects the caller's Origin back and still sends
Access-Control-Allow-Credentials: true, so the effective policy is "any
origin, WITH credentials", the opposite of what a wildcard normally means.
Rather than trust every deployment to remember CORS_ORIGINS, the pair is now
unrepresentable: a wildcard forces allow_credentials off and logs an ERROR
naming the variable to set. Correctly configured deployments that name their
origins are unaffected and keep credentialed requests.
Severity honestly: low today. c2-core is bearer-auth, and browsers do not
attach bearer tokens cross-origin the way they attach cookies. This is a
misconfiguration waiting for the day something starts trusting a cookie.
Also adds firebase_admin.auth.UserRecord and the list/update/create/delete_user
names to the conftest stub. routers/users.py annotates with UserRecord at
import time, so without it importing app.main failed at collection -- which is
why nothing had ever tested anything wired at app level, CORS included.
Tests: 5 new in test_cors_policy.py, covering the pure policy function, the
middleware actually mounted on the app (so re-hardcoding allow_credentials=True
fails here), and the presence of the build stamp.
Closes logan/server-26#20
Closes logan/server-26#21
163 lines
5.9 KiB
YAML
163 lines
5.9 KiB
YAML
name: Build & Deploy
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
|
|
env:
|
|
# REGISTRY secret = "git.vpn.cusano.net/logan" (full image prefix)
|
|
REGISTRY: ${{ secrets.REGISTRY }}
|
|
|
|
jobs:
|
|
build:
|
|
name: Build & push images
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@v3
|
|
|
|
- name: Log in to Gitea registry
|
|
uses: docker/login-action@v3
|
|
with:
|
|
registry: git.vpn.cusano.net
|
|
username: ${{ secrets.REGISTRY_USER }}
|
|
password: ${{ secrets.BUILD_TOKEN }}
|
|
|
|
- name: Build & push c2-core
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: ./drb-c2-core
|
|
push: true
|
|
build-args: |
|
|
GIT_SHA=${{ gitea.sha }}
|
|
tags: |
|
|
${{ env.REGISTRY }}/c2-core:latest
|
|
${{ env.REGISTRY }}/c2-core:${{ gitea.sha }}
|
|
|
|
- name: Build & push discord-bot
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: ./drb-server-discord-bot
|
|
push: true
|
|
tags: |
|
|
${{ env.REGISTRY }}/discord-bot:latest
|
|
${{ env.REGISTRY }}/discord-bot:${{ gitea.sha }}
|
|
|
|
- name: Build & push frontend
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: ./drb-frontend
|
|
push: true
|
|
tags: |
|
|
${{ env.REGISTRY }}/frontend:latest
|
|
${{ env.REGISTRY }}/frontend:${{ gitea.sha }}
|
|
build-args: |
|
|
NEXT_PUBLIC_C2_URL=https://api.${{ secrets.DRB_DOMAIN }}
|
|
NEXT_PUBLIC_FIREBASE_API_KEY=${{ secrets.FIREBASE_API_KEY }}
|
|
NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=${{ secrets.FIREBASE_AUTH_DOMAIN }}
|
|
NEXT_PUBLIC_FIREBASE_PROJECT_ID=${{ secrets.FIREBASE_PROJECT_ID }}
|
|
NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET=${{ secrets.FIREBASE_STORAGE_BUCKET }}
|
|
NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=${{ secrets.FIREBASE_MESSAGING_SENDER_ID }}
|
|
NEXT_PUBLIC_FIREBASE_APP_ID=${{ secrets.FIREBASE_APP_ID }}
|
|
NEXT_PUBLIC_FIRESTORE_DATABASE=${{ secrets.FIRESTORE_DATABASE }}
|
|
|
|
deploy:
|
|
name: Deploy to VM
|
|
needs: build
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Check runner outbound IP
|
|
run: curl -s ifconfig.me
|
|
|
|
- name: Write SSH key
|
|
run: |
|
|
printf '%s\n' "${{ secrets.SSH_PRIVATE_KEY }}" > /tmp/deploy_key
|
|
chmod 600 /tmp/deploy_key
|
|
ssh-keygen -l -f /tmp/deploy_key
|
|
|
|
- name: Deploy
|
|
run: |
|
|
ssh -o StrictHostKeyChecking=no \
|
|
-o HostKeyAlgorithms=ssh-ed25519,rsa-sha2-256,rsa-sha2-512 \
|
|
-o ConnectTimeout=15 \
|
|
-v \
|
|
-i /tmp/deploy_key \
|
|
drb@${{ secrets.SERVER_IP }} << 'ENDSSH'
|
|
set -e
|
|
cd /opt/drb
|
|
|
|
# Update compose files + mosquitto config
|
|
git pull origin main
|
|
|
|
# Pull pre-built images and restart (no build on the VM).
|
|
#
|
|
# The retry is not defensive padding: this exact step failed fifteen
|
|
# deploys in a row (2026-08-18 to 08-20) with containerd unable to
|
|
# extract a layer -- "failed to Lchown ... no such file or directory"
|
|
# -- a corrupted entry in the snapshot store. Pruning clears the bad
|
|
# layer and the second pull succeeds. If it fails again after a
|
|
# prune that is a real problem (check the VM's disk) and should stop
|
|
# the deploy rather than be retried forever.
|
|
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.prod.yml"
|
|
if ! $COMPOSE pull; then
|
|
echo "image pull failed - pruning and retrying once"
|
|
docker image prune -af
|
|
$COMPOSE pull
|
|
fi
|
|
$COMPOSE up -d --remove-orphans
|
|
docker image prune -f
|
|
ENDSSH
|
|
|
|
- name: Health check
|
|
run: |
|
|
sleep 20
|
|
BODY=$(curl -fsS https://api.${{ secrets.DRB_DOMAIN }}/health) || {
|
|
echo "Health check failed: /health did not respond"; exit 1; }
|
|
echo "$BODY"
|
|
|
|
# Liveness alone is not enough. A deploy can report success while the
|
|
# PREVIOUS container keeps serving -- that is how production ran
|
|
# 08-18 code for two days without a single red run. Assert that the
|
|
# build which answered is the commit we just pushed.
|
|
RUNNING=$(printf '%s' "$BODY" | tr ',' '\n' | grep git_sha | cut -d'"' -f4)
|
|
if [ "$RUNNING" != "${{ gitea.sha }}" ]; then
|
|
echo "Deployed build is '$RUNNING', expected '${{ gitea.sha }}'."
|
|
echo "The container was not actually replaced."
|
|
exit 1
|
|
fi
|
|
|
|
notify-failure:
|
|
name: Report a failed deploy
|
|
needs: [build, deploy]
|
|
if: failure()
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Post to Discord
|
|
# A red run in Gitea is only visible to someone who opens Gitea, and
|
|
# nobody did for two days. Same shape as an AI tier dying quietly,
|
|
# which is why both now push a message out of the box instead of
|
|
# waiting to be discovered. No webhook configured => skip quietly
|
|
# rather than fail, since not every deployment will set one.
|
|
env:
|
|
WEBHOOK: ${{ secrets.DEPLOY_ALERT_WEBHOOK }}
|
|
RUN_URL: ${{ gitea.server_url }}/${{ gitea.repository }}/actions/runs/${{ gitea.run_number }}
|
|
SHA: ${{ gitea.sha }}
|
|
run: |
|
|
if [ -z "$WEBHOOK" ]; then
|
|
echo "DEPLOY_ALERT_WEBHOOK is not set - skipping notification."
|
|
exit 0
|
|
fi
|
|
python3 - <<'PY' > /tmp/payload.json
|
|
import json, os
|
|
print(json.dumps({"content":
|
|
"**DRB deploy failed** on `%s`\n%s\nProduction is still running the previous build."
|
|
% (os.environ["SHA"][:8], os.environ["RUN_URL"])}))
|
|
PY
|
|
curl -sS -X POST -H "Content-Type: application/json" \
|
|
--data @/tmp/payload.json "$WEBHOOK" || echo "notification POST failed"
|