logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

No automated deploy path for Firestore rules — they can regress silently after every fix #51

New Issue
Closed
opened 2026-08-23 18:11:14 -04:00 by logan · 5 comments
logan commented 2026-08-23 18:11:14 -04:00
Owner
Copy Link
Copy Source

Split out of #13. #13 is the one-time operational task (get the current rules live). This issue is the durable gap: even after #13 is done, nothing keeps the deployed ruleset in sync with the repo.

Verified 2026-08-23

  • Server/infra/firestore/firestore.rules, firebase.json and firestore.indexes.json are in source control.
  • .gitea/workflows/deploy.yml has no firebase deploy, firebase-tools, or firestore:rules step. Its only Firestore reference is NEXT_PUBLIC_FIRESTORE_DATABASE as a frontend build-arg (line 65).
  • Grep across every *.md, *.yml, *.yaml, *.sh and Makefile under Server/ returns zero matches for firebase deploy or firestore:rules. No script, no make target, no runbook.
  • The three container images DO auto-deploy on push to main. Firestore rules specifically fall outside that, which makes the gap easy to miss — the reasonable assumption is that everything deploys.

Why this outlives #13

The rules are the only access control on browser direct-reads (useCalls, useIncidents, etc. read Firestore from the client, so auth.py is not in the read path). A security boundary that is edited in the repo but applied by hand will drift again the first time someone changes it and forgets — and the failure is silent in the safe direction for the reviewer (repo looks correct) and unsafe in production.

What done looks like

  • A deploy step that applies firestore.rules (and indexes) from main, ideally in deploy.yml so it shares the existing trigger.
  • Needs a service account with Firebase Rules Admin, added to Gitea secrets.
  • Should FAIL the deploy if the rules fail to apply — a silently-skipped rules deploy is the current state and is worse than a red build.
  • Note #33: firestore.indexes.json has already drifted from the live database and duplicates two indexes. Deploying indexes blindly may error; sequence this after #33 or scope this issue to rules only.

Links

  • #13 — deploy the current rules (operational, do first)
  • #33 — indexes already drifted
  • Board minutes #42 — Gate B blocks charging anyone while the prod tenant boundary is unverified
Split out of #13. **#13 is the one-time operational task** (get the current rules live). This issue is the durable gap: even after #13 is done, nothing keeps the deployed ruleset in sync with the repo. ## Verified 2026-08-23 - `Server/infra/firestore/firestore.rules`, `firebase.json` and `firestore.indexes.json` are in source control. - `.gitea/workflows/deploy.yml` has **no** `firebase deploy`, `firebase-tools`, or `firestore:rules` step. Its only Firestore reference is `NEXT_PUBLIC_FIRESTORE_DATABASE` as a frontend build-arg (line 65). - Grep across every `*.md`, `*.yml`, `*.yaml`, `*.sh` and Makefile under `Server/` returns **zero** matches for `firebase deploy` or `firestore:rules`. No script, no make target, no runbook. - The three container images DO auto-deploy on push to `main`. Firestore rules specifically fall outside that, which makes the gap easy to miss — the reasonable assumption is that everything deploys. ## Why this outlives #13 The rules are the **only** access control on browser direct-reads (`useCalls`, `useIncidents`, etc. read Firestore from the client, so `auth.py` is not in the read path). A security boundary that is edited in the repo but applied by hand will drift again the first time someone changes it and forgets — and the failure is silent in the safe direction for the reviewer (repo looks correct) and unsafe in production. ## What done looks like - A deploy step that applies `firestore.rules` (and indexes) from `main`, ideally in `deploy.yml` so it shares the existing trigger. - Needs a service account with Firebase Rules Admin, added to Gitea secrets. - Should FAIL the deploy if the rules fail to apply — a silently-skipped rules deploy is the current state and is worse than a red build. - Note #33: `firestore.indexes.json` has already drifted from the live database and duplicates two indexes. Deploying indexes blindly may error; sequence this after #33 or scope this issue to rules only. ## Links - #13 — deploy the current rules (operational, do first) - #33 — indexes already drifted - Board minutes #42 — Gate B blocks charging anyone while the prod tenant boundary is unverified
logan commented 2026-09-07 18:54:57 -04:00
Author
Owner
Copy Link
Copy Source

The concrete casualty: useUnacknowledgedAlerts throws requires an index on every page

drb-frontend/lib/useAlerts.ts useUnacknowledgedAlerts() runs where(org_id ==) where(acknowledged == false) orderBy(triggered_at desc) limit(100) on alert_events. No matching composite index is live, so it fails with FirebaseError: The query requires an index on every page load, and the /watch "Triggered Alerts" tab is dead. This is #13/#51's manual-deploy gap biting a user-visible surface.

File change (branch fix/51-alert-events-index, commit 8a0412b, off main)

Added one entry to infra/firestore/firestore.indexes.json:

{
  "collectionGroup": "alert_events",
  "queryScope": "COLLECTION",
  "fields": [
    { "fieldPath": "acknowledged", "order": "ASCENDING" },
    { "fieldPath": "org_id", "order": "ASCENDING" },
    { "fieldPath": "triggered_at", "order": "DESCENDING" }
  ]
}

Field tuple + triggered_at DESCENDING are copied verbatim from the console create_composite link in the live error, so a gcloud/console create and a firebase deploy converge on the same index (the ASC-vs-DESC split is exactly what caused #33).

Two caveats for whoever reconciles this — do NOT treat as blockers here, they belong to #33:

  1. Scope is COLLECTION, not COLLECTION_GROUP. The hook uses collection(db, "alert_events"), and the error's create_composite payload decodes to queryScope = 1 (COLLECTION). The collectionGroups/ segment in that URL is Firestore's path naming for both scopes, not the scope itself.
  2. The file already carried a near-miss entry commented "the nav badge" (COLLECTION, fields org_id, acknowledged, triggered_at all ASC) that was never deployed. The new entry matches Firestore's own generated spec for the failing query; the old one is now redundant/overlapping. Reconciling the two (and dropping the pre-tenancy live alert_events(acknowledged, triggered_at) index) is #33's job, not this change's.

Not deployed — no GCP creds on the authoring machine, which is the point of this issue.

Deploy options (any one)

1. gcloud (headless over SSH from the prod VM, which is authed as the project SA):

gcloud firestore indexes composite create \
  --project=discord-radio-bot-461301 \
  --database=c2-server \
  --collection-group=alert_events \
  --query-scope=collection \
  --field-config=field-path=acknowledged,order=ascending \
  --field-config=field-path=org_id,order=ascending \
  --field-config=field-path=triggered_at,order=descending

(--query-scope values are lowercase-hyphenated: collection / collection-group / collection-recursive; collection is also the default. Verified against the current gcloud reference.)

2. firebase CLI — from Server/infra/firestore/ (firebase.json pins database: c2-server):

firebase deploy --only firestore:indexes --project discord-radio-bot-461301

This will also prompt to delete the three indexes the file's //drift-2026-08-23 note lists — that note still says answer YES to those three only.

3. Console link from the error:

https://console.firebase.google.com/v1/r/project/discord-radio-bot-461301/firestore/databases/c2-server/indexes?create_composite=Cl1wcm9qZWN0cy9kaXNjb3JkLXJhZGlvLWJvdC00NjEzMDEvZGF0YWJhc2VzL2MyLXNlcnZlci9jb2xsZWN0aW9uR3JvdXBzL2FsZXJ0X2V2ZW50cy9pbmRleGVzL18QARoQCgxhY2tub3dsZWRnZWQQARoKCgZvcmdfaWQQARoQCgx0cmlnZ2VyZWRfYXQQAhoMCghfX25hbWVfXxAC

Index build is a few minutes on an empty/small alert_events; the hook and the Triggered Alerts tab recover on their own once it's READY.

## The concrete casualty: `useUnacknowledgedAlerts` throws `requires an index` on every page `drb-frontend/lib/useAlerts.ts` `useUnacknowledgedAlerts()` runs `where(org_id ==) where(acknowledged == false) orderBy(triggered_at desc) limit(100)` on `alert_events`. No matching composite index is live, so it fails with `FirebaseError: The query requires an index` on every page load, and the `/watch` "Triggered Alerts" tab is dead. This is #13/#51's manual-deploy gap biting a user-visible surface. ### File change (branch `fix/51-alert-events-index`, commit `8a0412b`, off `main`) Added one entry to `infra/firestore/firestore.indexes.json`: ```json { "collectionGroup": "alert_events", "queryScope": "COLLECTION", "fields": [ { "fieldPath": "acknowledged", "order": "ASCENDING" }, { "fieldPath": "org_id", "order": "ASCENDING" }, { "fieldPath": "triggered_at", "order": "DESCENDING" } ] } ``` Field tuple + `triggered_at DESCENDING` are copied verbatim from the console `create_composite` link in the live error, so a `gcloud`/console create and a `firebase deploy` converge on the same index (the ASC-vs-DESC split is exactly what caused #33). **Two caveats for whoever reconciles this — do NOT treat as blockers here, they belong to #33:** 1. **Scope is `COLLECTION`, not `COLLECTION_GROUP`.** The hook uses `collection(db, "alert_events")`, and the error's `create_composite` payload decodes to `queryScope = 1` (COLLECTION). The `collectionGroups/` segment in that URL is Firestore's path naming for both scopes, not the scope itself. 2. The file already carried a near-miss entry commented "the nav badge" (`COLLECTION`, fields `org_id, acknowledged, triggered_at` all ASC) that was never deployed. The new entry matches Firestore's own generated spec for the failing query; the old one is now redundant/overlapping. Reconciling the two (and dropping the pre-tenancy live `alert_events(acknowledged, triggered_at)` index) is #33's job, not this change's. Not deployed — no GCP creds on the authoring machine, which is the point of this issue. ### Deploy options (any one) **1. gcloud (headless over SSH from the prod VM, which is authed as the project SA):** ``` gcloud firestore indexes composite create \ --project=discord-radio-bot-461301 \ --database=c2-server \ --collection-group=alert_events \ --query-scope=collection \ --field-config=field-path=acknowledged,order=ascending \ --field-config=field-path=org_id,order=ascending \ --field-config=field-path=triggered_at,order=descending ``` (`--query-scope` values are lowercase-hyphenated: `collection` / `collection-group` / `collection-recursive`; `collection` is also the default. Verified against the current gcloud reference.) **2. firebase CLI** — from `Server/infra/firestore/` (`firebase.json` pins `database: c2-server`): ``` firebase deploy --only firestore:indexes --project discord-radio-bot-461301 ``` This will also prompt to delete the three indexes the file's `//drift-2026-08-23` note lists — that note still says answer YES to those three only. **3. Console link from the error:** https://console.firebase.google.com/v1/r/project/discord-radio-bot-461301/firestore/databases/c2-server/indexes?create_composite=Cl1wcm9qZWN0cy9kaXNjb3JkLXJhZGlvLWJvdC00NjEzMDEvZGF0YWJhc2VzL2MyLXNlcnZlci9jb2xsZWN0aW9uR3JvdXBzL2FsZXJ0X2V2ZW50cy9pbmRleGVzL18QARoQCgxhY2tub3dsZWRnZWQQARoKCgZvcmdfaWQQARoQCgx0cmlnZ2VyZWRfYXQQAhoMCghfX25hbWVfXxAC Index build is a few minutes on an empty/small `alert_events`; the hook and the Triggered Alerts tab recover on their own once it's `READY`.
logan commented 2026-09-07 19:36:55 -04:00
Author
Owner
Copy Link
Copy Source

Index still not deployed — this is now the top UI blocker. PR #121 (merged, bc3251e8) declares alert_events (acknowledged ASC, org_id ASC, triggered_at DESC) in infra/firestore/firestore.indexes.json, but there is still no deploy automation (the core of this issue), so the live c2-server DB does not have it. /watch Triggered Alerts + the site-wide useUnacknowledgedAlerts nav badge both stay dead until it is applied.

VM action (GCP service-account authed host): from the repo checkout, Server/infra/firestore/:
firebase deploy --only firestore:indexes --project discord-radio-bot-461301
Answer YES to the 3 deletion prompts per the //drift-2026-08-23 note. This also reconciles the whole index set, which may fix the Archive 503 (#33) if that is a missing calls(org_id, started_at DESC).

**Index still not deployed — this is now the top UI blocker.** PR #121 (merged, bc3251e8) declares `alert_events (acknowledged ASC, org_id ASC, triggered_at DESC)` in `infra/firestore/firestore.indexes.json`, but there is still no deploy automation (the core of this issue), so the live `c2-server` DB does not have it. `/watch` Triggered Alerts + the site-wide `useUnacknowledgedAlerts` nav badge both stay dead until it is applied. **VM action (GCP service-account authed host):** from the repo checkout, `Server/infra/firestore/`: `firebase deploy --only firestore:indexes --project discord-radio-bot-461301` Answer YES to the 3 deletion prompts per the `//drift-2026-08-23` note. This also reconciles the whole index set, which may fix the Archive 503 (#33) if that is a missing `calls(org_id, started_at DESC)`.
logan closed this issue 2026-09-07 23:22:39 -04:00
logan commented 2026-09-07 23:49:09 -04:00
Author
Owner
Copy Link
Copy Source

Resolved. Root cause was NOT the alert_events index alone — the whole firestore.indexes.json declared ASC-only indexes for orderBy(x,'desc') queries, which Firestore does not reverse-scan for these shapes. Created alert_events (org_id, triggered_at DESC) + (acknowledged, org_id, triggered_at DESC) + calls (org_id, started_at DESC) on c2-server via gcloud (2026-09-08). PR #124 (merged, deploy 7f4d684) rewrites the file to DESC and adds a CI firebase deploy --only firestore:rules,firestore:indexes step so it can't regress silently. /watch Triggered Alerts + the nav alert badge both load now. VM prereq for the CI step: npm i -g firebase-tools once.

Resolved. Root cause was NOT the alert_events index alone — the whole `firestore.indexes.json` declared ASC-only indexes for `orderBy(x,'desc')` queries, which Firestore does not reverse-scan for these shapes. Created `alert_events (org_id, triggered_at DESC)` + `(acknowledged, org_id, triggered_at DESC)` + `calls (org_id, started_at DESC)` on `c2-server` via gcloud (2026-09-08). PR #124 (merged, deploy 7f4d684) rewrites the file to DESC and adds a CI `firebase deploy --only firestore:rules,firestore:indexes` step so it can't regress silently. /watch Triggered Alerts + the nav alert badge both load now. VM prereq for the CI step: `npm i -g firebase-tools` once.
logan reopened this issue 2026-09-13 17:38:41 -04:00
logan commented 2026-09-13 17:38:42 -04:00
Author
Owner
Copy Link
Copy Source

Reopened per board minutes 2026-09-13 CIO draft (server-26#144): PR #124's "Closes #N" auto-closed this, but the CI firebase deploy step (.gitea/workflows/deploy.yml:131-137) gates on command -v firebase and the VM has no firebase-tools installed -- the step silently no-ops on every deploy. Firestore rules/indexes are NOT actually being applied in prod. See DEFERRED.md row 54. Fix: install firebase-tools on the VM, confirm next deploy log shows a real firebase deploy run, not the WARNING line.

Reopened per board minutes 2026-09-13 CIO draft (server-26#144): PR #124's "Closes #N" auto-closed this, but the CI firebase deploy step (`.gitea/workflows/deploy.yml:131-137`) gates on `command -v firebase` and the VM has no firebase-tools installed -- the step silently no-ops on every deploy. Firestore rules/indexes are NOT actually being applied in prod. See DEFERRED.md row 54. Fix: install firebase-tools on the VM, confirm next deploy log shows a real `firebase deploy` run, not the WARNING line.
logan commented 2026-09-27 13:07:13 -04:00
Author
Owner
Copy Link
Copy Source

Fixed in 9b83f0e: rules job now authenticates with the drb-ci-firestore-deploy service account (FIREBASE_SA_KEY). Run 633 — all jobs green, rules deployed to c2-server (aircraft/vessels reads confirmed working in prod).

Fixed in 9b83f0e: rules job now authenticates with the drb-ci-firestore-deploy service account (FIREBASE_SA_KEY). Run 633 — all jobs green, rules deployed to c2-server (aircraft/vessels reads confirmed working in prod).
logan closed this issue 2026-09-27 13:07:13 -04:00
logan added the area/autharea/infra-deploy
type
chore
 labels 2026-10-05 02:04:29 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
 minutes:draft
One exec pre-read, closed by CEO
 minutes:final
CEO ruling; open until enacted
 role:ceo
Filed by the CEO
 role:cfo
Filed by the CFO
 role:cio
Filed by the CIO
 role:ciso
Filed by the CISO
 role:cmo
Filed by the CMO
 role:coo
Filed by the COO
 role:cto
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/auth area/infra-deploy
type
chore
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#51