logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

Close the /admin/features Firestore SSH side-door: require_service_key_or_admin + distinct agent credential #64

New Issue
Open
opened 2026-08-24 23:41:26 -04:00 by logan · 3 comments
logan commented 2026-08-24 23:41:26 -04:00
Owner
Copy Link
Copy Source

Decision 2, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31. CTO draft #60 finding 1 and CISO draft #61 finding 3 reached this independently, from different directions.

GET and PUT /admin/features (drb-c2-core/app/routers/admin.py:27-41) use require_admin_token, which accepts only a Firebase admin token. There is no service-key path, so the unattended runbook SSHes into the c2-core container and writes config/ai_features straight to Firestore with the admin SDK (.claude/scheduled/drb-worksession.md:117-130) to stay headless. That side-door needs a full container shell to flip a global AI-cost switch, and feature_flags.set_flags() writes no audit_log entry either way.

Work

  1. Swap the dependency on both routes to require_service_key_or_admin (drb-c2-core/app/internal/auth.py:201-219) - already the established bot-vs-admin pattern, already wired on routers/nodes.py:95,157,243,261 and routers/trips.py:298-463. Two lines.
  2. Mint the agent a distinct service credential. Do not reuse the Discord bot shared SERVICE_KEY - that collapses two principals into one unattributable identity in every log line.
  3. Add an audit_log write in feature_flags.set_flags() so a flag flip is attributable at all.
  4. Fold the two-location AI-on step into one helper that always sets config/ai_features and both systems ai_flags (CTO #60 rec. 2). OFF already cannot half-apply; ON can, and one missed manual step leaves a radio system hot after shutoff.
  5. Retire the SSH and Firestore-direct write from drb-worksession.md sections 5b and 5d; call the HTTPS route instead.
  6. Once this lands, remove ssh drb from the AI-flag path in the runbook (#62 Decision 1c).

Refs #62, #60, #61.

**Decision 2, board minutes #62 (2026-08-24). Owner: CTO. Due 2026-08-31.** CTO draft #60 finding 1 and CISO draft #61 finding 3 reached this independently, from different directions. `GET` and `PUT /admin/features` (`drb-c2-core/app/routers/admin.py:27-41`) use `require_admin_token`, which accepts only a Firebase admin token. There is no service-key path, so the unattended runbook SSHes into the `c2-core` container and writes `config/ai_features` straight to Firestore with the admin SDK (`.claude/scheduled/drb-worksession.md:117-130`) to stay headless. That side-door needs a full container shell to flip a global AI-cost switch, and `feature_flags.set_flags()` writes no `audit_log` entry either way. ## Work 1. Swap the dependency on both routes to `require_service_key_or_admin` (`drb-c2-core/app/internal/auth.py:201-219`) - already the established bot-vs-admin pattern, already wired on `routers/nodes.py:95,157,243,261` and `routers/trips.py:298-463`. Two lines. 2. Mint the agent a **distinct** service credential. Do **not** reuse the Discord bot shared `SERVICE_KEY` - that collapses two principals into one unattributable identity in every log line. 3. Add an `audit_log` write in `feature_flags.set_flags()` so a flag flip is attributable at all. 4. Fold the two-location AI-on step into one helper that always sets `config/ai_features` **and** both systems `ai_flags` (CTO #60 rec. 2). OFF already cannot half-apply; ON can, and one missed manual step leaves a radio system hot after shutoff. 5. Retire the SSH and Firestore-direct write from `drb-worksession.md` sections 5b and 5d; call the HTTPS route instead. 6. Once this lands, remove `ssh drb` from the AI-flag path in the runbook (#62 Decision 1c). Refs #62, #60, #61.
logan commented 2026-08-30 02:52:32 -04:00
Author
Owner
Copy Link
Copy Source

Items 1, 3 and 4 implemented and pushed — 865b5b4, unattended run 2026-08-30, one day inside the 2026-08-31 due date. This issue stays OPEN: items 2, 5 and 6 are not finished.

Tests 273 → 289 (16 new). The suite is green.

What landed

  • A distinct agent credential. New agent_service_key (env AGENT_SERVICE_KEY), separate from the Discord bot's service_key. New dependency require_agent_key_or_admin accepts the agent key or a Firebase admin, and rejects the Discord key — sharing one key was the failure this issue names, and the bot has no business flipping AI flags regardless. Both /admin/features routes moved to it; every other route in admin.py is untouched.
  • A bypass avoided. The settings.agent_service_key and ... guard is load-bearing rather than stylistic: secrets.compare_digest("", "") returns True, so any compare_digest(token, settings.agent_service_key or "") shape would turn a deployment that never configured the key into one that accepts an empty credential. Tested explicitly.
  • Flag flips are now attributable. feature_flags.set_flags() writes an audit_log entry carrying before/after values and the actor, with the agent principal distinguishable from a human admin. The audit write is wrapped so its failure can neither lose the flag write nor 500 the route.
  • One helper, both levels. The cascade sets the global config/ai_features doc and every system document that actually carries an ai_flags map. It scans for the override rather than hardcoding the two known system IDs, so a newly added system cannot silently defeat it.

The cascade default is False, deliberately

A human-facing editor for per-system flags does exist — PUT /systems/{id}/ai-flags (routers/systems.py:108) and the AiFlagsPanel toggle in the frontend systems page. A per-system override is therefore visible operator intent, and cascading by default would silently erase it on any unrelated global flip. The runbook opts in with ?cascade=true, which is where the "one missed manual step leaves a radio system hot" hazard actually lives.

Item 2 — NEEDS OWNER, ~5 minutes

The code path exists; the credential value does not. Until it is set, AGENT_SERVICE_KEY is unset and the route accepts admins only — safe, and unchanged from today's behaviour.

  1. Generate a value: openssl rand -hex 32
  2. Put it in infra/ansible/vault.yml as vault_agent_service_key — a new value, not a copy of vault_service_key, or this issue's whole point is lost
  3. Run the ansible deploy role to re-template /opt/drb/drb-c2-core/.env and restart c2-core

Editing that .env by hand works but is overwritten on the next ansible run. No environment: block was added to compose on purpose: c2-core is env_file-only, and a compose environment: entry outranks env_file, so adding one would blank out a correctly-pasted value.

Items 5 and 6 — deliberately NOT done, and drb-worksession.md is untouched

Retiring the SSH and Firestore-direct write from the runbook is blocked on item 2. The new credential does not exist in production, so the SSH path is still the only one that works; swapping the runbook to HTTPS tonight would have broken the next unattended run's ability to close an AI window — a strictly worse failure than the side-door this issue is closing. They unblock the moment the owner completes item 2, and are the whole of what remains here.

**Items 1, 3 and 4 implemented and pushed — `865b5b4`, unattended run 2026-08-30, one day inside the 2026-08-31 due date. This issue stays OPEN: items 2, 5 and 6 are not finished.** Tests 273 → **289** (16 new). The suite is green. ## What landed - **A distinct agent credential.** New `agent_service_key` (env `AGENT_SERVICE_KEY`), separate from the Discord bot's `service_key`. New dependency `require_agent_key_or_admin` accepts the agent key **or** a Firebase admin, and **rejects the Discord key** — sharing one key was the failure this issue names, and the bot has no business flipping AI flags regardless. Both `/admin/features` routes moved to it; every other route in `admin.py` is untouched. - **A bypass avoided.** The `settings.agent_service_key and ...` guard is load-bearing rather than stylistic: `secrets.compare_digest("", "")` returns **True**, so any `compare_digest(token, settings.agent_service_key or "")` shape would turn a deployment that never configured the key into one that accepts an empty credential. Tested explicitly. - **Flag flips are now attributable.** `feature_flags.set_flags()` writes an `audit_log` entry carrying before/after values and the actor, with the agent principal distinguishable from a human admin. The audit write is wrapped so its failure can neither lose the flag write nor 500 the route. - **One helper, both levels.** The cascade sets the global `config/ai_features` doc and every system document that actually carries an `ai_flags` map. It **scans** for the override rather than hardcoding the two known system IDs, so a newly added system cannot silently defeat it. ## The cascade default is `False`, deliberately A human-facing editor for per-system flags **does** exist — `PUT /systems/{id}/ai-flags` (`routers/systems.py:108`) and the `AiFlagsPanel` toggle in the frontend systems page. A per-system override is therefore visible operator intent, and cascading by default would silently erase it on any unrelated global flip. The runbook opts in with `?cascade=true`, which is where the "one missed manual step leaves a radio system hot" hazard actually lives. ## Item 2 — NEEDS OWNER, ~5 minutes The code path exists; the credential value does not. Until it is set, `AGENT_SERVICE_KEY` is unset and the route accepts admins only — safe, and unchanged from today's behaviour. 1. Generate a value: `openssl rand -hex 32` 2. Put it in `infra/ansible/vault.yml` as `vault_agent_service_key` — **a new value, not a copy of `vault_service_key`**, or this issue's whole point is lost 3. Run the ansible deploy role to re-template `/opt/drb/drb-c2-core/.env` and restart `c2-core` Editing that `.env` by hand works but is overwritten on the next ansible run. No `environment:` block was added to compose on purpose: `c2-core` is `env_file`-only, and a compose `environment:` entry outranks `env_file`, so adding one would blank out a correctly-pasted value. ## Items 5 and 6 — deliberately NOT done, and `drb-worksession.md` is untouched Retiring the SSH and Firestore-direct write from the runbook is blocked on item 2. The new credential does not exist in production, so the SSH path is still the only one that works; swapping the runbook to HTTPS tonight would have broken the next unattended run's ability to close an AI window — a strictly worse failure than the side-door this issue is closing. They unblock the moment the owner completes item 2, and are the whole of what remains here.
logan commented 2026-08-31 02:47:07 -04:00
Author
Owner
Copy Link
Copy Source

Status correction, verified in source and against production, unattended run 2026-08-31 (this issue's due date).

This issue reads as though none of it has shipped. Items 1-4 shipped a day ago in 865b5b4 ("Close the /admin/features side-door that needed a container shell to flip AI spend"), which is an ancestor of the live /health SHA 29c2fb1 and is therefore in production:

  • GET /admin/features (routers/admin.py:27-28) and PUT /admin/features (:43-53) both now depend on require_agent_key_or_admin.
  • A distinct principal exists - require_agent_key_or_admin (internal/auth.py:233) checks settings.agent_service_key, not the Discord bot's SERVICE_KEY. Item 2's "do not collapse two principals" requirement is honoured in the code.
  • feature_flags.set_flags() writes an audit_log entry (internal/feature_flags.py:116-164).

But item 5 must not be enacted, and here is why.

settings.agent_service_key is Optional[str] = None (config.py:157), and the agent-key branch is guarded by settings.agent_service_key and secrets.compare_digest(...) (auth.py:259) - deliberately, so that an unset key cannot be matched by an empty token.

Checked against production tonight: AGENT_SERVICE_KEY is not set. It is absent from /opt/drb/.env, and inside the running c2-core container settings.agent_service_key resolves falsy. (Value not printed; only presence was tested.)

So in production the agent-key branch is dead code. The only way through /admin/features today is a Firebase admin ID token, which an unattended run cannot mint. The SSH/Firestore-direct write is therefore still the only working path for an unattended run to turn the AI window off.

Consequence: retiring the SSH path from drb-worksession.md sections 5b/5d right now would leave the unattended run with no way to close an AI window. An open window bills every minute until something closes it. That is the single worst outcome available in this runbook, and it would have been introduced by "finishing" this issue on schedule.

This is the general failure mode worth naming: the code half of a credential-dependent change shipped, the provisioning half did not, and nothing failed loudly - because the fallback still works. DEFERRED.md already records that CI deploys code only (git pull, compose pull, up -d) and never writes the server's .env, so no deployment step was ever going to set this.

What actually remains on this issue

  1. Provision AGENT_SERVICE_KEY in /opt/drb/.env and restart c2-core. Not a code change; CI cannot do it. Until this is done, items 1-4 are shipped but inert.
  2. Then, and only then, item 5 - retire the SSH/Firestore-direct write from drb-worksession.md 5b/5d, and prove the HTTP path can turn the window off before the SSH path is removed, not after.
  3. Item 4 (the single AI-on helper covering config/ai_features and both systems' ai_flags) should be re-verified against /opt/drb/aiwin/ separately - it is independent of the credential and may already be satisfied.

Ordering is the whole point here: the credential lands first, the runbook changes last. Reversing that order is what turns this issue from a security improvement into an outage that bills by the minute.

**Status correction, verified in source and against production, unattended run 2026-08-31 (this issue's due date).** This issue reads as though none of it has shipped. **Items 1-4 shipped a day ago** in `865b5b4` ("Close the /admin/features side-door that needed a container shell to flip AI spend"), which is an ancestor of the live `/health` SHA `29c2fb1` and is therefore in production: - `GET /admin/features` (`routers/admin.py:27-28`) and `PUT /admin/features` (`:43-53`) both now depend on `require_agent_key_or_admin`. - A **distinct** principal exists - `require_agent_key_or_admin` (`internal/auth.py:233`) checks `settings.agent_service_key`, not the Discord bot's `SERVICE_KEY`. Item 2's "do not collapse two principals" requirement is honoured in the code. - `feature_flags.set_flags()` writes an `audit_log` entry (`internal/feature_flags.py:116-164`). **But item 5 must not be enacted, and here is why.** `settings.agent_service_key` is `Optional[str] = None` (`config.py:157`), and the agent-key branch is guarded by `settings.agent_service_key and secrets.compare_digest(...)` (`auth.py:259`) - deliberately, so that an unset key cannot be matched by an empty token. **Checked against production tonight: `AGENT_SERVICE_KEY` is not set.** It is absent from `/opt/drb/.env`, and inside the running `c2-core` container `settings.agent_service_key` resolves falsy. (Value not printed; only presence was tested.) So in production **the agent-key branch is dead code**. The only way through `/admin/features` today is a Firebase admin ID token, which an unattended run cannot mint. The SSH/Firestore-direct write is therefore still the *only* working path for an unattended run to turn the AI window **off**. **Consequence: retiring the SSH path from `drb-worksession.md` sections 5b/5d right now would leave the unattended run with no way to close an AI window.** An open window bills every minute until something closes it. That is the single worst outcome available in this runbook, and it would have been introduced by "finishing" this issue on schedule. This is the general failure mode worth naming: **the code half of a credential-dependent change shipped, the provisioning half did not, and nothing failed loudly** - because the fallback still works. `DEFERRED.md` already records that CI deploys code only (`git pull`, `compose pull`, `up -d`) and **never writes the server's `.env`**, so no deployment step was ever going to set this. ## What actually remains on this issue 1. **Provision `AGENT_SERVICE_KEY` in `/opt/drb/.env` and restart `c2-core`.** Not a code change; CI cannot do it. Until this is done, items 1-4 are shipped but inert. 2. **Then, and only then**, item 5 - retire the SSH/Firestore-direct write from `drb-worksession.md` 5b/5d, and prove the HTTP path can turn the window off *before* the SSH path is removed, not after. 3. Item 4 (the single AI-on helper covering `config/ai_features` **and** both systems' `ai_flags`) should be re-verified against `/opt/drb/aiwin/` separately - it is independent of the credential and may already be satisfied. **Ordering is the whole point here: the credential lands first, the runbook changes last.** Reversing that order is what turns this issue from a security improvement into an outage that bills by the minute.
logan commented 2026-09-01 02:46:29 -04:00
Author
Owner
Copy Link
Copy Source

Board ruling — FINAL MINUTES #97, 2026-09-01.

  • Items 1, 3, 4: ratified as substantively shipped and deployed (865b5b4, ancestor of the live b722223). Independently verified in source by COO #90, CTO #94 and CISO #93. The board does not re-litigate these again (#97 D2).
  • Items 5 and 6: HELD. The COO's recommendation to retire the SSH/Firestore-direct path from drb-worksession.md 5a/5b/5d tonight is overruled (#97 D1). AGENT_SERVICE_KEY was confirmed unset in production by two officers independently, so the agent-key branch is dead code and the SSH path is still the only working way for an unattended run to close an AI window.
  • The 2026-08-31 due date is recorded MISSED (#97 D3), not extended retroactively.

New dates. Item 2 by 2026-09-02, as task 2 of a single batched ~30-minute owner credential session (#97 D4, alongside #67 and the node-002 stopgap). Items 5/6 within 24h of item 2 being confirmed live, done in one sitting rather than split across runs — CTO.

The security-posture question this issue rests on (agent key over one audited HTTPS route vs. a container shell with Firestore admin credentials) is a unanimous board finding but the ruling is held for the owner — #97 H1. It does not block anything, because the work is held on provisioning either way.

**Board ruling — FINAL MINUTES #97, 2026-09-01.** - **Items 1, 3, 4: ratified as substantively shipped and deployed** (`865b5b4`, ancestor of the live `b722223`). Independently verified in source by COO #90, CTO #94 and CISO #93. The board does not re-litigate these again (#97 D2). - **Items 5 and 6: HELD.** The COO's recommendation to retire the SSH/Firestore-direct path from `drb-worksession.md` 5a/5b/5d tonight is **overruled** (#97 D1). `AGENT_SERVICE_KEY` was confirmed unset in production by two officers independently, so the agent-key branch is dead code and the SSH path is still the only working way for an unattended run to close an AI window. - **The 2026-08-31 due date is recorded MISSED** (#97 D3), not extended retroactively. **New dates.** Item 2 by **2026-09-02**, as task 2 of a single batched ~30-minute owner credential session (#97 D4, alongside #67 and the node-002 stopgap). Items 5/6 within **24h of item 2 being confirmed live**, done in one sitting rather than split across runs — CTO. The security-posture question this issue rests on (agent key over one audited HTTPS route vs. a container shell with Firestore admin credentials) is a **unanimous board finding** but the ruling is held for the owner — #97 H1. It does not block anything, because the work is held on provisioning either way.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
 minutes:draft
One exec pre-read, closed by CEO
 minutes:final
CEO ruling; open until enacted
 role:ceo
Filed by the CEO
 role:cfo
Filed by the CFO
 role:cio
Filed by the CIO
 role:ciso
Filed by the CISO
 role:cmo
Filed by the CMO
 role:coo
Filed by the COO
 role:cto
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/auth area/governance
priority
p2
status
needs-decision
type
security
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#64