logan/node-26
1
0
Fork 0
Code Issues 14 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Node enrolled via install.sh never comes online if approved after install (Toughny) — no client-side key pickup, and re-run doesn't restart edge-node #14

New Issue
Closed
opened 2026-10-04 00:50:57 -04:00 by logan · 4 comments
logan commented 2026-10-04 00:50:57 -04:00
Owner
Copy Link
Copy Source

Symptom (2026-10-04): node "Toughny" was installed + enrolled with install.sh but never shows online in prod.

Top hypothesis (high confidence from source; prod-side not yet confirmed): the node is sitting in a credential deadlock.

  1. install.sh defaults ENROLL_WAIT=0. If the admin hadn't approved Toughny at install time, §5 prints say_pending, writes no configs/credentials.json, and §6 starts the stack anyway.
  2. mqtt_manager._build_client() with no api_key and no legacy MQTT_USER connects anonymously; the dynsec broker (mqtt.drb.cusano.net:8883) refuses it. It retries forever (2–60s backoff) and logs MQTT connect refused.
  3. When the admin later approves, approve_node() upserts dynsec + retain-publishes the key on nodes/{id}/api_key — which the node can never read because it can't authenticate. The edge-node app has no HTTPS pickup of its own (GET /nodes/{id}/credentials lives only in install.sh; mqtt_manager.py TODO(mqtt-cutover) says so).
  4. Even re-running install.sh (the documented recovery) writes credentials.json but then runs docker compose up -d, which is a no-op for an already-running container. credentials.load() runs only in the FastAPI lifespan at startup, and uvicorn --reload watches /app, not /configs — so the running process keeps api_key=None until docker compose restart edge-node.

Ruled out: WireGuard. install.sh intentionally doesn't provision it; nodes use public mqtt.drb.cusano.net:8883 (verified from the dev box 2026-10-04: TCP+TLS OK, LE cert CN=mqtt.drb.cusano.net, verify 0). api.drb.cusano.net/health = ok, mqtt_connected=true, sha fb29900. The WireGuard comments in Server/docker-compose.prod.yml / infra/main.tf are stale.

Not verified: prod admin UI state for Toughny (Chrome got an error page on drb.cusano.net), and broker logs (ssh drb does not resolve from the dev box). Confirm on the node: cd /opt/drb/node-26 && ls -l configs/ && docker compose logs --tail=50 edge-node | grep -iE 'mqtt|api key' — expect no credentials.json and repeated MQTT connect refused.

Owner recovery on Toughny (in order):

  1. Approve Toughny at https://drb.cusano.net/settings/nodes (if not already).
  2. On Toughny: cd /opt/drb/node-26 && sudo bash install.sh (re-run; reuses configs/pickup_secret, collects the key). If it says pickup secret 401/404, re-run with --token DRB-….
  3. test -s configs/credentials.json && echo key-present
  4. sudo docker compose restart edge-node
  5. sudo docker compose logs --tail=30 edge-node | grep -i mqtt → expect MQTT connected.

Proposed fix:

  • install.sh: after writing creds, docker compose restart edge-node if it was already running (or up -d --force-recreate edge-node).
  • edge-node app: own the pickup — persist pickup_secret, and while api_key is absent poll GET {C2_URL}/nodes/{id}/credentials (e.g. every 60s), then save the key and rebuild the MQTT client in place. This is the 'client-side enrollment' follow-up named in node-26#4/#6.
  • install.sh: default --wait-approval to something non-zero when interactive, and make the final banner say plainly that the node stays offline until re-run + restart.

Related: node-26#4, node-26#6.

**Symptom (2026-10-04):** node "Toughny" was installed + enrolled with `install.sh` but never shows online in prod. **Top hypothesis (high confidence from source; prod-side not yet confirmed):** the node is sitting in a credential deadlock. 1. `install.sh` defaults `ENROLL_WAIT=0`. If the admin hadn't approved Toughny at install time, §5 prints `say_pending`, writes **no** `configs/credentials.json`, and §6 starts the stack anyway. 2. `mqtt_manager._build_client()` with no api_key and no legacy MQTT_USER connects anonymously; the dynsec broker (`mqtt.drb.cusano.net:8883`) refuses it. It retries forever (2–60s backoff) and logs `MQTT connect refused`. 3. When the admin later approves, `approve_node()` upserts dynsec + retain-publishes the key on `nodes/{id}/api_key` — which the node can never read because it can't authenticate. The edge-node app has **no** HTTPS pickup of its own (`GET /nodes/{id}/credentials` lives only in install.sh; `mqtt_manager.py` TODO(mqtt-cutover) says so). 4. Even re-running install.sh (the documented recovery) writes credentials.json but then runs `docker compose up -d`, which is a no-op for an already-running container. `credentials.load()` runs only in the FastAPI lifespan at startup, and `uvicorn --reload` watches /app, not /configs — so the running process keeps api_key=None until `docker compose restart edge-node`. **Ruled out:** WireGuard. install.sh intentionally doesn't provision it; nodes use public `mqtt.drb.cusano.net:8883` (verified from the dev box 2026-10-04: TCP+TLS OK, LE cert CN=mqtt.drb.cusano.net, verify 0). `api.drb.cusano.net/health` = ok, mqtt_connected=true, sha fb29900. The `WireGuard` comments in `Server/docker-compose.prod.yml` / `infra/main.tf` are stale. **Not verified:** prod admin UI state for Toughny (Chrome got an error page on drb.cusano.net), and broker logs (`ssh drb` does not resolve from the dev box). Confirm on the node: `cd /opt/drb/node-26 && ls -l configs/ && docker compose logs --tail=50 edge-node | grep -iE 'mqtt|api key'` — expect no credentials.json and repeated `MQTT connect refused`. **Owner recovery on Toughny (in order):** 1. Approve Toughny at https://drb.cusano.net/settings/nodes (if not already). 2. On Toughny: `cd /opt/drb/node-26 && sudo bash install.sh` (re-run; reuses configs/pickup_secret, collects the key). If it says pickup secret 401/404, re-run with `--token DRB-…`. 3. `test -s configs/credentials.json && echo key-present` 4. `sudo docker compose restart edge-node` 5. `sudo docker compose logs --tail=30 edge-node | grep -i mqtt` → expect `MQTT connected.` **Proposed fix:** - install.sh: after writing creds, `docker compose restart edge-node` if it was already running (or `up -d --force-recreate edge-node`). - edge-node app: own the pickup — persist pickup_secret, and while api_key is absent poll `GET {C2_URL}/nodes/{id}/credentials` (e.g. every 60s), then save the key and rebuild the MQTT client in place. This is the 'client-side enrollment' follow-up named in node-26#4/#6. - install.sh: default `--wait-approval` to something non-zero when interactive, and make the final banner say plainly that the node stays offline until re-run + restart. Related: node-26#4, node-26#6.
logan commented 2026-10-04 00:53:52 -04:00
Author
Owner
Copy Link
Copy Source

Prod evidence 2026-10-04 (correcting the original write-up, which never checked prod):

  • Toughny IS approved (owner). /nodes shows Last seen: never — it has never completed one MQTT checkin. Enrollment token Toughny = 1 use; a second Toughny token revoked; New-Toughny token 0 uses.
  • Owner reports the broker answers access denied.
  • Server side is consistent: approve_node() writes the dynsec client (username=node_id, password=new api_key) before Firestore, and reconcile_all() re-syncs at startup. So the denial is node-side: the node presents either no key (approved after install.sh ran with default --wait-approval 0) or a stale key — approve_node() mints a fresh secrets.token_hex(32) on every call and node_keys is overwritten (merge=False), so any second approve/reissue silently kills the key already on the node.
  • There is no remote recovery path. A node that can't authenticate to MQTT can't receive nodes/{id}/api_key, and the edge-node app never calls GET /nodes/{id}/credentials itself. A remote node with no SSH is stuck until someone on site re-runs install.sh + restarts edge-node.

Fix scope (both needed): (1) edge-node app polls GET /nodes/{id}/credentials with configs/pickup_secret on startup and on every MQTT not authorised, hot-reloads the key and reconnects; (2) approve on an already-approved node must not rotate the key (or the UI must warn it will lock the node out).

Prod evidence 2026-10-04 (correcting the original write-up, which never checked prod): - Toughny IS approved (owner). /nodes shows `Last seen: never` — it has never completed one MQTT checkin. Enrollment token `Toughny` = 1 use; a second `Toughny` token revoked; `New-Toughny` token 0 uses. - Owner reports the broker answers **access denied**. - Server side is consistent: `approve_node()` writes the dynsec client (username=node_id, password=new api_key) before Firestore, and `reconcile_all()` re-syncs at startup. So the denial is node-side: the node presents either **no key** (approved after install.sh ran with default `--wait-approval 0`) or a **stale key** — `approve_node()` mints a fresh `secrets.token_hex(32)` on every call and `node_keys` is overwritten (`merge=False`), so any second approve/reissue silently kills the key already on the node. - **There is no remote recovery path.** A node that can't authenticate to MQTT can't receive `nodes/{id}/api_key`, and the edge-node app never calls `GET /nodes/{id}/credentials` itself. A remote node with no SSH is stuck until someone on site re-runs install.sh + restarts edge-node. Fix scope (both needed): (1) edge-node app polls `GET /nodes/{id}/credentials` with `configs/pickup_secret` on startup and on every MQTT `not authorised`, hot-reloads the key and reconnects; (2) approve on an already-approved node must not rotate the key (or the UI must warn it will lock the node out).
logan commented 2026-10-04 00:55:34 -04:00
Author
Owner
Copy Link
Copy Source

Owner confirms install.sh WAS re-run after approval. That narrows it to the restart gap: re-run writes configs/credentials.json (install.sh write_creds), then docker compose up -d / up --no-build -d leaves the already-running edge-node untouched. credentials.load() (app/internal/credentials.py:26) reads the key once at startup into a module global; mqtt_manager builds the client from that cached value, so paho keeps reconnecting with no credentials → not authorised, forever. Any restart of edge-node (incl. a device power-cycle, compose has restart: unless-stopped) should pick the key up — unless the node was approved/reissued again after the re-run, which would make the on-disk key stale. Minimal installer fix: docker compose up -d --force-recreate edge-node (or restart edge-node) after write_creds.

Owner confirms install.sh WAS re-run after approval. That narrows it to the restart gap: re-run writes `configs/credentials.json` (install.sh `write_creds`), then `docker compose up -d` / `up --no-build -d` leaves the already-running edge-node untouched. `credentials.load()` (app/internal/credentials.py:26) reads the key once at startup into a module global; `mqtt_manager` builds the client from that cached value, so paho keeps reconnecting with no credentials → `not authorised`, forever. Any restart of edge-node (incl. a device power-cycle, compose has `restart: unless-stopped`) should pick the key up — unless the node was approved/reissued again after the re-run, which would make the on-disk key stale. Minimal installer fix: `docker compose up -d --force-recreate edge-node` (or `restart edge-node`) after `write_creds`.
logan commented 2026-10-04 01:12:07 -04:00
Author
Owner
Copy Link
Copy Source

Fix is up and not merged. The owner decides when to merge.

  • node-26#16:
    • install.sh blocks until the node is approved, refreshes a stale key on re-run, and always force-recreates edge-node.
    • The edge-node app self-heals: with no key at startup, or when the broker refuses auth, it re-reads credentials.json, then falls back to GET /nodes/{id}/credentials with the pickup secret, and rebuilds the MQTT client in place.
  • server-26#194: approve_node reuses an existing key and never rotates it.

Toughny recovery (after #16 merges and v1 is re-cut at the merge commit):

  1. cd /opt/drb/node-26 && sudo ls -l configs/. Expect pickup_secret.
  2. curl -fsSL https://git.vpn.cusano.net/logan/node-26/raw/tag/v1/install.sh -o /tmp/drb-install.sh
  3. sudo bash /tmp/drb-install.sh. It keeps .env, checks the key against C2 and replaces it if stale, recreates edge-node, and installs the updater.
  4. sudo docker compose logs --tail=40 edge-node | grep -iE 'mqtt|key'. Expect MQTT connected.
  5. Last seen at https://drb.cusano.net/nodes should update within ~30s.

Follow-ups filed: #17 (OTA signing), #18 (--token on argv), #19 (code/image skew), server-26#195 (reject doesn't revoke).

Fix is up and **not merged**. The owner decides when to merge. - **node-26#16**: - install.sh blocks until the node is approved, refreshes a stale key on re-run, and always force-recreates edge-node. - The edge-node app self-heals: with no key at startup, or when the broker refuses auth, it re-reads `credentials.json`, then falls back to `GET /nodes/{id}/credentials` with the pickup secret, and rebuilds the MQTT client in place. - **server-26#194**: `approve_node` reuses an existing key and never rotates it. **Toughny recovery** (after #16 merges and `v1` is re-cut at the merge commit): 1. `cd /opt/drb/node-26 && sudo ls -l configs/`. Expect `pickup_secret`. 2. `curl -fsSL https://git.vpn.cusano.net/logan/node-26/raw/tag/v1/install.sh -o /tmp/drb-install.sh` 3. `sudo bash /tmp/drb-install.sh`. It keeps `.env`, checks the key against C2 and replaces it if stale, recreates edge-node, and installs the updater. 4. `sudo docker compose logs --tail=40 edge-node | grep -iE 'mqtt|key'`. Expect `MQTT connected.` 5. Last seen at https://drb.cusano.net/nodes should update within ~30s. Follow-ups filed: #17 (OTA signing), #18 (`--token` on argv), #19 (code/image skew), server-26#195 (reject doesn't revoke).
logan commented 2026-10-04 01:26:41 -04:00
Author
Owner
Copy Link
Copy Source

Fixed: node-26#16 (f46c798) + server-26#194 (ae0ece9); v1 re-cut at 1e1dab6. Toughny itself still needs the one-time SSH re-run (steps above) — existing nodes can only receive the updater by running the new install.sh once.

Fixed: node-26#16 (f46c798) + server-26#194 (ae0ece9); v1 re-cut at 1e1dab6. Toughny itself still needs the one-time SSH re-run (steps above) — existing nodes can only receive the updater by running the new install.sh once.
logan closed this issue 2026-10-04 01:26:41 -04:00
logan added the area/mqttarea/node-ota
type
bug
 labels 2026-10-05 02:05:09 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/mqtt area/node-ota
type
bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/node-26#14