Secure the broker for public exposure: TLS and per-node credentials

Edge nodes are deployed to arbitrary locations by arbitrary people, so the
broker has to be reachable from the internet and secured on its own merits
rather than by a VPN.

Three defects made that impossible. The broker only had a plaintext 1883
listener; every node shared one drb-node password; and the ACL pattern used
%c, the client-supplied client id, so any holder of that shared password
could set client_id to another node and take over its namespace. The comment
claiming this cryptographically prevented cross-node access was wrong and is
gone.

Authentication now uses mosquitto 2.x's built-in dynamic-security plugin on
the stock eclipse-mosquitto image. c2-core administers it over the control
topic, creating each node's client on approval with username=<node_id> and
password=<its node_keys api_key>, attached to a role whose ACL is nodes/%u/#
against the authenticated username. One credential, one revocation point.
An HTTP-callback plugin was implemented first and rejected: that project is
archived upstream, which is not an acceptable dependency on an
internet-facing broker.

Because dynsec state is a second source of truth alongside Firestore,
approve/reissue/delete now write to the broker first and surface a 502
rather than drifting, and c2-core reconciles every approved node into dynsec
on startup.

Adds node self-enrollment (POST /nodes/enroll, GET /nodes/{id}/credentials)
so a new node can obtain its key over HTTPS without an operator handling
secrets by hand. Enrolling an already-approved node_id is refused on the
fleet token alone — otherwise a leaked token plus a guessable id would let
an attacker steal a live node's key before the real node asked for it.
Pickup secrets are stored hashed and returned once, and the endpoint is rate
limited per source IP.

Infrastructure: an 8883 TLS listener fed by Caddy's certificate via a
systemd path unit, a firewall rule for it, and Caddy now 404s /internal/*
so the api vhost cannot proxy internal routes.

Also fixes CORS, which allowed https://app.<domain> while the frontend is
served on the bare domain — every call from the portal would have failed —
and widens the vault gitignore to a glob, since ansible-vault leaves
backup siblings that the exact-name rule left committable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

infra/ansible/roles/deploy/handlers/main.yml

@@ -11,3 +11,9 @@
     name: caddy
     state: reloaded
     enabled: true
+
+# For the mqtt-cert-sync.service/.path unit files — systemd won't pick up a
+# new/changed unit file until the manager config is reloaded.
+- name: Reload systemd daemon
+  ansible.builtin.systemd_service:
+    daemon_reload: true

infra/ansible/roles/deploy/tasks/main.yml

@@ -66,6 +66,83 @@
     mode: "0644"
   notify: Reload Caddy
 
+# --- MQTT TLS cert sync (Caddy -> mosquitto) --------------------------------
+# See MQTT-PUBLIC-AUTH-PLAN.md "Infra". mosquitto reads its cert from this
+# directory (docker-compose.prod.yml bind-mounts it in); nothing but root can
+# read Caddy's own cert storage, so a systemd path unit + oneshot service
+# copies a readable copy out and SIGHUPs the broker on every change.
+
+- name: Create mosquitto certs directory
+  file:
+    path: /opt/drb/mosquitto-certs
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+# dynamic-security.json (node credentials — see app/internal/dynsec.py)
+# lives here. Root-owned is fine: the mosquitto container itself runs as
+# root (no `user` directive in mosquitto.conf, matching the pre-existing
+# setup this project already ran before the dynsec change), so it can
+# read/write this directory directly without any host-side chown dance.
+- name: Create mosquitto data directory
+  file:
+    path: /opt/drb/mosquitto-data
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy MQTT cert-sync script
+  template:
+    src: sync-mqtt-cert.sh.j2
+    dest: /opt/drb/sync-mqtt-cert.sh
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Deploy MQTT cert-sync systemd service unit
+  template:
+    src: mqtt-cert-sync.service.j2
+    dest: /etc/systemd/system/mqtt-cert-sync.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd daemon
+
+- name: Deploy MQTT cert-sync systemd path unit
+  template:
+    src: mqtt-cert-sync.path.j2
+    dest: /etc/systemd/system/mqtt-cert-sync.path
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd daemon
+
+# Flush the daemon-reload handler now (rather than at end-of-play) so the
+# path unit is registered and actively watching BEFORE the "Reload Caddy"
+# handler below fires and Caddy goes to obtain the mqtt.{{ domain }} cert —
+# otherwise the unit could miss the very first PathChanged event.
+- name: Apply pending handlers (systemd daemon-reload)
+  meta: flush_handlers
+
+- name: Enable and start MQTT cert-sync path unit
+  ansible.builtin.systemd_service:
+    name: mqtt-cert-sync.path
+    state: started
+    enabled: true
+
+# Best-effort initial sync in case Caddy already has a cert from a previous
+# run (e.g. re-running this playbook after the first successful deploy) —
+# the path unit only fires on a CHANGE, so it won't pick up a cert that was
+# already sitting there unchanged before it started watching. Non-fatal if
+# nothing exists yet (first-ever run, before Caddy has issued anything).
+- name: Best-effort initial MQTT cert sync
+  command: /opt/drb/sync-mqtt-cert.sh
+  register: _initial_sync
+  changed_when: "'copied cert' in _initial_sync.stdout"
+  failed_when: false
+
 - name: Log in to container registry
   command: >
     docker login {{ vault_registry_host }}

infra/ansible/roles/deploy/templates/Caddyfile.j2

@@ -1,11 +1,34 @@
 # Managed by Ansible — do not edit manually.
 
 api.{{ domain }} {
-  reverse_proxy localhost:8888 {
-    header_up X-Forwarded-For {remote_host}
+  # MQTT auth is no longer an HTTP backend c2-core exposes (it moved to
+  # mosquitto's own built-in dynamic-security plugin, administered over MQTT
+  # control topics — see app/internal/dynsec.py) — there is currently no
+  # /internal/* route in c2-core at all. This block stays anyway as defence
+  # in depth: c2-core's app-wide reverse_proxy below forwards every path by
+  # default, so this guarantees any FUTURE /internal/* route (or a
+  # regression that reintroduces one) is still unreachable from the public
+  # internet unless someone also deliberately deletes this block. `route`
+  # forces top-to-bottom evaluation instead of Caddy's automatic directive
+  # sorting, so this is guaranteed to run before reverse_proxy.
+  route {
+    respond /internal/* 404
+    reverse_proxy localhost:8888 {
+      header_up X-Forwarded-For {remote_host}
+    }
   }
 }
 
+# mqtt.{{ domain }} has no application behind it — mosquitto's TLS listener
+# (8883) is a raw MQTT socket, not HTTP, so Caddy can't reverse_proxy to it.
+# This block's only job is to make Caddy request+manage a Let's Encrypt cert
+# for the name via ACME HTTP-01, which infra/ansible's cert-sync unit then
+# copies out to mosquitto. The DNS A record for mqtt.{{ domain }} must exist
+# before this runs, or ACME issuance fails (see MQTT-PUBLIC-AUTH-PLAN.md).
+mqtt.{{ domain }} {
+  respond 404
+}
+
 # Frontend is served on the bare domain, not app.{{ domain }}: only drb and api
 # have public DNS records. A vhost for a name with no A record still starts,
 # but Caddy retries ACME against it forever and logs a failure each time.

infra/ansible/roles/deploy/templates/c2-core.env.j2

@@ -5,6 +5,11 @@ MQTT_PORT=1883
 MQTT_USER={{ vault_mqtt_c2_user }}
 MQTT_PASS={{ vault_mqtt_c2_pass }}
 
+# Same value as mosquitto's MOSQUITTO_DYNSEC_PASSWORD (root.env.j2) — lets
+# c2-core log in as the dynsec plugin's built-in "admin" client to
+# administer node credentials. See app/internal/dynsec.py.
+MQTT_DYNSEC_ADMIN_PASS={{ vault_mqtt_dynsec_admin_pass }}
+
 # No GCP_CREDENTIALS_PATH — the VM uses Application Default Credentials
 # via the GCE metadata server. The Terraform IAM bindings grant the required roles.
 FIRESTORE_DATABASE={{ vault_firestore_database }}
@@ -15,6 +20,11 @@ GOOGLE_MAPS_API_KEY={{ vault_google_maps_api_key }}
 GEMINI_API_KEY={{ vault_gemini_api_key }}
 
 SERVICE_KEY={{ vault_service_key }}
-NODE_API_KEY={{ vault_node_api_key }}
+ENROLLMENT_TOKEN={{ vault_enrollment_token }}
 
-CORS_ORIGINS=["https://app.{{ domain }}"]
+# Bare domain, not app.<domain>: the frontend is served on {{ domain }} itself
+# (see Caddyfile.j2 — only api. and the bare name have DNS records). This said
+# app.{{ domain }} while the browser origin was https://{{ domain }}, so every
+# frontend call to the API would have failed CORS. If the frontend ever moves
+# to app.{{ domain }}, change this at the same time.
+CORS_ORIGINS=["https://{{ domain }}"]

infra/ansible/roles/deploy/templates/mqtt-cert-sync.path.j2

@@ -0,0 +1,18 @@
+# Managed by Ansible — do not edit manually.
+#
+# Watches Caddy's on-disk cert for mqtt.{{ domain }} and fires
+# mqtt-cert-sync.service on every change (initial issuance + every renewal).
+# See sync-mqtt-cert.sh.j2 for the "unverified path" caveat — if Caddy's
+# storage layout doesn't match, this unit simply never fires and mosquitto
+# keeps using its self-signed placeholder cert (see mosquitto/entrypoint.sh)
+# rather than failing loudly, so check `systemctl status mqtt-cert-sync.path`
+# after the first deploy.
+[Unit]
+Description=Watch for a renewed MQTT TLS cert from Caddy (mqtt.{{ domain }})
+
+[Path]
+PathChanged=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/mqtt.{{ domain }}/mqtt.{{ domain }}.crt
+Unit=mqtt-cert-sync.service
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/deploy/templates/mqtt-cert-sync.service.j2

@@ -0,0 +1,11 @@
+# Managed by Ansible — do not edit manually.
+#
+# Runs as root: it needs read access to Caddy's 0700 cert storage AND the
+# ability to run `docker compose kill -s HUP` regardless of docker group
+# membership. Triggered by mqtt-cert-sync.path, not run standalone.
+[Unit]
+Description=Sync Caddy-issued MQTT TLS cert to mosquitto and reload
+
+[Service]
+Type=oneshot
+ExecStart=/opt/drb/sync-mqtt-cert.sh

infra/ansible/roles/deploy/templates/root.env.j2

@@ -12,8 +12,15 @@
 
 MQTT_C2_USER={{ vault_mqtt_c2_user }}
 MQTT_C2_PASS={{ vault_mqtt_c2_pass | replace('$', '$$') }}
-MQTT_NODE_USER={{ vault_mqtt_node_user }}
-MQTT_NODE_PASS={{ vault_mqtt_node_pass | replace('$', '$$') }}
+
+# Seeds mosquitto's built-in dynamic-security plugin's one-time "admin"
+# bootstrap client on first boot (read directly by the plugin's C code via
+# getenv — see app/internal/dynsec.py). Must be >=12 chars (plugin-enforced
+# minimum). c2-core needs this SAME value as MQTT_DYNSEC_ADMIN_PASS in its
+# own env (c2-core.env.j2) to log in as "admin" and administer node
+# credentials — kept as one vault var (vault_mqtt_dynsec_admin_pass) so the
+# two can't drift.
+MOSQUITTO_DYNSEC_PASSWORD={{ vault_mqtt_dynsec_admin_pass | replace('$', '$$') }}
 
 # Container registry prefix — docker compose uses this for image: ${REGISTRY}/name:latest
 REGISTRY={{ vault_registry }}

infra/ansible/roles/deploy/templates/sync-mqtt-cert.sh.j2

@@ -0,0 +1,51 @@
+#!/bin/bash
+# Managed by Ansible — do not edit manually.
+#
+# Copies Caddy's managed TLS cert for mqtt.{{ domain }} out of Caddy's
+# storage (root:caddy, 0700 — nothing else can read it) into a location the
+# mosquitto container can read, then SIGHUPs the broker so it picks up the
+# new cert without a full restart.
+#
+# Triggered by mqtt-cert-sync.path.j2 (a systemd path unit) watching the
+# source cert file for changes — a path unit rather than cron so this fires
+# on the actual write instead of racing a polling interval.
+#
+# UNVERIFIED: the exact source path below assumes Caddy's default file
+# storage layout and Let's Encrypt's production ACME directory name. This
+# has not been confirmed against a real Caddy cert issuance for this
+# project — check `caddy storage` / find the actual path under
+# /var/lib/caddy the first time this runs, and correct CADDY_CERT_DIR below
+# if it doesn't match.
+#
+# UNVERIFIED: mosquitto 2.x reloading TLS certs on SIGHUP without dropping
+# connections is documented upstream but untested here. If listener 8883
+# doesn't pick up the new cert (check `docker compose logs mosquitto` after
+# a sync), replace the `kill -s HUP` line below with a full
+# `docker compose ... restart mosquitto` instead.
+set -euo pipefail
+
+DOMAIN="mqtt.{{ domain }}"
+CADDY_CERT_DIR="/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/${DOMAIN}"
+DEST_DIR="/opt/drb/mosquitto-certs"
+APP_DIR="{{ app_dir }}"
+
+SRC_CERT="${CADDY_CERT_DIR}/${DOMAIN}.crt"
+SRC_KEY="${CADDY_CERT_DIR}/${DOMAIN}.key"
+
+if [ ! -f "$SRC_CERT" ] || [ ! -f "$SRC_KEY" ]; then
+    echo "sync-mqtt-cert: source cert/key not found yet at $CADDY_CERT_DIR — Caddy may not have issued it yet." >&2
+    exit 0
+fi
+
+mkdir -p "$DEST_DIR"
+# Copy, don't symlink — nothing outside the caddy user can read the
+# originals (0700-owned), so mosquitto (running as a different container/
+# user) needs its own readable copy, not a pointer to an unreadable file.
+cp "$SRC_CERT" "$DEST_DIR/mqtt.crt"
+cp "$SRC_KEY"  "$DEST_DIR/mqtt.key"
+chmod 600 "$DEST_DIR/mqtt.crt" "$DEST_DIR/mqtt.key"
+chown root:root "$DEST_DIR/mqtt.crt" "$DEST_DIR/mqtt.key"
+
+cd "$APP_DIR"
+docker compose -f docker-compose.yml -f docker-compose.prod.yml kill -s HUP mosquitto
+echo "sync-mqtt-cert: copied cert for ${DOMAIN} and sent SIGHUP to mosquitto."

infra/ansible/vault.yml.example

@@ -12,14 +12,18 @@
 # Generate with: openssl rand -hex 32  (hex output has no shell metacharacters)
 
 # ── MQTT ─────────────────────────────────────────────────────────────────────
+# No more shared node credential (vault_mqtt_node_user/pass) — nodes now
+# authenticate as username=<node_id>, password=<their node_keys.api_key>,
+# checked by mosquitto's built-in dynamic-security plugin (c2-core
+# administers it — see app/internal/dynsec.py). See vault_enrollment_token
+# below for how a node gets that key in the first place.
 vault_mqtt_c2_user: drb-c2-core
 vault_mqtt_c2_pass: "CHANGE_ME"
-vault_mqtt_node_user: drb-node
-vault_mqtt_node_pass: "CHANGE_ME"
+vault_mqtt_dynsec_admin_pass: "CHANGE_ME"  # openssl rand -hex 32 — must be >=12 chars, plugin-enforced minimum
 
 # ── C2 Core ───────────────────────────────────────────────────────────────────
 vault_service_key: ""            # openssl rand -hex 32
-vault_node_api_key: ""           # openssl rand -hex 32
+vault_enrollment_token: ""       # openssl rand -hex 32 — fleet-wide, shared by every node's POST /nodes/enroll
 vault_openai_api_key: ""
 vault_google_maps_api_key: ""
 vault_gemini_api_key: ""

infra/main.tf

@@ -64,20 +64,25 @@ resource "google_compute_firewall" "allow_ssh" {
   target_tags   = ["drb-server"]
 }
 
-# MQTT is NOT exposed externally — edge nodes connect via WireGuard (see below)
-# If you need to temporarily allow direct MQTT access for testing, uncomment and
-# restrict source_ranges to your node IPs.
-#
-# resource "google_compute_firewall" "allow_mqtt" {
-#   name    = "drb-allow-mqtt"
-#   network = "default"
-#   allow {
-#     protocol = "tcp"
-#     ports    = ["8883"]  # TLS MQTT, not 1883
-#   }
-#   source_ranges = ["YOUR_NODE_CIDR"]
-#   target_tags   = ["drb-server"]
-# }
+# MQTT is now publicly exposed on 8883 (TLS) — nodes get deployed to
+# arbitrary locations by arbitrary people, so there is no fixed CIDR to
+# restrict this to (WireGuard-per-node was evaluated and rejected; see
+# MQTT-PUBLIC-AUTH-PLAN.md). Security is enforced by mosquitto-go-auth
+# (per-node api_key over TLS), not by network ACL. 1883 (plaintext) is
+# intentionally NOT opened here — it stays on the docker bridge for
+# c2-core's own connection only.
+resource "google_compute_firewall" "allow_mqtt" {
+  name    = "drb-allow-mqtt"
+  network = "default"
+
+  allow {
+    protocol = "tcp"
+    ports    = ["8883"]  # TLS MQTT only, not 1883
+  }
+
+  source_ranges = ["0.0.0.0/0"]
+  target_tags   = ["drb-server"]
+}
 
 # ---------------------------------------------------------------------------
 # Compute Engine VM
@@ -185,5 +190,9 @@ resource "google_storage_bucket" "audio" {
 # After terraform apply, add these A records in Route 53:
 #   app.drb.cusano.net  → server_ip output
 #   api.drb.cusano.net  → server_ip output
+#   mqtt.drb.cusano.net → server_ip output  — MUST exist before the ansible
+#                                              deploy that adds the Caddy
+#                                              mqtt.<domain> block, or ACME
+#                                              issuance for it fails.
 # Or use a single wildcard: *.drb.cusano.net → server_ip
 # ---------------------------------------------------------------------------