Secure the broker for public exposure: TLS and per-node credentials

Edge nodes are deployed to arbitrary locations by arbitrary people, so the
broker has to be reachable from the internet and secured on its own merits
rather than by a VPN.

Three defects made that impossible. The broker only had a plaintext 1883
listener; every node shared one drb-node password; and the ACL pattern used
%c, the client-supplied client id, so any holder of that shared password
could set client_id to another node and take over its namespace. The comment
claiming this cryptographically prevented cross-node access was wrong and is
gone.

Authentication now uses mosquitto 2.x's built-in dynamic-security plugin on
the stock eclipse-mosquitto image. c2-core administers it over the control
topic, creating each node's client on approval with username=<node_id> and
password=<its node_keys api_key>, attached to a role whose ACL is nodes/%u/#
against the authenticated username. One credential, one revocation point.
An HTTP-callback plugin was implemented first and rejected: that project is
archived upstream, which is not an acceptable dependency on an
internet-facing broker.

Because dynsec state is a second source of truth alongside Firestore,
approve/reissue/delete now write to the broker first and surface a 502
rather than drifting, and c2-core reconciles every approved node into dynsec
on startup.

Adds node self-enrollment (POST /nodes/enroll, GET /nodes/{id}/credentials)
so a new node can obtain its key over HTTPS without an operator handling
secrets by hand. Enrolling an already-approved node_id is refused on the
fleet token alone — otherwise a leaked token plus a guessable id would let
an attacker steal a live node's key before the real node asked for it.
Pickup secrets are stored hashed and returned once, and the endpoint is rate
limited per source IP.

Infrastructure: an 8883 TLS listener fed by Caddy's certificate via a
systemd path unit, a firewall rule for it, and Caddy now 404s /internal/*
so the api vhost cannot proxy internal routes.

Also fixes CORS, which allowed https://app.<domain> while the frontend is
served on the bare domain — every call from the portal would have failed —
and widens the vault gitignore to a glob, since ansible-vault leaves
backup siblings that the exact-name rule left committable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

drb-c2-core/mosquitto/acl.conf

@@ -1,19 +0,0 @@
-# -----------------------------------------------------------------------
-# Mosquitto ACL — DRB C2 Server
-# -----------------------------------------------------------------------
-# Two principals:
-#   drb-c2-core  — the backend service; needs full broker access
-#   drb-node     — shared credential for all edge nodes; scoped to their
-#                  own namespace via MQTT client ID (%c = NODE_ID)
-# -----------------------------------------------------------------------
-
-# C2-core service — full read/write on every topic
-user drb-c2-core
-topic readwrite #
-
-# Edge nodes — each node may only read/write topics under nodes/<its-own-ID>/
-# Mosquitto substitutes %c with the connecting client's MQTT client ID at
-# runtime. Edge nodes set client_id = NODE_ID in mqtt_manager.py, so this
-# cryptographically prevents node-A from publishing to nodes/node-B/api_key
-# or any other node's namespace.
-pattern readwrite nodes/%c/#

drb-c2-core/mosquitto/entrypoint.sh

@@ -1,37 +0,0 @@
-#!/bin/sh
-# Mosquitto entrypoint — generates /mosquitto/config/passwd from env vars
-# before handing off to the broker process.
-#
-# Required environment variables (set in docker-compose.yml):
-#   MQTT_C2_USER   — username for the drb-c2-core service
-#   MQTT_C2_PASS   — password for the drb-c2-core service
-#   MQTT_NODE_USER — shared username for all edge nodes
-#   MQTT_NODE_PASS — shared password for all edge nodes
-set -e
-
-PASSWD_FILE=/tmp/passwd
-
-# Remove any stale file so we start clean on every container start
-rm -f "$PASSWD_FILE"
-
-if [ -z "$MQTT_C2_USER" ] || [ -z "$MQTT_C2_PASS" ]; then
-    echo "ERROR: MQTT_C2_USER and MQTT_C2_PASS must be set" >&2
-    exit 1
-fi
-
-if [ -z "$MQTT_NODE_USER" ] || [ -z "$MQTT_NODE_PASS" ]; then
-    echo "ERROR: MQTT_NODE_USER and MQTT_NODE_PASS must be set" >&2
-    exit 1
-fi
-
-# -c creates/overwrites the file; subsequent calls append without -c
-mosquitto_passwd -c -b "$PASSWD_FILE" "$MQTT_C2_USER"   "$MQTT_C2_PASS"
-mosquitto_passwd    -b "$PASSWD_FILE" "$MQTT_NODE_USER" "$MQTT_NODE_PASS"
-
-# mosquitto_passwd creates the file 0600 (root-only); mosquitto drops to
-# the mosquitto user before reading it, so make it world-readable.
-chmod 644 "$PASSWD_FILE"
-
-echo "Mosquitto: password file written for users: $MQTT_C2_USER, $MQTT_NODE_USER"
-
-exec /usr/sbin/mosquitto -c /mosquitto/config/mosquitto.conf

drb-c2-core/mosquitto/mosquitto.conf

@@ -1,11 +1,43 @@
-listener 1883
+# Auth: mosquitto's own built-in dynamic-security plugin — NOT
+# mosquitto-go-auth (that project is archived upstream, no CVE patches;
+# rejected for an internet-facing broker). This plugin ships in and is
+# maintained alongside the official eclipse-mosquitto image itself.
+# See MQTT-PUBLIC-AUTH-PLAN.md and app/internal/dynsec.py for the full
+# design (bootstrap, roles, the two-sources-of-truth reconcile).
+#
+# Plugin path is DERIVED FROM SOURCE (docker/2.1-alpine/Dockerfile in
+# eclipse-mosquitto/mosquitto), not observed by running the image —
+# nothing in this project executes/pulls images from this machine. Verify
+# it on first real deploy: `docker compose logs mosquitto` will say
+# "Error: Unable to load plugin" at the exact path below if it's wrong for
+# whatever patch tag ends up pinned.
+plugin /usr/lib/mosquitto_dynamic_security.so
+# Lives on the same persistent volume as `persistence_location` below —
+# one durable volume for all broker state, survives redeploys.
+plugin_opt_config_file /mosquitto/data/dynamic-security.json
+
 allow_anonymous false
+# No password_file/acl_file directive anywhere in this file — the plugin
+# above is the only registered auth backend. There is no "coexist" mode:
+# nothing else is registered to conflict with it.
 
-# Credentials and ACLs are generated/mounted at container startup
-password_file /tmp/passwd
-acl_file      /mosquitto/config/acl.conf
+# Internal, plaintext — c2-core's own connection only (its dynsec-admin
+# control-plane calls AND its regular data-plane pub/sub both use this).
+# Never published to the host in prod (docker-compose.prod.yml removes the
+# port mapping); external nodes use the TLS listener below instead.
+listener 1883
+
+# Public, TLS — edge nodes connect here as username=node_id, password=api_key
+# (the same credential /upload already trusts via node_keys), authorized by
+# the "node" dynsec role (nodes/%u/# — %u is the dynsec-authenticated
+# username, fixing the old %c-based ACL's client-ID-spoofing hole). Cert/key
+# come from infra/ansible's Caddy cert-sync unit; see
+# MQTT-PUBLIC-AUTH-PLAN.md "Infra" and the "Rollout order" cert-verification
+# step for what happens before that cert exists.
+listener 8883
+certfile /mosquitto/certs/mqtt.crt
+keyfile  /mosquitto/certs/mqtt.key
 
-# Persist retained messages (e.g. api_key, node status) across broker restarts
 persistence true
 persistence_location /mosquitto/data/