Secure the broker for public exposure: TLS and per-node credentials
Edge nodes are deployed to arbitrary locations by arbitrary people, so the
broker has to be reachable from the internet and secured on its own merits
rather than by a VPN.
Three defects made that impossible. The broker only had a plaintext 1883
listener; every node shared one drb-node password; and the ACL pattern used
%c, the client-supplied client id, so any holder of that shared password
could set client_id to another node and take over its namespace. The comment
claiming this cryptographically prevented cross-node access was wrong and is
gone.
Authentication now uses mosquitto 2.x's built-in dynamic-security plugin on
the stock eclipse-mosquitto image. c2-core administers it over the control
topic, creating each node's client on approval with username=<node_id> and
password=<its node_keys api_key>, attached to a role whose ACL is nodes/%u/#
against the authenticated username. One credential, one revocation point.
An HTTP-callback plugin was implemented first and rejected: that project is
archived upstream, which is not an acceptable dependency on an
internet-facing broker.
Because dynsec state is a second source of truth alongside Firestore,
approve/reissue/delete now write to the broker first and surface a 502
rather than drifting, and c2-core reconciles every approved node into dynsec
on startup.
Adds node self-enrollment (POST /nodes/enroll, GET /nodes/{id}/credentials)
so a new node can obtain its key over HTTPS without an operator handling
secrets by hand. Enrolling an already-approved node_id is refused on the
fleet token alone — otherwise a leaked token plus a guessable id would let
an attacker steal a live node's key before the real node asked for it.
Pickup secrets are stored hashed and returned once, and the endpoint is rate
limited per source IP.
Infrastructure: an 8883 TLS listener fed by Caddy's certificate via a
systemd path unit, a firewall rule for it, and Caddy now 404s /internal/*
so the api vhost cannot proxy internal routes.
Also fixes CORS, which allowed https://app.<domain> while the frontend is
served on the bare domain — every call from the portal would have failed —
and widens the vault gitignore to a glob, since ansible-vault leaves
backup siblings that the exact-name rule left committable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1f5f1fede8
commit
ee633cbe46
@@ -66,6 +66,83 @@
|
||||
mode: "0644"
|
||||
notify: Reload Caddy
|
||||
|
||||
# --- MQTT TLS cert sync (Caddy -> mosquitto) --------------------------------
|
||||
# See MQTT-PUBLIC-AUTH-PLAN.md "Infra". mosquitto reads its cert from this
|
||||
# directory (docker-compose.prod.yml bind-mounts it in); nothing but root can
|
||||
# read Caddy's own cert storage, so a systemd path unit + oneshot service
|
||||
# copies a readable copy out and SIGHUPs the broker on every change.
|
||||
|
||||
- name: Create mosquitto certs directory
|
||||
file:
|
||||
path: /opt/drb/mosquitto-certs
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
|
||||
# dynamic-security.json (node credentials — see app/internal/dynsec.py)
|
||||
# lives here. Root-owned is fine: the mosquitto container itself runs as
|
||||
# root (no `user` directive in mosquitto.conf, matching the pre-existing
|
||||
# setup this project already ran before the dynsec change), so it can
|
||||
# read/write this directory directly without any host-side chown dance.
|
||||
- name: Create mosquitto data directory
|
||||
file:
|
||||
path: /opt/drb/mosquitto-data
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
|
||||
- name: Deploy MQTT cert-sync script
|
||||
template:
|
||||
src: sync-mqtt-cert.sh.j2
|
||||
dest: /opt/drb/sync-mqtt-cert.sh
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
|
||||
- name: Deploy MQTT cert-sync systemd service unit
|
||||
template:
|
||||
src: mqtt-cert-sync.service.j2
|
||||
dest: /etc/systemd/system/mqtt-cert-sync.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Reload systemd daemon
|
||||
|
||||
- name: Deploy MQTT cert-sync systemd path unit
|
||||
template:
|
||||
src: mqtt-cert-sync.path.j2
|
||||
dest: /etc/systemd/system/mqtt-cert-sync.path
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Reload systemd daemon
|
||||
|
||||
# Flush the daemon-reload handler now (rather than at end-of-play) so the
|
||||
# path unit is registered and actively watching BEFORE the "Reload Caddy"
|
||||
# handler below fires and Caddy goes to obtain the mqtt.{{ domain }} cert —
|
||||
# otherwise the unit could miss the very first PathChanged event.
|
||||
- name: Apply pending handlers (systemd daemon-reload)
|
||||
meta: flush_handlers
|
||||
|
||||
- name: Enable and start MQTT cert-sync path unit
|
||||
ansible.builtin.systemd_service:
|
||||
name: mqtt-cert-sync.path
|
||||
state: started
|
||||
enabled: true
|
||||
|
||||
# Best-effort initial sync in case Caddy already has a cert from a previous
|
||||
# run (e.g. re-running this playbook after the first successful deploy) —
|
||||
# the path unit only fires on a CHANGE, so it won't pick up a cert that was
|
||||
# already sitting there unchanged before it started watching. Non-fatal if
|
||||
# nothing exists yet (first-ever run, before Caddy has issued anything).
|
||||
- name: Best-effort initial MQTT cert sync
|
||||
command: /opt/drb/sync-mqtt-cert.sh
|
||||
register: _initial_sync
|
||||
changed_when: "'copied cert' in _initial_sync.stdout"
|
||||
failed_when: false
|
||||
|
||||
- name: Log in to container registry
|
||||
command: >
|
||||
docker login {{ vault_registry_host }}
|
||||
|
||||
Reference in New Issue
Block a user