Bot-token pool leaks and races: preferred_token_id silently dropped, re-join double-assigns, failed MQTT send never releases #204

Closed
opened 2026-10-04 15:34:20 -04:00 by logan · 2 comments
Owner

Found during the Discord system design audit (2026-10-04, design-canvas/discord-system-notes.md).

drb-c2-core/app/routers/tokens.py + routers/nodes.py:95-127:

  1. preferred_token_id is dropped before it is read. CommandPayload (models.py:111-114) declares only action/guild_id/channel_id; pydantic ignores extras, so payload.pop("preferred_token_id") at nodes.py:111 is always None. The Discord bot's /join token: option (radio.py:43-77) has never worked. Only the system-level preferred_token_id path is live.
  2. Re-join double-assigns. A second discord_join to a node that already holds a token calls assign_token again without releasing the first. The edge swaps to the new token (discord_radio._start_bot -> stop()), the old token stays in_use=True, assigned_node_id=<node> until a discord_leave or the offline sweeper. Repeated /joins drain the pool.
  3. Assigned, never delivered. If mqtt_handler.send_command fails (nodes.py:125) the endpoint 503s after assign_token already marked the token in use. Never released.
  4. Not atomic. assign_token is a query-then-update outside a transaction (tokens.py:116-138); two concurrent joins can receive the same token, which then runs two gateway sessions on two nodes.
  5. No validation on add. POST /tokens stores any string. A token whose application lacks the Message Content privileged intent (the edge requests it, discord_radio.py:213) fails at bot.start(), times out after 15s, and the user sees nothing.
  6. GET /tokens (L21) has no per-route dependency, so any Firebase user of any org (viewer included) lists the platform pool. Masked, but tokens carry no org_id at all.

Fix direction: add preferred_token_id to CommandPayload; make assign a Firestore transaction keyed on node (re-join reuses the node's existing token); release on send failure; validate a token on add via GET https://discord.com/api/v10/users/@me + applications/@me flags; admin-only list. Related: node-26 edge session issue (release must also stop the edge gateway session).

Found during the Discord system design audit (2026-10-04, `design-canvas/discord-system-notes.md`). `drb-c2-core/app/routers/tokens.py` + `routers/nodes.py:95-127`: 1. **`preferred_token_id` is dropped before it is read.** `CommandPayload` (`models.py:111-114`) declares only `action/guild_id/channel_id`; pydantic ignores extras, so `payload.pop("preferred_token_id")` at `nodes.py:111` is always `None`. The Discord bot's `/join token:` option (`radio.py:43-77`) has never worked. Only the system-level `preferred_token_id` path is live. 2. **Re-join double-assigns.** A second `discord_join` to a node that already holds a token calls `assign_token` again without releasing the first. The edge swaps to the new token (`discord_radio._start_bot` -> `stop()`), the old token stays `in_use=True, assigned_node_id=<node>` until a `discord_leave` or the offline sweeper. Repeated `/join`s drain the pool. 3. **Assigned, never delivered.** If `mqtt_handler.send_command` fails (`nodes.py:125`) the endpoint 503s after `assign_token` already marked the token in use. Never released. 4. **Not atomic.** `assign_token` is a query-then-update outside a transaction (`tokens.py:116-138`); two concurrent joins can receive the same token, which then runs two gateway sessions on two nodes. 5. **No validation on add.** `POST /tokens` stores any string. A token whose application lacks the Message Content privileged intent (the edge requests it, `discord_radio.py:213`) fails at `bot.start()`, times out after 15s, and the user sees nothing. 6. `GET /tokens` (L21) has no per-route dependency, so any Firebase user of any org (viewer included) lists the platform pool. Masked, but tokens carry no `org_id` at all. **Fix direction:** add `preferred_token_id` to `CommandPayload`; make assign a Firestore transaction keyed on node (re-join reuses the node's existing token); release on send failure; validate a token on add via `GET https://discord.com/api/v10/users/@me` + `applications/@me` flags; admin-only list. Related: node-26 edge session issue (release must also stop the edge gateway session).
Author
Owner

PR: #216 (server) + logan/node-26#32 (edge). Merge both together. Follow-ups filed: server-26#213 (guild link UI/admin page), server-26#214 (ad-hoc idle dismiss). drb-audio-review still to run on the edge PR before merge.

PR: https://git.vpn.cusano.net/logan/server-26/pulls/216 (server) + https://git.vpn.cusano.net/logan/node-26/pulls/32 (edge). Merge both together. Follow-ups filed: server-26#213 (guild link UI/admin page), server-26#214 (ad-hoc idle dismiss). drb-audio-review still to run on the edge PR before merge.
logan added the area/discord
kind
behavior-bug
priority
p2
type
bug
labels 2026-10-05 02:05:02 -04:00
Author
Owner

Fixed by PR #216 (merge 99407c1). Follow-up fix 03ba7ea added a sweep that frees leases on nodes that are offline or deleted. Live in prod.

Fixed by PR #216 (merge 99407c1). Follow-up fix 03ba7ea added a sweep that frees leases on nodes that are offline or deleted. Live in prod.
logan closed this issue 2026-10-05 02:08:56 -04:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#204