Bot-token pool leaks and races: preferred_token_id silently dropped, re-join double-assigns, failed MQTT send never releases #204
Closed
opened 2026-10-04 15:34:20 -04:00 by logan
·
2 comments
No Branch/Tag Specified
Labels
Clear labels
area/ai-pipeline
area/auth
area/billing
area/correlation
area/discord
area/frontend
area/governance
area/growth
area/infra-deploy
area/legal
area/map
area/mqtt
area/node-ota
area/osint
area/recording-audio
area/sdr-radio
area/transcription
board
board:agenda
minutes:draft
minutes:final
role:ceo
role:cfo
role:cio
role:ciso
role:cmo
role:coo
role:cto
Scene/entity extraction, summarizer, AI providers, embeddings
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
Plans, entitlements, AI spend accounting and caps
Incident correlation, linking, lifecycle, severity, corr_debug
Discord bot, radio bots, voice streams, bot tokens, webhooks
Web UI: Next.js app and the edge-node local dashboard
Board process, minutes, unattended runner, agent tooling
Go-to-market, outreach, cold email, discovery conversations
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
Live map: pins, paths, tiles, legend, geocoding
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
Node install, enrollment, OTA updater, image skew
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
STT, transcript correction, place verification, name accuracy
Board process: minutes and board-filed items
Needs board scoping at the next sitting; CEO removes the label once scoped
One exec pre-read, closed by CEO
CEO ruling; open until enacted
Filed by the CEO
Filed by the CFO
Filed by the CIO
Filed by the CISO
Filed by the CMO
Filed by the COO
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: logan/server-26#204
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found during the Discord system design audit (2026-10-04,
design-canvas/discord-system-notes.md).drb-c2-core/app/routers/tokens.py+routers/nodes.py:95-127:preferred_token_idis dropped before it is read.CommandPayload(models.py:111-114) declares onlyaction/guild_id/channel_id; pydantic ignores extras, sopayload.pop("preferred_token_id")atnodes.py:111is alwaysNone. The Discord bot's/join token:option (radio.py:43-77) has never worked. Only the system-levelpreferred_token_idpath is live.discord_jointo a node that already holds a token callsassign_tokenagain without releasing the first. The edge swaps to the new token (discord_radio._start_bot->stop()), the old token staysin_use=True, assigned_node_id=<node>until adiscord_leaveor the offline sweeper. Repeated/joins drain the pool.mqtt_handler.send_commandfails (nodes.py:125) the endpoint 503s afterassign_tokenalready marked the token in use. Never released.assign_tokenis a query-then-update outside a transaction (tokens.py:116-138); two concurrent joins can receive the same token, which then runs two gateway sessions on two nodes.POST /tokensstores any string. A token whose application lacks the Message Content privileged intent (the edge requests it,discord_radio.py:213) fails atbot.start(), times out after 15s, and the user sees nothing.GET /tokens(L21) has no per-route dependency, so any Firebase user of any org (viewer included) lists the platform pool. Masked, but tokens carry noorg_idat all.Fix direction: add
preferred_token_idtoCommandPayload; make assign a Firestore transaction keyed on node (re-join reuses the node's existing token); release on send failure; validate a token on add viaGET https://discord.com/api/v10/users/@me+applications/@meflags; admin-only list. Related: node-26 edge session issue (release must also stop the edge gateway session).PR: #216 (server) + logan/node-26#32 (edge). Merge both together. Follow-ups filed: server-26#213 (guild link UI/admin page), server-26#214 (ad-hoc idle dismiss). drb-audio-review still to run on the edge PR before merge.
Fixed by PR #216 (merge
99407c1). Follow-up fix03ba7eaadded a sweep that frees leases on nodes that are offline or deleted. Live in prod.