- Every configured node offline now degrades the Radio stage ("all N nodes
offline") so the wall no longer shows "Listening" with nothing up.
- nodes-hearing only counts configured nodes, so it can't exceed N/N.
- AI chip copy: "no incident writes in the recent window" instead of
"last incident write none in window".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The picker footer reads "Permissions checked N ago" from checked_at, but
voice_check stamped it with the response time. A denied result is cached
for 10 min, so a user who fixed the permission in Discord and reopened the
picker saw "checked 2s ago · Needs fix" from a stale snapshot. checked_at
now carries the oldest fetched_at behind the answer (now when there is none).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backend: /media/calls/{id}/audio now honours a single HTTP Range
(bytes=a-b, a-, -n) with 206 + Content-Range, 416 for unsatisfiable,
and a 200 full body (Accept-Ranges: bytes) otherwise. The body is
streamed from GCS in 512 KiB span reads (storage.audio_object_size +
download_audio_range) instead of downloading the whole object first,
so a 600 s FLAC starts after one span and seeks fetch only what they
need. HMAC check and Firestore lookup unchanged and still first.
Frontend: InlinePlayer moved to components/InlinePlayer.tsx and now
drives lib/audioQueue.ts, a useSyncExternalStore store over ONE audio
element (one call plays at a time). Wall spine rows get play buttons
and a "Play incident" queue (oldest first) with the playing row
highlighted; map path-stop popups get a play button and the playing
stop gets an accent ring; rail cards get "play latest". Playback is
gated by canPlayAudio() (BUSINESS_MODEL section 3 / Gate A hook).
Nothing plays until clicked. Spine truncation is now stated
("showing latest 30 of M", show-all toggle, count outside the loaded
window) for server-26#239.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The '● Live · Ns ago' pill was reset by any node/activeCalls/incident change,
so heartbeats kept it green while STT/correlation were dead (#196).
- lib/usePipelineHealth.ts: polls /health/ai (auth-free) every 60 s and
derives stage staleness from data too, since /health/ai resets on restart.
Radio = newest call; STT = calls heard after the newest transcript, dry when
>=3 are waiting and the oldest is > N min; AI = newest incident write /
linked call + any degraded /health/ai tier. N = NEXT_PUBLIC_WALL_STALE_MIN
(default 10); radio silence and AI absence get 3N (weaker evidence).
- components/live/TruthStrip.tsx: per-stage chips; full-width amber
DEGRADED bar naming the dry stage and any degraded tier.
- LiveView: heartbeat lastUpdated removed and no longer passed to MapView
(hides the old pill without touching MapView); 'Listening' only when every
stage is healthy; map + rail greyed when degraded; nodes KPI is now
'nodes hearing' (a call in the last N min).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- guild_overview offered an invite link to a full pool bot whenever it was
in ANY server (`if member` tested the non-empty guild_ids list, not
membership of this guild). Now: in this guild, or invitable.
- start_stream re-reads membership once on pool_full as it already does on
no_bot_in_guild, so a full pool bot an admin just invited by hand is found.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Node-page picker and the admin Add form now show, per voice channel, whether
a radio bot can stream there, from S3's voice-check routes:
- Badge per channel: Ready / Needs fix (names the missing permission) /
Couldn't check / Bots busy / No radio bot / Not a voice channel / Limit N.
- The chosen channel shows "<bot> can't Speak in #dispatch: <what removed it>"
plus c2-core's fix sentence, and an expandable per-bot list.
- Footer: "Permissions checked 40s ago · Re-check" (refresh=1, rate limited
server-side; a limited re-check says so). Invite links when no bot fits.
- Start is disabled only on a definite refusal (needs_fix, bad_channel);
unknown fails open. A channel this system already plays in is judged by
the bot that carries it, so Update/move is never blocked by the other bots.
- Admin Add form disables bots that are definitely denied in the bot list.
- 409 refusals (missing_permissions, preferred_busy, preferred_other_node,
bad_channel) are shown verbatim, multi-line.
Org owners (owner decision Q2) get "Radio bot per system" on
Settings > Organization: their own systems, their own BYO bots, set/clear
through S3's PUT /tokens/{id}/prefer/{system}. New GET /tokens/org/preferred
lists only the caller org's systems and bots (no pool bots, no token
strings; a pool bot an admin chose is flagged, not named).
The prefer route now drops the 5-minute doc_get_cached entry for every
system it rewrites, so a new preference applies to the next start.
Stacked on feat/225-s3-perm-check: must merge after S3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
start_stream now asks Discord (cached REST snapshots, S2) whether each bot
it could pick may View, Connect and Speak in the voice channel, before the
lease. Covers all four entry points: bot POST /discord/streams, web picker,
nodes discord_join, and the reconciler's re-place.
- pick_token: denied bots skipped; confirmed-ok sorts before unknown;
unknown is allowed (fail open); all denied -> missing_permissions naming
each bot, missing permission, trace reason and fix. The reuse branch
falls through when its token is denied.
- Bounded: discord_perm_check_concurrency at once, within
discord_perm_check_budget_seconds; unanswered = unknown.
- Non-voice channel -> 400 bad_channel (not missing_permissions).
- systems/{id}.preferred_token_id is now read (plan finding 1), as a hint.
- Pool bots at discord_pool_bot_guild_cap are never offered as invites;
pool_full when nothing else is left (owner Q3).
- handle_actual drops cached access on missing_permissions, channel_full,
not_in_guild, bad_channel.
- New GET /discord/nodes/{node_id}/guilds/{guild_id}/voice-check (org
member, names only) and GET /discord/guilds/{guild_id}/voice-check
(service key or admin).
- PUT /tokens/{id}/prefer/{system_id}: org owners for their own systems
and own BYO bots; shared systems and pool bots stay admin-only (owner Q2).
Keeps both pointers single-valued; audited.
- Kill switch: DISCORD_PERM_CHECK_ENABLED=false.
- tests/conftest.py: autouse fixture so no test ever reaches discord.com.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Firestore error inside classification (dropping membership, marking a
token invalid) propagated out of guild_snapshot and, behind asyncio.shield,
could die unretrieved. S3's start_stream must fail open, so the snapshot
now returns UNKNOWN (cached briefly) and logs only the exception type.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New app/internal/discord_rest.py: one shared httpx client with the
DiscordBot User-Agent, per-token rate-limit buckets (X-RateLimit-Bucket +
guild), one retry on a 429 of <=3s, global-429 pause per token, and a
process-wide breaker at 4000 401/403/unshared-429 responses per 10 min so
C2 can never get the shared egress IP banned. Tokens are never logged.
inspect_token / refresh_membership / leave_guild / TokenRejected move off
discord_leases onto it. /users/@me/guilds now follows after= pages (it read
one page of 200 and silently dropped the rest); a failed page is unknown,
never a truncated list.
guild_snapshot(token_doc, guild_id): cached, single-flight read of
/guilds/{g}, the bot's own member and /channels for S3's permission check.
401 marks the token invalid; 10004/10007 drop the server from guild_ids;
50001 is confirmed with one membership read and never changes guild_ids
on its own. TTLs: ok 120s, not-member/invalid 600s, unknown 30s.
config.py gets every setting the #225 slices use (perm-check kill switch,
TTLs, breaker budget, OAuth redirect URI, discord_pool_bot_guild_cap=90).
No behaviour change beyond pagination, User-Agent and the breaker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A guild without its @everyone role (or that role's permissions) was read
as base 0, a definite deny that would block a start. It now raises
ValueError, which S3 treats as unknown and lets the start go ahead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New pure module app/internal/discord_perms.py:
- effective_permissions(guild, member, channel) -> (bits, trace), in
Discord's order: owner, base (@everyone | member roles), ADMINISTRATOR,
@everyone overwrite, role overwrites (all denies then all allows),
member overwrite, no View Channel = 0, no Connect drops voice perms
(as discord.py does, so labels match the edge), timeout keeps only
View + Read History.
- trace names what removed each radio permission, e.g. "denied by
#dispatch's override for @Muted".
- check_voice(bits, channel, trace=None) -> {ok, is_voice, missing,
reasons, user_limit, limit_bypass}; only type-2 channels count; labels
match node-26#38's _join_refusal ("View Channel", "Connect", "Speak").
- Permission strings parsed with int (values above 2**53 tested);
malformed values raise ValueError so callers treat them as unknown.
No caller yet; S3 wires it into pick_token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-adding the same system to a voice channel returns the running stream
untouched (already_on). The form closed as if a stream had been added and
the new label/bot was silently dropped; it now stays open and explains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner-reported on live: adding a radio channel for a server + voice channel
that already had one (different label, system, even bot) overwrote the
existing stream and made its bot leave, because stream_id was guild:channel.
- New streams get guild:channel:<12 hex> ids (leases.new_stream_id); old
guild:channel docs keep working for read/stop/remove/node reports. Route
validators accept both (leases.STREAM_ID_PATTERN).
- start_stream: no stream_id = ADD. Same system already on in that channel
is idempotent (already_on, label/mode untouched); a stopped one of that
system is restarted; anything else is a new doc. stream_id = explicit
restart/edit (admin Start button, reconciler), incl. moving to another bot.
- preferred_token_id is strict: bot already in this server -> 409
preferred_busy naming the stream; on another node -> preferred_other_node;
unusable -> preferred_unavailable. No silent swap. pool_exhausted now says
one-bot-per-server and to add + invite another bot.
- One-token-per-guild was already enforced inside the lease transaction;
now covered by race tests. Per-guild asyncio lock serialises the
already-on / server-cap check with the create (guild half of #223).
- Stop by stream_id (POST /discord/guilds/{id}/stop {stream_id}); admin UI
Stop/Start/Remove act on the row's stream; node picker lists every stream
in a channel and adds beside other systems.
- Bot: /leave stream: (autocomplete) stops one; /leave channel: with several
streams asks which unless all:True; bare /leave still stops the server.
/status and /join name each stream's bot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /nodes/request-update-all (admin) does what Update Node does for each
approved node: flags update_requested_at for the HTTPS updater and nudges it
over MQTT. Pending and rejected nodes are skipped. Admin-only button on the
Nodes page reports which nodes weren't reachable over MQTT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DELETE /discord/guilds/{id} deletes channels one by one. If a channel was
restarted mid-removal, delete_stream raised "restarted" and the route
returned 409 with no audit entry, though the channels before it were
already deleted for good. The error now carries the channels already
removed and the route writes discord.guild.remove with outcome
"incomplete" before returning the 409. The server stays linked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- handle_actual logs every node-reported state TRANSITION (not heartbeats)
with stream/node/guild/channel/token ids and the node's error text: info
for connected/connecting/stopped, warning for error, and a dedicated
"radio bot X failed to log in on node N: <reason>" line for login_failed.
The node's reason code is now stored in actual.code and shown next to the
error on Settings > Discord and the node page.
- Remove a radio channel permanently: streams.delete_stream stops it through
desired state first (node republished without it, token -> releasing per
#216, freed on the node's ack), then deletes the doc in a transaction that
refuses if the channel was restarted in between. DELETE
/discord/streams/{id} (platform admin) and DELETE
/discord/nodes/{node}/streams/{id} (admin or org owner, same org + node
only; operators can still stop, not remove). Audited.
- Remove a server: DELETE /discord/guilds/{id} (admin) deletes every radio
channel in it the same way, then tombstones discord_guilds (org_id None +
unlinked_at, #218 semantics, plus removed_at) so it never falls back to
DISCORD_UNLINKED_GUILD_ORG_ID; the admin page hides it, and /drb link or the
admin link brings it back. Optional leave_radio_bots makes pool bots and the
server's org's own BYO bots leave via Discord REST; the DRB server bot stays
until kicked, and the UI says so. Audited.
- UI: Remove (with confirm) on Radio channels, Servers and the node page list.
Refs #213, #221
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- POST /discord/nodes/{id}/streams/stop: stream_id must be guild:channel
snowflakes, so a malformed id is a 422 instead of a Firestore path error.
- Offline footer said channels resume on reconnect; ad-hoc ones stop once
the node is gone past the lease grace (reconcile_once). Say so.
- Picker: say why Start is disabled when the node has no system.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #218 (server-26#213 item 1) unlinks a server by clearing org_id and
stamping unlinked_at, and resolve_guild_org then refuses the fallback org.
The overview showed such a server as "Unlinked · fallback org" and offered it
in Add radio channel, where start_stream would 403. Mirror the rule here
(forward-compatible: unlinked_at does not exist on main yet). Also key each
stream row by the resolved guild id, and drop panel copy that pointed at a
"Needed" figure the page doesn't show / misstated the invite permissions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The card only loads for owners/admins (the list endpoint is owner-only), so a
member saw "Loading…" forever. Show an owner-only note instead. Also assert the
one-server/two-org race really retried, as the PR body already claimed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frontend, built from design-canvas/discord-system-mockup.html ("DRB admin"):
- Settings > Discord (/settings/discord, platform admin only; the tab is
hidden from org owners because every route behind it is admin-only).
KPIs + tabs:
* Radio channels: every discord_streams doc, desired -> actual, the node's
real error, red after 2 min desired-on without connecting. Stop
(POST /discord/guilds/{id}/stop), Start for a stopped channel and
"+ Add radio channel" (POST /discord/streams), with the invite links a
"no bot in this server" refusal carries.
* Servers: linked servers plus servers seen running radio while unlinked
(acting for DISCORD_UNLINKED_GUILD_ORG_ID); Link / Change org
(PUT /discord/guilds/{id}/link).
* Radio bots: valid / check_error, guild membership with a per-server
invite link for each server the bot is missing from, lease (node,
since, releasing), fan-out, Refresh (POST /tokens/{id}/refresh),
Remove, Release all leases (POST /tokens/flush), add with optional BYO org.
* How it works.
- /tokens now renders the same radio-bot table (new token fields + refresh).
- Left out, no backend yet: alert routes, incident forum/roles/plan columns,
per-channel talkgroup chips, "Generate link code" (item 1).
Backend (routers/discord.py):
- GET /discord/admin/overview (platform admin): guilds, streams with node
and bot names, orgs, systems, limits. No token strings, no Discord user
ids, no linking-admin uid.
- PUT /discord/guilds/{id}/link: digits-only id, optional guild_name label,
and re-linking a server to a different org stops its live channels
(the reconciler would otherwise keep re-placing them on the old org's
nodes, using its stored org_id).
- POST /discord/streams: guild_id/channel_id must be digits (the web form
types them in; the bot always sent snowflakes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A server an org explicitly unlinked (unlinked_at) must not fall back to the
default org in the picker either, or the founding org could stream into it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core routers/discord_web.py:
PUT /discord/guilds/{id}/directory (service key): the server bot pushes each
server's name + voice channels; merged onto discord_guilds, never touches org_id.
GET /discord/nodes/{id}/join-options: the node org's servers (same
resolve rule as the bot), their voice channels with any live stream,
radio-bot readiness + invite links, and this node's streams. No tokens.
POST /discord/nodes/{id}/streams: stream pinned to the node; server must be
the node org's, channel must be in the bot's directory, guild + node caps.
POST /discord/nodes/{id}/streams/stop: one stream on this node, or all.
Control = platform admin, org owner, or operator owning the node.
- nodes.py: legacy raw-ID discord_join now refuses a server outside the node's org.
- discord bot: DirectorySync cog pushes on ready (15-min loop), join/leave,
rename, voice/category channel changes (debounced).
- frontend: NodeDiscordRadio panel + picker replaces DiscordJoinModal and the
admin-only Join/Leave buttons.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- c2-core: POST /discord/link-codes (org owner/admin) mints a single-use,
10-minute code; minting revokes the org's older codes. The bot redeems it
via POST /discord/guilds/{id}/link/redeem (service key) inside one
Firestore transaction, so two servers racing for one code cannot both win
and a server linked to org A refuses org B's code. GET/DELETE
/discord/linked-guilds lists/unlinks the caller's org's servers; unlink
stops that server's streams. Rate-limited (mint per user, redeem per
server and per Discord user); every mint/link/refusal/unlink audited.
- resolve_guild_org: an explicitly unlinked server no longer falls back to
DISCORD_UNLINKED_GUILD_ORG_ID. Linking away from the fallback org stops
the streams it had running for that org.
- Discord bot: /drb link code: (Manage Server, enforced at runtime too);
GatedTree answers every other command in an unlinked server with how to
link (autocomplete returns nothing). With the default fallback org set,
nothing is gated — unchanged behaviour.
- Frontend: "Linked Discord servers" card on Settings -> Organization —
generate code with countdown + copy, polls for the new link, unlink.
- 16 tests: single-use, two-server race, one-server/two-org race, expiry,
revoked/malformed codes, wrong-org takeover, idempotent relink, rate
limits, fallback stop, scoped list, unlink without fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /tokens and PUT /tokens/{id}/prefer became platform-admin only in #216
(server-26#204 item 6). Three frontend surfaces still offered them to
operators:
- systems page "Preferred Bot Token" panel: now rendered for admins only.
No non-admin listing endpoint was added: the write behind the picker is
admin-only too, so a list would only feed a control that 403s on save.
The panel also no longer turns a fetch error into "No tokens in pool."
- /tokens page: admin-only (operators are sent to /network).
- /network hub: the Bot Tokens card is shown to admins only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remote 'exit 0' came back as ssh exit 1 on the VM, and under bash -e
the step died before the SUPERSEDED check ran (runs 1139, 1144). Capture
the ssh status, check SUPERSEDED first, then fail on a non-zero status.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- reconcile_once only walked desired=on streams, so a token left "releasing"
by /leave while its node was offline (or a deleted node) never returned to
the pool until that node came back (server-26#204 leak shape). Sweep leased
tokens whose node is offline/deleted past the lease grace.
- publish_desired stamped release_seq AFTER building the list, so a stop
landing between the two got a seq whose list still held the stream; an ack
for it could free a token the node was still told to use. Stamp first.
Three regression tests; each fails on the previous commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tip check's git fetch ran before the disk-relief prune that #129 says
must be the first thing a deploy does; on a full disk the fetch could fail
and skip the prune. Prune only removes unreferenced images, so running it
on a run that then turns out superseded is harmless.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Feed/catalog error text (shown to every signed-in user via GET /cameras and
/traffic/511 status fields) carried httpx's full request URL, including
?key= for Ontario 511 and the 511NY DAA key. Query strings are now dropped.
- TfL and Calgary licences are per GEV only (TfL's terms page 403'd), the same
footing as Ontario 511, so they are UNVERIFIED (acknowledgement required)
instead of ungated attribution_required. Attribution still renders.
- The map camera layer clamps its bbox to +/-180/+/-90; zoomed-out bounds
produced a 422 ("could not reach server").
Refs server-26#198
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c2-core
- discord_streams/discord_leases wired in: POST /discord/streams,
POST /discord/guilds/{id}/stop, GET /discord/guilds/{id}/overview,
GET /discord/streams/{id}, PUT /discord/guilds/{id}/link (admin),
GET /discord/node/{node}/tokens/{id} (node key, lease holder only).
- Retained nodes/{id}/discord/desired (token ids, never token strings),
nodes/+/discord/actual subscribed and handled; per-node publish lock keeps
the retained list the newest; failed publishes retried by the reconciler.
- Tokens freed only after the node acks logout (or offline past
discord_lease_offline_release_seconds) - sweeper and C2 startup no longer
release leases.
- /nodes/{id}/command discord_join/leave map onto streams; CommandPayload
keeps preferred_token_id. /tokens: admin-only list, validate on add against
Discord, refresh, flush via transactional force release.
- Guild -> org via discord_guilds, unlinked guilds fall back to
DISCORD_UNLINKED_GUILD_ORG_ID (default "founding").
- tests/fake_firestore.py: optimistic-transaction fake; 27 lease/race tests.
discord bot
- /join, /leave, /status, autocomplete scoped to the calling guild; /leave
never touches another server; Manage Channels default permission;
/join edits its reply with the node's real result; /help copy fixed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two merges minutes apart ran two "Deploy to VM" jobs on the VM at once
(616/617, 652/656): racing compose pull/up, one run's prune breaking the
other's retry, health checks judging the wrong containers.
- deploy job: concurrency group drb-prod-deploy, cancel-in-progress false
(Gitea >= 1.26 honours it, go-gitea/gitea#32751; server is 1.27.3).
- Deploy step exits clean, before touching the VM, when its SHA is no
longer origin/main's tip; Health check is skipped for a superseded run.
Stops an older run landing after a newer one and rolling prod back.
- deploy-firestore-rules: its own group plus the same tip check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
logan
Logan CusanoandClaude Opus 5.5