diff --git a/README.md b/README.md index f6b2f9a..af17b71 100644 --- a/README.md +++ b/README.md @@ -47,15 +47,21 @@ Server/ │ │ ├── routers/ │ │ │ ├── nodes.py # Node CRUD, approve/reject, command dispatch, system assignment │ │ │ ├── systems.py # Radio system CRUD (P25/DMR/NBFM configs) -│ │ │ ├── calls.py # Call log retrieval +│ │ │ ├── calls.py # Call log retrieval (read-only — calls created by MQTT handler) │ │ │ ├── tokens.py # Discord bot token pool — add/delete/list; assign_token/release_token helpers -│ │ │ └── upload.py # Audio file upload endpoint (called by edge nodes) +│ │ │ ├── upload.py # Audio file upload endpoint (called by edge nodes) +│ │ │ ├── incidents.py # [PLANNED] Incident CRUD, manual link/unlink calls, resolve +│ │ │ └── alerts.py # [PLANNED] Alert rule CRUD │ │ └── internal/ │ │ ├── auth.py # Auth — Firebase ID token OR service key (Bearer) │ │ ├── firestore.py # Async Firestore wrappers (doc_get, doc_set, doc_update, collection_list) -│ │ ├── mqtt_handler.py # MQTT publisher — commands, config pushes, API key provisioning +│ │ ├── mqtt_handler.py # MQTT subscriber — call_start/call_end handlers, node checkin/status │ │ ├── node_sweeper.py # Background task — marks nodes offline after 90s without heartbeat -│ │ └── storage.py # GCS audio file uploads +│ │ ├── storage.py # GCS audio file uploads +│ │ ├── transcription.py # [PLANNED] STT pipeline — Whisper or Google Speech API +│ │ ├── intelligence.py # [PLANNED] Keyword/entity extraction, severity scoring +│ │ ├── incident_correlator.py # [PLANNED] Match calls to incidents or create new ones +│ │ └── alerter.py # [PLANNED] Alert dispatch (Discord webhook, push, etc.) │ ├── scripts/ │ │ └── set_admin.py # CLI to grant/revoke Firebase admin custom claim │ ├── gcp-key.json # GCP service account key — place here, NOT committed to git @@ -74,11 +80,13 @@ Server/ └── drb-frontend/ # Next.js 14 admin dashboard ├── app/ │ ├── dashboard/ # Overview: active node grid + live call feed - │ ├── map/ # Leaflet map — node locations, status colors, active call popups - │ ├── calls/ # Full call history with duration, audio playback + │ ├── map/ # Leaflet map — node locations + [PLANNED] incident pins + │ ├── calls/ # Call history — talkgroup, duration, audio, transcript (when populated) │ ├── nodes/ # Node list + per-node detail (approve, reject, assign system) - │ ├── systems/ # Radio system CRUD + │ ├── systems/ # Radio system CRUD (P25 form with talkgroup editor, DMR/NBFM JSON) │ ├── tokens/ # Discord bot token pool management + │ ├── incidents/ # [PLANNED] Incident list/detail — linked calls, location, resolve + │ ├── alerts/ # [PLANNED] Alert rule configuration │ └── login/ # Firebase Auth login page ├── components/ │ ├── MapView.tsx # react-leaflet map with status-colored markers @@ -208,16 +216,141 @@ Edge nodes join Discord voice channels using bot tokens managed by the server. A 5. Node sends `online` / `recording` / `unconfigured` status via 30s MQTT heartbeats 6. `node_sweeper` background task marks any node offline after 90s without a heartbeat +## Call & Intelligence Pipeline + +This is the full intended data lifecycle for every radio call — from raw RF to searchable, cross-referenced intelligence. The data models and Firestore schema are already designed for this; several pipeline stages are stubs awaiting implementation. + +### Call lifecycle (current — fully working) + +``` +Edge node ──► MQTT call_start ──► c2-core creates CallRecord (status: active) + │ talkgroup_id, talkgroup_name, freq, + │ node_id, system_id, started_at + ▼ + Firestore "calls" collection + │ + ▼ + Frontend live call feed / map popups + │ +Edge node ──► MQTT call_end ──► c2-core updates CallRecord (status: ended) + │ ended_at, audio_url (GCS link) + ▼ + Frontend call history / audio playback +``` + +### Intelligence pipeline (designed — implementation pending) + +After a call ends, the following stages should fire in order: + +``` +CallRecord (ended, audio_url set) + │ + ▼ +[1] TRANSCRIPTION + Speech-to-text on the GCS audio file (Whisper or Google Speech-to-Text) + → writes CallRecord.transcript + + │ + ▼ +[2] INTELLIGENCE EXTRACTION + Analyze transcript for: + - Named entities: unit IDs, street addresses, location references + - Keywords / keyword sets: fire/EMS/police/hazmat/pursuit/shots fired/etc. + - Radio codes (10-codes, signals) mapped to plain English + - Severity scoring + → writes CallRecord.tags[], CallRecord.location (geocoded if address found) + + │ + ▼ +[3] INCIDENT CORRELATION + Given the extracted entities + tags, decide: + - Does this call match an existing active IncidentRecord? + (same location ± radius, overlapping tags, recent time window) + → link: append CallRecord.call_id to IncidentRecord.call_ids, + set CallRecord.incident_id, update IncidentRecord.summary + - Or does this call describe a new event? + → create IncidentRecord (type, location, title, tags, status: active) + link the call, set CallRecord.incident_id + + │ + ▼ +[4] ALERTS + If incident is new OR severity exceeds threshold: + - Trigger configured alert channels (Discord webhook, push notification, etc.) + - Include: incident type, location, talkgroup, transcript excerpt +``` + +### Data model fields involved + +**`CallRecord`** (Firestore collection: `calls`): + +| Field | Type | Populated by | +|---|---|---| +| `call_id` | string | MQTT handler on call_start | +| `node_id` | string | MQTT handler | +| `system_id` | string | MQTT handler (from node's assigned system) | +| `talkgroup_id` | number | MQTT handler | +| `talkgroup_name` | string | MQTT handler | +| `freq` | number | MQTT handler | +| `started_at` | timestamp | MQTT handler | +| `ended_at` | timestamp | MQTT handler on call_end | +| `status` | `active` \| `ended` | MQTT handler | +| `audio_url` | string | Edge node upload → GCS | +| `transcript` | string \| null | **[stub]** Transcription pipeline | +| `incident_id` | string \| null | **[stub]** Incident correlation | +| `location` | `{lat, lng}` \| null | **[stub]** Geocoding from transcript | +| `tags` | string[] | **[stub]** Intelligence extraction | + +**`IncidentRecord`** (Firestore collection: `incidents`): + +| Field | Type | Description | +|---|---|---| +| `incident_id` | string | UUID | +| `title` | string | Auto-generated or manually set | +| `type` | string | `fire`, `ems`, `police`, `hazmat`, `pursuit`, etc. | +| `status` | `active` \| `resolved` | Updated as calls accumulate or manually resolved | +| `location` | `{lat, lng}` | Geocoded from first call with a location | +| `call_ids` | string[] | All linked CallRecord IDs | +| `started_at` | timestamp | Timestamp of first linked call | +| `updated_at` | timestamp | Updated on each new linked call | +| `summary` | string \| null | Auto-generated from transcripts or manually written | +| `tags` | string[] | Union of tags from all linked calls | + +### Backend — what needs to be built + +The following do **not exist yet** and need to be created: + +- `drb-c2-core/app/routers/incidents.py` — CRUD + manual incident management +- `drb-c2-core/app/routers/alerts.py` — Alert rule configuration +- `drb-c2-core/app/internal/transcription.py` — STT integration (Whisper local or Google Speech API) +- `drb-c2-core/app/internal/intelligence.py` — Keyword/entity extraction, severity scoring +- `drb-c2-core/app/internal/incident_correlator.py` — Match calls to incidents or create new ones +- `drb-c2-core/app/internal/alerter.py` — Dispatch alerts (Discord webhook, etc.) + +The `_on_call_end()` handler in `mqtt_handler.py` is the natural trigger point — after updating the CallRecord, it should enqueue the transcription + intelligence pipeline. + +### Frontend — what needs to be built + +The following pages and nav items are not yet implemented: + +- `/incidents` — Incident list and detail view (linked calls, map pin, transcript summary, tags, resolve button) +- `/alerts` — Alert rule configuration (keyword sets, talkgroup filters, notification channels) +- `Nav.tsx` should add **Calls** and **Incidents** as primary nav items; the current Calls page exists but isn't in the design's intended nav hierarchy + +The **Map** page should eventually show both node markers and incident pins — incidents with `location` set should appear as color-coded markers (by type/severity) alongside the node status markers. + ## Frontend Pages -| Page | URL | Description | -|---|---|---| -| Dashboard | `/dashboard` | Live node grid + active call stream | -| Map | `/map` | Leaflet map — nodes color-coded by status, click for active call details | -| Calls | `/calls` | Full call history — talkgroup, duration, audio playback | -| Nodes | `/nodes` | Node list; per-node page for approve/reject/assign system | -| Systems | `/systems` | Create and manage radio system configurations | -| Tokens | `/tokens` | Manage Discord bot token pool | +| Page | URL | Status | Description | +|---|---|---|---| +| Dashboard | `/dashboard` | Working | Live node grid + active call stream | +| Map | `/map` | Working | Leaflet map — nodes color-coded by status, active call popups | +| Calls | `/calls` | Working | Full call history — talkgroup, duration, audio playback, transcript (when populated) | +| Nodes | `/nodes` | Working | Node list; per-node detail for approve/reject/assign system | +| Systems | `/systems` | Working | Create and manage P25/DMR/NBFM radio system configurations | +| Tokens | `/tokens` | Working | Discord bot token pool management | +| Incidents | `/incidents` | **Not built** | Incident list/detail — linked calls, location, summary, tags, resolve | +| Alerts | `/alerts` | **Not built** | Alert rule configuration — keywords, talkgroups, notification channels | ## Makefile Targets