area_context v2 + Maps place verification (server-26#36, #37)
Build & Deploy / Build & push images (push) Successful in 4m9s
Build & Deploy / Deploy to VM (push) Successful in 1m54s
Build & Deploy / Report a failed deploy (push) Skipped

#36 — the correction pass shipped in 58efdbd was right, its reference-data
shape was not. One shape now, at both scopes, every field nullable:

  area_context: { municipality?, county?, state?,
                  center?, radius_km?, resolved_from?, resolved_at?,
                  local_knowledge?: [{term, meaning}] }

`state` closes the ambiguity that made "Ossining" a national guess.
`local_knowledge` replaces roads[]/landmarks[], which could not hold
intersections, schools or nicknames and carried no meanings — `11-X-ray` is
useless alone, `11-X-ray — MTA PD patrol unit` is what a corrector can act on.
Pre-#36 roads[]/landmarks[] are read forward as bare terms so nothing an
operator already entered is lost.

Nullability is the mechanism: which scope gets filled is the operator's
declaration of how homogeneous the system is. One town — fill it once at system
level. Statewide — leave it blank and fill each talkgroup.

The backend owns the derived anchor. PUT /systems/{id} merges config.talkgroups[]
against what is stored instead of writing the client's blob verbatim, which
would have erased the anchor and the pending queue — the same defect as the
ten_codes wipe.

#37 — Maps as a verifier, not as prompt stuffing. The corrector emits its
location nouns; each is geocoded against the talkgroup's anchor, and on a miss
we look for a sound-alike that does resolve there, correct to it, and propose
{term, meaning} to that talkgroup. Cost scales with location nouns, not calls.

No anchor means SKIP. An area too wide to discriminate stores no anchor at all,
because a statewide radius would confirm anything inside it — verification that
passes everything is worse than none, since it reads as a check in the data.

Also re-anchors _geocode_location, which rejected results >40km from the NODE
(server-26#6). An antenna is not a jurisdiction; distance-from-node was always
a stand-in for the anchor and is now only the fallback.

The induction loop proposes at talkgroup level and never promotes. Blast
radius: a wrong term on a channel misleads that channel, the same term
system-wide misleads one 400km away on a statewide system.

38 new tests; 240 pass. Frontend typechecks clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-08-23 16:43:59 -04:00
co-authored by Claude Opus 5
parent 58efdbd6eb
commit 964343c819
14 changed files with 2052 additions and 167 deletions
+544
View File
@@ -0,0 +1,544 @@
"""
Area context — the ground truth an operator sets about where a channel operates.
One shape, used at two scopes (server-26#36):
area_context: {
municipality?, county?, state?,
center?, radius_km?, resolved_from?, resolved_at?, # backend-written
local_knowledge?: [ { term, meaning } ]
}
WHY EVERY FIELD IS NULLABLE. The system level is only meaningful when it is true
of *every* talkgroup on that system. White Plains PD — it is, so an operator
fills it once and every talkgroup inherits. A statewide Colorado system — it is
not, so they leave it null and fill each talkgroup. Which level someone fills IS
their declaration of how homogeneous the system is, which is what lets one
schema serve both without a `system_type` flag to get out of sync.
WHY `local_knowledge` REPLACED `roads[]`/`landmarks[]`. Radio traffic references
intersections, schools, housing developments, rail stations and nicknames ("the
flats"), none of which fit two lists. And a bare term is half the information:
`11-X-ray` tells a corrector nothing, `11-X-ray — MTA PD patrol unit` is what
lets it recognise the sound.
WHY THE ANCHOR CAN BE ABSENT ON PURPOSE. `center`/`radius_km` exist so a
geocoded place name can be sanity-checked against the area the channel actually
covers (server-26#37). If municipality/county/state only resolve to something as
wide as a state, that check would confirm anything inside it while appearing to
work — worse than useless. So an anchor wider than
`settings.area_anchor_max_radius_km` is not written at all, and an absent anchor
means SKIP THE CHECK, never "accept anything".
THE CLIENT DOES NOT WRITE THE DERIVED FIELDS. `center`, `radius_km`,
`resolved_from` and `resolved_at` are computed here and merged in by the server.
Taking them from the request body is the same defect as the `ten_codes` wipe
fixed in 58efdbd: the frontend does not decide what is in a system document.
"""
import asyncio
import math
from datetime import datetime, timezone
from typing import Any, Optional
from app.config import settings
from app.internal.logger import logger
# Fields an operator sets. Anything else in an incoming body is dropped.
CLIENT_FIELDS = ("municipality", "county", "state", "local_knowledge")
# Fields this module owns. Carried forward from the stored document on every
# write, never read from the request.
SERVER_FIELDS = ("center", "radius_km", "resolved_from", "resolved_at")
# The three that identify a place, in the order they are geocoded.
PLACE_FIELDS = ("municipality", "county", "state")
_anchor_cache: dict[str, Optional[dict]] = {}
def geo_dist_km(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
"""Haversine distance in km between two lat/lon points."""
R = 6371.0
dlat = math.radians(lat2 - lat1)
dlon = math.radians(lon2 - lon1)
a = (
math.sin(dlat / 2) ** 2
+ math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * math.sin(dlon / 2) ** 2
)
return R * 2 * math.asin(math.sqrt(a))
# -- Normalisation -------------------------------------------------------------
def normalize_local_knowledge(raw: Any) -> list[dict]:
"""
Coerce whatever arrived into [{term, meaning}], dropping junk.
Accepts a bare string list too — that is what `roads[]`/`landmarks[]` and the
old flat `vocabulary` look like, and a term with no meaning is still worth
having in the reference list.
"""
if not isinstance(raw, list):
return []
out: list[dict] = []
seen: set[str] = set()
for item in raw:
if isinstance(item, str):
term, meaning = item.strip(), None
elif isinstance(item, dict):
term = str(item.get("term") or "").strip()
meaning = str(item.get("meaning") or "").strip() or None
else:
continue
key = term.lower()
if not term or key in seen:
continue
seen.add(key)
out.append({"term": term, "meaning": meaning} if meaning else {"term": term})
return out
def knowledge_of(area: Optional[dict]) -> list[dict]:
"""
This scope's local knowledge, folding the pre-#36 shape forward.
`roads[]` and `landmarks[]` were the original fields and real systems still
have them stored. Reading them as bare terms means the corrector keeps the
ground truth an operator already entered instead of silently losing it the
day this shipped; they disappear from the document the next time that scope
is saved.
"""
area = area or {}
legacy = list(area.get("roads") or []) + list(area.get("landmarks") or [])
return normalize_local_knowledge(list(area.get("local_knowledge") or []) + legacy)
def normalize(raw: Any) -> dict:
"""Client-supplied area_context -> the stored shape, server fields excluded."""
if not isinstance(raw, dict):
return {}
out: dict[str, Any] = {}
for field in PLACE_FIELDS:
value = raw.get(field)
if isinstance(value, str) and value.strip():
out[field] = value.strip()
knowledge = knowledge_of(raw)
if knowledge:
out["local_knowledge"] = knowledge
return out
def merge_server_fields(incoming: dict, existing: Optional[dict]) -> dict:
"""Carry the backend-owned anchor forward across a client write."""
out = dict(incoming)
for field in SERVER_FIELDS:
if existing and existing.get(field) is not None:
out[field] = existing[field]
return out
def merge_config(incoming: Any, existing: Optional[dict]) -> Any:
"""
Reconcile a client-sent config blob with what the server already owns.
The systems form sends `config.talkgroups[]` in full, so writing it verbatim
destroys everything the backend put there — the resolved anchor and the
pending term queue. That is the same defect as the `ten_codes` wipe fixed in
58efdbd, and the same fix applies: the backend merges its own fields back in
rather than taking dictation from the frontend.
"""
if not isinstance(incoming, dict):
return incoming
incoming_tgs = incoming.get("talkgroups")
if not isinstance(incoming_tgs, list):
return incoming
by_id: dict[int, dict] = {}
for tg in ((existing or {}).get("talkgroups") or []):
if isinstance(tg, dict):
try:
by_id[int(tg.get("id", -1))] = tg
except (TypeError, ValueError):
continue
merged: list[Any] = []
for tg in incoming_tgs:
if not isinstance(tg, dict):
merged.append(tg)
continue
try:
prior = by_id.get(int(tg.get("id", -1))) or {}
except (TypeError, ValueError):
prior = {}
out = dict(tg)
area = normalize(tg.get("area_context"))
prior_area = prior.get("area_context") or {}
if area:
out["area_context"] = merge_server_fields(area, prior_area)
else:
out.pop("area_context", None)
if prior.get(PENDING_KEY):
out[PENDING_KEY] = prior[PENDING_KEY]
merged.append(out)
return {**incoming, "talkgroups": merged}
# -- Scope resolution ----------------------------------------------------------
def effective(system_area: Optional[dict], tg_area: Optional[dict]) -> dict:
"""
Merge the two scopes: talkgroup wins where set, system fills the gaps.
`local_knowledge` concatenates rather than replaces, talkgroup entries first
so they survive any downstream truncation and outrank a system entry for the
same term. A multi-county system whose talkgroup covers one municipality must
not have that municipality's terms buried under a county-wide list.
"""
system_area = system_area or {}
tg_area = tg_area or {}
out: dict[str, Any] = {}
for field in PLACE_FIELDS:
value = tg_area.get(field) or system_area.get(field)
if value:
out[field] = value
knowledge = normalize_local_knowledge(knowledge_of(tg_area) + knowledge_of(system_area))
if knowledge:
out["local_knowledge"] = knowledge
return out
def talkgroup_entry(system_doc: Optional[dict], talkgroup_id: Any) -> dict:
"""The `config.talkgroups[]` entry for this talkgroup, or `{}`."""
if not system_doc or talkgroup_id is None:
return {}
talkgroups = (system_doc.get("config") or {}).get("talkgroups") or []
idx = _tg_index(talkgroups, talkgroup_id)
return talkgroups[idx] if idx >= 0 else {}
def anchor_key(area: Optional[dict]) -> str:
"""
Stable identity of the place an anchor was resolved from.
Stored as `resolved_from`, which is what makes "did this actually change?"
decidable — so the geocode happens when someone edits a town name, not on
every read or every five minutes.
"""
area = area or {}
return "|".join((area.get(f) or "").strip().lower() for f in PLACE_FIELDS)
def has_place(area: Optional[dict]) -> bool:
return bool(anchor_key(area).strip("|"))
def anchor_for(system_area: Optional[dict], tg_area: Optional[dict]) -> Optional[dict]:
"""
The anchor to sanity-check geocoded locations against, or None.
None has one meaning and it is load-bearing: SKIP THE CHECK. It covers an
unconfigured system, an area too wide to discriminate, and a stored anchor
whose `resolved_from` no longer matches the place it was computed for (an
edit landed and the refresh has not run). Accepting a stale or oversized
anchor would rubber-stamp locations while looking like verification.
"""
key = anchor_key(effective(system_area, tg_area))
for area in (tg_area, system_area):
if not area:
continue
center, radius = area.get("center"), area.get("radius_km")
if area.get("resolved_from") == key and center and radius:
try:
return {
"lat": float(center["lat"]),
"lng": float(center["lng"]),
"radius_km": float(radius),
}
except (KeyError, TypeError, ValueError):
continue
return None
# -- Anchor geocoding ----------------------------------------------------------
def _query(area: dict) -> str:
return ", ".join(area[f] for f in PLACE_FIELDS if area.get(f))
async def resolve_anchor(area: dict) -> Optional[dict]:
"""
Geocode municipality/county/state into {center, radius_km}, or None.
The radius comes from the result's own viewport — half its diagonal — so a
village anchors tightly and a county loosely, which is the real difference
we care about. Anything wider than `area_anchor_max_radius_km` is discarded
rather than stored: see the module docstring.
"""
if not has_place(area):
return None
query = _query(area)
if query in _anchor_cache:
return _anchor_cache[query]
if not settings.google_maps_api_key:
logger.warning("GOOGLE_MAPS_API_KEY not set — area anchors cannot be resolved")
return None
import httpx
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(
"https://maps.googleapis.com/maps/api/geocode/json",
params={"address": query, "region": "us", "key": settings.google_maps_api_key},
)
r.raise_for_status()
data = r.json()
if data.get("status") != "OK" or not data.get("results"):
logger.warning(f"Area anchor: {query!r} did not geocode ({data.get('status')})")
_anchor_cache[query] = None
return None
geometry = data["results"][0].get("geometry") or {}
loc = geometry.get("location") or {}
lat, lng = float(loc["lat"]), float(loc["lng"])
viewport = geometry.get("viewport") or {}
ne, sw = viewport.get("northeast"), viewport.get("southwest")
if ne and sw:
radius_km = geo_dist_km(sw["lat"], sw["lng"], ne["lat"], ne["lng"]) / 2
else:
radius_km = settings.geocode_max_km
except Exception as e:
logger.warning(f"Area anchor geocoding failed for {query!r}: {e}")
return None # not cached — a transient failure should be retried
if radius_km > settings.area_anchor_max_radius_km:
logger.info(
f"Area anchor: {query!r} spans ~{radius_km:.0f}km, wider than "
f"area_anchor_max_radius_km={settings.area_anchor_max_radius_km} — storing no "
f"anchor, so verification skips rather than rubber-stamps"
)
_anchor_cache[query] = None
return None
anchor = {
"center": {"lat": lat, "lng": lng},
"radius_km": round(radius_km, 2),
"resolved_from": anchor_key(area),
"resolved_at": datetime.now(timezone.utc).isoformat(),
}
_anchor_cache[query] = anchor
logger.info(f"Area anchor: {query!r} -> ({lat:.4f}, {lng:.4f}) r={radius_km:.1f}km")
return anchor
def _apply(area: dict, anchor: Optional[dict], key: str) -> dict:
"""Write (or clear) the derived fields on one scope's area_context."""
out = {k: v for k, v in area.items() if k not in SERVER_FIELDS}
if anchor:
out.update(anchor)
elif key.strip("|"):
# A place is set but produced no usable anchor. Record that we tried, so
# the next write does not geocode it again for the same answer.
out["resolved_from"] = key
out["resolved_at"] = datetime.now(timezone.utc).isoformat()
return out
async def refresh_anchors(system_doc: dict) -> dict:
"""
Recompute anchors for a system and every talkgroup that sets a place.
Returns a Firestore patch — `{}` when nothing needed resolving. Talkgroups
are refreshed alongside the system because a talkgroup's anchor is derived
from its EFFECTIVE place (its own fields over the system's), so editing the
system's county silently changes what every talkgroup should be anchored to.
Only scopes whose `resolved_from` no longer matches are geocoded, and the
per-query cache means N talkgroups in one town cost one request.
"""
system_area = dict(system_doc.get("area_context") or {})
patch: dict[str, Any] = {}
system_key = anchor_key(system_area)
if system_area.get("resolved_from") != system_key:
anchor = await resolve_anchor(system_area) if has_place(system_area) else None
patch["area_context"] = _apply(system_area, anchor, system_key)
system_area = patch["area_context"]
config = system_doc.get("config") or {}
talkgroups = config.get("talkgroups")
if not isinstance(talkgroups, list):
return patch
updated: list[dict] = []
changed = False
for tg in talkgroups:
if not isinstance(tg, dict):
updated.append(tg)
continue
tg_area = tg.get("area_context") or {}
# No place of its own means it inherits the system's anchor wholesale —
# nothing to store here, and anchor_for() falls back to the system.
if not has_place(tg_area):
if any(tg_area.get(f) is not None for f in SERVER_FIELDS):
tg = {**tg, "area_context": {k: v for k, v in tg_area.items() if k not in SERVER_FIELDS}}
changed = True
updated.append(tg)
continue
key = anchor_key(effective(system_area, tg_area))
if tg_area.get("resolved_from") == key:
updated.append(tg)
continue
anchor = await resolve_anchor(effective(system_area, tg_area))
updated.append({**tg, "area_context": _apply(tg_area, anchor, key)})
changed = True
if changed:
patch["config"] = {**config, "talkgroups": updated}
return patch
# -- Talkgroup-level pending terms ---------------------------------------------
#
# Proposals land on the TALKGROUP and are never promoted to the system
# automatically (server-26#37). The argument is blast radius: a wrong term on a
# talkgroup misleads one channel, the same term at system level misleads every
# channel on that system — including one 400km away on a statewide system, which
# is exactly the context poisoning the scope rule exists to prevent. If a term
# genuinely applies system-wide, carrying it on several talkgroups costs almost
# nothing; auto-promoting a wrong one is expensive to notice.
PENDING_KEY = "local_knowledge_pending"
def _tg_index(talkgroups: list, talkgroup_id: Any) -> int:
try:
wanted = int(talkgroup_id)
except (TypeError, ValueError):
return -1
for i, tg in enumerate(talkgroups):
if not isinstance(tg, dict):
continue
try:
if int(tg.get("id", -1)) == wanted:
return i
except (TypeError, ValueError):
continue
return -1
def _known_terms(tg: dict, system_doc: dict) -> set[str]:
"""Everything this talkgroup already knows, at either scope, plus pending."""
known = {
e["term"].lower()
for e in effective(system_doc.get("area_context"), tg.get("area_context"))
.get("local_knowledge", [])
}
known |= {str(t).lower() for t in (tg.get("vocabulary") or [])}
known |= {str(t).lower() for t in (system_doc.get("vocabulary") or [])}
known |= {str(p.get("term", "")).lower() for p in (tg.get(PENDING_KEY) or [])}
return known
async def add_pending(system_id: str, talkgroup_id: Any, entries: list[dict]) -> int:
"""
Queue proposed {term, meaning} entries on one talkgroup for human review.
Returns how many were actually queued. Nothing here writes to
`local_knowledge` — approval is a person's decision, always.
"""
from app.internal import firestore as fstore
if not system_id or talkgroup_id is None or not entries:
return 0
system_doc = await fstore.doc_get("systems", system_id)
if not system_doc:
return 0
config = dict(system_doc.get("config") or {})
talkgroups = list(config.get("talkgroups") or [])
idx = _tg_index(talkgroups, talkgroup_id)
if idx < 0:
return 0
tg = dict(talkgroups[idx])
known = _known_terms(tg, system_doc)
now = datetime.now(timezone.utc).isoformat()
queued: list[dict] = []
for entry in entries:
term = str(entry.get("term") or "").strip()
if not term or term.lower() in known:
continue
known.add(term.lower())
queued.append({
"term": term,
"meaning": entry.get("meaning") or None,
"source": entry.get("source") or "verifier",
"added_at": now,
"source_call_ids": entry.get("source_call_ids") or [],
})
if not queued:
return 0
tg[PENDING_KEY] = list(tg.get(PENDING_KEY) or []) + queued
talkgroups[idx] = tg
config["talkgroups"] = talkgroups
await fstore.doc_update("systems", system_id, {"config": config})
logger.info(
f"Local knowledge: {len(queued)} term(s) proposed for talkgroup "
f"{talkgroup_id} on system {system_id}: {[q['term'] for q in queued]}"
)
return len(queued)
async def resolve_pending(system_id: str, talkgroup_id: Any, term: str, approve: bool) -> bool:
"""Approve a pending term onto the talkgroup, or dismiss it. Never promotes."""
from app.internal import firestore as fstore
system_doc = await fstore.doc_get("systems", system_id)
if not system_doc:
return False
config = dict(system_doc.get("config") or {})
talkgroups = list(config.get("talkgroups") or [])
idx = _tg_index(talkgroups, talkgroup_id)
if idx < 0:
return False
tg = dict(talkgroups[idx])
pending = list(tg.get(PENDING_KEY) or [])
match = next((p for p in pending if str(p.get("term", "")).lower() == term.lower()), None)
if match is None:
return False
tg[PENDING_KEY] = [p for p in pending if p is not match]
if approve:
area = dict(tg.get("area_context") or {})
area["local_knowledge"] = normalize_local_knowledge(
list(area.get("local_knowledge") or [])
+ [{"term": match["term"], "meaning": match.get("meaning")}]
)
tg["area_context"] = area
talkgroups[idx] = tg
config["talkgroups"] = talkgroups
await fstore.doc_update("systems", system_id, {"config": config})
return True
async def refresh_anchors_bg(system_id: str) -> None:
"""Fire-and-forget refresh, for callers that must not block on Maps."""
from app.internal import firestore as fstore
try:
doc = await fstore.doc_get("systems", system_id)
if not doc:
return
patch = await refresh_anchors(doc)
if patch:
await fstore.doc_update("systems", system_id, patch)
except Exception as e:
logger.warning(f"Area anchor refresh failed for system {system_id}: {e}")
def schedule_refresh(system_id: str) -> None:
"""Kick a refresh without making the caller wait for the geocoder."""
try:
asyncio.get_running_loop().create_task(refresh_anchors_bg(system_id))
except RuntimeError: # no loop (tests, scripts) — nothing to schedule
pass
+75 -22
View File
@@ -15,6 +15,7 @@ import re
from typing import Optional
from app.internal.logger import logger
from app.internal import firestore as fstore
from app.internal import area_context
# Location validity is defined once, by the module that owns the incident's
# location/pin invariant. incident_correlator does not import this module, so
# this is not a cycle.
@@ -225,6 +226,20 @@ async def extract_scenes(
node_lat = node_doc.get("lat")
node_lon = node_doc.get("lon")
# The talkgroup's own anchor and place, when an operator has described it
# (server-26#36). This is what "where is this channel" should mean; the node
# position below is only the fallback for a system nobody has described.
tg_anchor: Optional[dict] = None
tg_area: dict = {}
if system_id:
system_doc = await fstore.doc_get_cached("systems", system_id)
if system_doc:
system_area = system_doc.get("area_context") or {}
tg_entry = area_context.talkgroup_entry(system_doc, talkgroup_id)
own_area = tg_entry.get("area_context") or {}
tg_area = area_context.effective(system_area, own_area)
tg_anchor = area_context.anchor_for(system_area, own_area)
processed: list[dict] = []
for scene in raw_scenes:
tags: list[str] = scene.get("tags") or []
@@ -273,20 +288,27 @@ async def extract_scenes(
# Build the most specific query possible: location + municipality + state.
# e.g. "High Street" → "High Street, Yorktown, New York"
# This prevents generic street names from resolving to wrong-country results.
#
# Prefer the place an operator actually set over the one guessed from
# the talkgroup's name and the node's reverse-geocoded position. A name
# like "Ossining PD" gives a municipality with no state behind it, which
# is how a generic street name ends up resolving in the wrong half of
# the country.
location_coords: Optional[dict] = None
if location and node_lat is not None and node_lon is not None:
muni = _municipality_from_tg(talkgroup_name)
state = await _get_node_state(node_id or "", node_lat, node_lon) if node_id else ""
county = _node_county_cache.get(node_id or "") if node_id else ""
parts = [location]
if muni:
parts.append(muni)
if county:
parts.append(county)
if state:
parts.append(state)
if location:
parts = [location]
if tg_area.get("municipality") or tg_area.get("county") or tg_area.get("state"):
parts += [tg_area[f] for f in area_context.PLACE_FIELDS if tg_area.get(f)]
elif node_lat is not None and node_lon is not None:
muni = _municipality_from_tg(talkgroup_name)
state = await _get_node_state(node_id or "", node_lat, node_lon) if node_id else ""
county = _node_county_cache.get(node_id or "") if node_id else ""
parts += [p for p in (muni, county, state) if p]
query = ", ".join(parts)
location_coords = await _geocode_location(query, node_lat, node_lon)
if tg_anchor or (node_lat is not None and node_lon is not None):
location_coords = await _geocode_location(
query, node_lat, node_lon, anchor=tg_anchor
)
# Embed this scene's content
scene_text = _build_scene_embed_text(
@@ -411,12 +433,25 @@ async def _get_node_state(node_id: str, lat: float, lon: float) -> str:
async def _geocode_location(
location_str: str, node_lat: float, node_lon: float
location_str: str,
node_lat: Optional[float] = None,
node_lon: Optional[float] = None,
anchor: Optional[dict] = None,
) -> Optional[dict]:
"""
Geocode using Google Maps Geocoding API, biased toward the node's area.
Returns {"lat": float, "lng": float} or None if geocoding fails or the
result is farther than geocode_max_km from the node (wrong-jurisdiction guard).
Geocode using Google Maps Geocoding API, biased toward the channel's area.
Returns {"lat": float, "lng": float}, or None if geocoding fails or the
result lands outside the area this channel covers.
THE REFERENCE POINT IS THE TALKGROUP, NOT THE NODE (server-26#6 / #37). This
used to reject anything more than geocode_max_km (40km) from the receiving
node, which conflates an antenna with a jurisdiction: a system can span a
county or several, so a node legitimately sits far from the area a talkgroup
covers, and real dispatch locations were being thrown away for it. When the
talkgroup has a resolved anchor, that is the reference and its own radius is
the bound. Distance-from-node stays only as the fallback for a system nobody
has described yet — it was always a stand-in for this.
"""
import httpx
from app.config import settings
@@ -425,9 +460,24 @@ async def _geocode_location(
logger.warning("GOOGLE_MAPS_API_KEY not set — geocoding disabled")
return None
if anchor:
ref_lat, ref_lon = anchor["lat"], anchor["lng"]
max_km = anchor["radius_km"]
# Bias box scaled to the anchor rather than a fixed half-degree, so a
# village biases tightly and a county loosely.
delta = max(max_km / 111.0, 0.05)
ref_label = "anchor"
elif node_lat is not None and node_lon is not None:
ref_lat, ref_lon = node_lat, node_lon
max_km = settings.geocode_max_km
delta = _GEO_DELTA
ref_label = "node"
else:
return None
bounds = (
f"{node_lat - _GEO_DELTA},{node_lon - _GEO_DELTA}"
f"|{node_lat + _GEO_DELTA},{node_lon + _GEO_DELTA}"
f"{ref_lat - delta},{ref_lon - delta}"
f"|{ref_lat + delta},{ref_lon + delta}"
)
params = {
"address": location_str,
@@ -465,15 +515,18 @@ async def _geocode_location(
return None
loc = result["geometry"]["location"]
lat, lng = float(loc["lat"]), float(loc["lng"])
dist_km = _geo_dist_km(node_lat, node_lon, lat, lng)
if dist_km > settings.geocode_max_km:
dist_km = _geo_dist_km(ref_lat, ref_lon, lat, lng)
if dist_km > max_km:
logger.warning(
f"Geocoding rejected '{location_str}' → ({lat:.4f}, {lng:.4f}) "
f"— {dist_km:.1f}km from node exceeds geocode_max_km={settings.geocode_max_km}"
f"— {dist_km:.1f}km from {ref_label} exceeds {max_km:.1f}km"
)
return None
coords = {"lat": lat, "lng": lng}
logger.info(f"Geocoded '{location_str}' → {coords} ({dist_km:.1f}km from node) [{location_type}]")
logger.info(
f"Geocoded '{location_str}' → {coords} "
f"({dist_km:.1f}km from {ref_label}) [{location_type}]"
)
return coords
except Exception as e:
logger.warning(f"Geocoding failed for '{location_str}': {e}")
+255
View File
@@ -0,0 +1,255 @@
"""
Place verification — is the name the corrector produced a real place *here*?
The transcript corrector (`transcript_correction.py`) substitutes sound-alikes
against a reference list. It has no way to tell whether its own output is a real
place, so "Cool Parts, Illinois" and "Shout out to Optum" are exactly as
acceptable to it as a genuine street name. This module is the check
(server-26#37).
MAPS AS A VERIFIER, NOT AS PROMPT STUFFING. Injecting every road and POI in a
town would be hundreds of names on a pass that runs on every transcribed call.
Instead we take the handful of location-shaped nouns a transcript actually
contains and ask one question per noun:
1. Geocode it, bounded by the talkgroup's anchor.
2. Inside the radius -> accept, done.
3. Outside, or no result -> look for a sound-alike that DOES resolve inside.
4. Found one -> correct to it, and propose {term, meaning} to that
talkgroup's local_knowledge as pending.
Cost scales with location nouns, not call volume, and every verified miss
permanently improves the reference data for that channel.
NO ANCHOR MEANS SKIP, NOT ACCEPT. An anchor too wide to discriminate is not
stored at all (see `area_context`), and without one this module returns
immediately. A statewide radius would confirm anything inside it, which is worse
than not checking — it looks like verification and is not.
THE FREE TIER RUNS FIRST. A sound-alike among the terms the operator already
entered costs nothing and is more trustworthy than anything Maps guesses, so
`local_knowledge` and `vocabulary` are searched before any request goes out.
"""
import re
from difflib import SequenceMatcher
from typing import Any, Optional
from app.config import settings
from app.internal import area_context
from app.internal.logger import logger
# Soundex-style consonant classes. Letters that a vocoder + Whisper routinely
# swap land in the same bucket, so "Optum"/"Ossining" stay far apart while
# "Snowden"/"Snowdon" collapse together.
_CLASSES = {
"b": "1", "f": "1", "p": "1", "v": "1",
"c": "2", "g": "2", "j": "2", "k": "2", "q": "2", "s": "2", "x": "2", "z": "2",
"d": "3", "t": "3",
"l": "4",
"m": "5", "n": "5",
"r": "6",
}
_DIGRAPHS = (("ph", "f"), ("gh", "g"), ("ck", "k"), ("wr", "r"), ("kn", "n"), ("wh", "w"))
def _norm(text: str) -> str:
return re.sub(r"[^a-z0-9]+", " ", (text or "").lower()).strip()
def phonetic_key(text: str) -> str:
"""
Consonant-class skeleton of a name. Vowels drop out; a run of the same class
collapses unless a vowel separates it.
"""
letters = re.sub(r"[^a-z]", "", (text or "").lower())
for a, b in _DIGRAPHS:
letters = letters.replace(a, b)
out: list[str] = []
prev = ""
for ch in letters:
code = _CLASSES.get(ch, "")
if code and code != prev:
out.append(code)
prev = code if ch not in "aeiouyhw" else ""
return "".join(out)
def sounds_like(heard: str, candidate: str) -> float:
"""
0..1 similarity, the better of the phonetic and the literal comparison.
Both are needed: Whisper errors are sometimes phonetic ("5 acre" for
"5-baker") and sometimes near-spellings ("Croton Ave" for "Croton Avenue"),
and a key comparison alone scores the second one poorly.
"""
literal = SequenceMatcher(None, _norm(heard), _norm(candidate)).ratio()
ka, kb = phonetic_key(heard), phonetic_key(candidate)
phonetic = SequenceMatcher(None, ka, kb).ratio() if ka and kb else 0.0
return max(literal, phonetic)
# -- Maps ----------------------------------------------------------------------
def _place_suffix(area: dict) -> str:
parts = [area[f] for f in area_context.PLACE_FIELDS if area.get(f)]
return ", ".join(parts)
async def _geocode_in_anchor(query: str, anchor: dict) -> Optional[dict]:
"""Geocode `query` and return its coords only if they land inside the anchor."""
from app.internal.intelligence import _geocode_location
coords = await _geocode_location(query, anchor=anchor)
return coords
async def _places_soundalike(heard: str, anchor: dict) -> Optional[dict]:
"""
Ask Maps for places near the anchor matching the misheard text.
Places Text Search does its own fuzzy matching against a biased region, which
is usually enough — but "usually" is not a standard, so the result still has
to pass `sounds_like` before it is allowed to rewrite a transcript. Without
that guard the API happily returns the nearest gas station for any garbage
string.
"""
if not settings.google_maps_api_key:
return None
import httpx
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(
"https://maps.googleapis.com/maps/api/place/textsearch/json",
params={
"query": heard,
"location": f"{anchor['lat']},{anchor['lng']}",
"radius": int(anchor["radius_km"] * 1000),
"key": settings.google_maps_api_key,
},
)
r.raise_for_status()
data = r.json()
except Exception as e:
logger.warning(f"Place search failed for {heard!r}: {e}")
return None
if data.get("status") not in ("OK", "ZERO_RESULTS"):
logger.warning(f"Place search for {heard!r} returned {data.get('status')}")
return None
for result in (data.get("results") or [])[:5]:
name = (result.get("name") or "").strip()
loc = (result.get("geometry") or {}).get("location") or {}
if not name or "lat" not in loc:
continue
distance = area_context.geo_dist_km(
anchor["lat"], anchor["lng"], float(loc["lat"]), float(loc["lng"])
)
if distance > anchor["radius_km"]:
continue
score = sounds_like(heard, name)
if score >= settings.place_soundalike_min_ratio:
return {"term": name, "meaning": result.get("formatted_address") or None, "score": score}
return None
def _known_soundalike(heard: str, area: dict) -> Optional[dict]:
"""Best sound-alike among terms the operator already entered. Free."""
best: Optional[dict] = None
for entry in area.get("local_knowledge") or []:
term = entry.get("term") or ""
if not term or _norm(term) == _norm(heard):
continue
score = sounds_like(heard, term)
if score >= settings.place_soundalike_min_ratio and (best is None or score > best["score"]):
best = {"term": term, "meaning": entry.get("meaning"), "score": score, "known": True}
return best
# -- Public --------------------------------------------------------------------
def _substitute(text: str, swaps: list[tuple[str, str]]) -> str:
for heard, replacement in swaps:
text = re.sub(rf"\b{re.escape(heard)}\b", replacement, text, flags=re.IGNORECASE)
return text
async def verify(
call_id: str,
text: str,
segments: Optional[list[dict]],
locations: list[str],
system_area: Optional[dict],
tg_area: Optional[dict],
system_id: Optional[str] = None,
talkgroup_id: Optional[Any] = None,
) -> tuple[Optional[str], Optional[list[dict]]]:
"""
Check the corrector's location nouns against the talkgroup's anchor.
Returns (text, segments) with verified substitutions applied, or (None, None)
when nothing changed. Like correction itself, this is an improvement and
never a dependency: any failure leaves the transcript exactly as it was.
"""
if not settings.place_verification_enabled or not locations:
return None, None
anchor = area_context.anchor_for(system_area, tg_area)
if not anchor:
return None, None # load-bearing: no anchor means skip, never accept
area = area_context.effective(system_area, tg_area)
suffix = _place_suffix(area)
swaps: list[tuple[str, str]] = []
proposals: list[dict] = []
for heard in locations[: settings.place_verify_max_per_call]:
heard = (heard or "").strip()
if not heard:
continue
query = f"{heard}, {suffix}" if suffix else heard
try:
if await _geocode_in_anchor(query, anchor):
continue # real place, in the right area — nothing to do
candidate = _known_soundalike(heard, area) or await _places_soundalike(heard, anchor)
except Exception as e:
logger.warning(f"Place verification failed for {heard!r} on call {call_id}: {e}")
continue
if not candidate:
logger.info(
f"Place verification: {heard!r} (call {call_id}) does not resolve near the "
f"anchor and has no sound-alike that does — leaving it alone"
)
continue
swaps.append((heard, candidate["term"]))
if not candidate.get("known"):
proposals.append({
"term": candidate["term"],
"meaning": candidate.get("meaning"),
"source": "place_verifier",
"source_call_ids": [call_id],
})
logger.info(
f"Place verification: {heard!r} -> {candidate['term']!r} "
f"(score {candidate['score']:.2f}, call {call_id})"
)
if not swaps:
return None, None
if proposals and system_id and talkgroup_id is not None:
try:
await area_context.add_pending(system_id, talkgroup_id, proposals)
except Exception as e:
logger.warning(f"Could not queue verified terms for call {call_id}: {e}")
new_text = _substitute(text or "", swaps)
new_segments = None
if segments:
new_segments = [{**s, "text": _substitute(s.get("text", ""), swaps)} for s in segments]
if all(a["text"] == b.get("text") for a, b in zip(new_segments, segments)):
new_segments = None
return (new_text if new_text != (text or "") else None), new_segments
@@ -35,7 +35,9 @@ import json
from typing import Any, Optional
from app.config import settings
from app.internal import area_context
from app.internal import firestore as fstore
from app.internal import place_verifier
from app.internal.logger import logger
# A transcript this short has no proper nouns to get wrong — "10-4.", "6-2,
@@ -65,6 +67,10 @@ Return JSON:
are numbered.
not_speech: true if this is recogniser noise rather than a transmission
changed: list of ["heard" -> "corrected"] pairs you applied, for audit
locations: every place name in your corrected output, exactly as it appears
there — streets, intersections, businesses, schools, towns,
landmarks. Include ones you are unsure of; that is the point.
A unit call sign or a person's name is NOT a location.
{transcript}"""
@@ -107,35 +113,45 @@ def _talkgroup_entry(system_doc: dict, talkgroup_id: Optional[int]) -> dict:
return {}
def _area_lines(area: dict, label: str) -> list[str]:
"""Render an area_context dict as prompt lines. Empty when nothing is set."""
def _area_lines(area: dict) -> list[str]:
"""
Render a merged area_context as prompt lines. Empty when nothing is set.
One block, not one per scope: by the time this runs the two scopes have
already been merged with talkgroup ahead of system, and showing the model
two competing lists invites it to pick from the wrong one.
"""
if not area:
return []
parts: list[str] = []
for key, heading in (
("municipality", "Municipality"),
("county", "County"),
("roads", "Roads and highways"),
("landmarks", "Landmarks, businesses and facilities"),
):
value = area.get(key)
if not value:
continue
if isinstance(value, list):
value = ", ".join(str(v) for v in value if v)
if value:
parts.append(f" {heading}: {value}")
return [f"{label}:"] + parts if parts else []
lines: list[str] = []
place = ", ".join(
str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f)
)
if place:
lines.append(f"Area covered by this channel: {place}")
knowledge = area.get("local_knowledge") or []
if knowledge:
lines.append("Local names heard on this channel:")
lines.extend(
f" {e['term']} — {e['meaning']}" if e.get("meaning") else f" {e['term']}"
for e in knowledge
)
return lines
async def resolve_context(system_id: Optional[str], talkgroup_id: Optional[int]) -> dict:
"""
Merge the reference data a corrector needs, talkgroup ahead of system.
Returns {"vocabulary": [...], "ten_codes": {...}, "area_lines": [...]}.
Empty everywhere is legitimate — a system nobody has configured yet.
Returns {"vocabulary", "ten_codes", "area_lines", "area", "system_area",
"tg_area"}. Empty everywhere is legitimate — a system nobody has configured
yet. The two raw scopes come back alongside the merge because the place
verifier needs them to pick an anchor (server-26#37).
"""
empty: dict[str, Any] = {"vocabulary": [], "ten_codes": {}, "area_lines": []}
empty: dict[str, Any] = {
"vocabulary": [], "ten_codes": {}, "area_lines": [],
"area": {}, "system_area": {}, "tg_area": {},
}
if not system_id:
return empty
@@ -155,12 +171,18 @@ async def resolve_context(system_id: Optional[str], talkgroup_id: Optional[int])
ten_codes = dict(system_doc.get("ten_codes") or {})
ten_codes.update(tg.get("ten_codes") or {})
area_lines = (
_area_lines(tg.get("area_context") or {}, "Area covered by this talkgroup")
+ _area_lines(system_doc.get("area_context") or {}, "Area covered by this system")
)
system_area = system_doc.get("area_context") or {}
tg_area = tg.get("area_context") or {}
area = area_context.effective(system_area, tg_area)
return {"vocabulary": vocabulary, "ten_codes": ten_codes, "area_lines": area_lines}
return {
"vocabulary": vocabulary,
"ten_codes": ten_codes,
"area_lines": _area_lines(area),
"area": area,
"system_area": system_area,
"tg_area": tg_area,
}
def build_context_block(context: dict, talkgroup_name: Optional[str]) -> str:
@@ -267,6 +289,34 @@ async def correct(
f"segment(s) against {len(segments)} — discarding segment corrections"
)
# Maps has the last word on place names (server-26#37). The corrector can
# only match against the list it was handed, so a plausible-sounding invention
# — "Cool Parts, Illinois" — reads exactly like a real street to it. The
# verifier geocodes each location noun against the talkgroup's anchor and,
# on a miss, looks for a sound-alike that does resolve there. It runs on the
# corrected copy so it judges the text everything downstream will actually
# read, and it skips entirely when there is no discriminating anchor.
if not not_speech:
locations = [x for x in (raw.get("locations") or []) if isinstance(x, str)]
try:
verified_text, verified_segments = await place_verifier.verify(
call_id,
corrected or text,
corrected_segments or segments,
locations,
context.get("system_area"),
context.get("tg_area"),
system_id=system_id,
talkgroup_id=talkgroup_id,
)
except Exception as e:
logger.warning(f"Place verification failed for call {call_id}: {e}")
verified_text, verified_segments = None, None
if verified_text:
corrected = verified_text
if verified_segments:
corrected_segments = verified_segments
if corrected or corrected_segments or not_speech:
changed = raw.get("changed") or []
logger.info(
+118 -57
View File
@@ -20,8 +20,9 @@ import json
import random
import re
from datetime import datetime, timezone, timedelta
from typing import Optional
from typing import Any, Optional
from app.internal.logger import logger
from app.internal import area_context
from app.internal import firestore as fstore
from app.config import settings
@@ -57,26 +58,32 @@ Do NOT include common English words. Max 80 terms. Only include what you are con
accurate for this specific area; return fewer terms rather than guessing."""
_INDUCTION_PROMPT = """\
You are analyzing P25 emergency radio transcripts to find vocabulary terms that should be \
added to improve future speech-to-text accuracy for this system.
You are analyzing P25 emergency radio transcripts from ONE talkgroup (a single radio channel) \
to find local terms that should be added to improve future speech-to-text accuracy for that \
channel.
System: {system_name}
Existing approved vocabulary (do not re-propose these): {existing_vocab}
Channel: {talkgroup_name}
Area: {area_hint}
Terms this channel already knows (do not re-propose these): {existing_vocab}
Sampled transcripts:
{transcript_block}
Find terms that are LIKELY STT errors or local terms missing from the vocabulary:
Find terms that are LIKELY STT errors or local terms missing from the list:
- Unit IDs that appear garbled (e.g. "5 acre" → "5-baker")
- Agency acronyms spelled out phonetically (e.g. "why vac" → "YVAC")
- Street names or locations that look misspelled or oddly transcribed
- Callsigns or local codes not yet in the vocabulary
- Callsigns or local codes not yet known
Return ONLY a JSON object:
{{"new_terms": ["term1", "term2", ...]}}
{{"new_terms": [{{"term": "YVAC", "meaning": "Yorktown Volunteer Ambulance Corps"}}, ...]}}
Only include high-confidence additions not already in existing vocabulary.
Return {{"new_terms": []}} if nothing new is found."""
`meaning` is what the term refers to — an agency, a road, a unit type. Omit it or use null \
when you genuinely do not know; a term with no meaning is still worth proposing.
Only propose what is specific to THIS channel and this area. Do not propose a term just \
because it appears often. Return {{"new_terms": []}} if nothing new is found."""
# ─────────────────────────────────────────────────────────────────────────────
@@ -97,10 +104,17 @@ async def bootstrap_system_vocabulary(system_id: str) -> list[str]:
system_name = system_doc.get("name", "Unknown")
system_type = system_doc.get("type", "P25")
# Build area hint from configured talkgroup names
talkgroups = system_doc.get("config", {}).get("talkgroups", [])
tg_names = [tg.get("name", "") for tg in talkgroups if tg.get("name")][:8]
area_hint = f"Talkgroups include: {', '.join(tg_names)}" if tg_names else "Unknown area"
# Prefer the place an operator actually set. Guessing the area from talkgroup
# names is thin for a single-municipality system and close to useless for a
# multi-county one (server-26#36), so it is only the fallback now.
area = system_doc.get("area_context") or {}
place = ", ".join(str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f))
if place:
area_hint = place
else:
talkgroups = system_doc.get("config", {}).get("talkgroups", [])
tg_names = [tg.get("name", "") for tg in talkgroups if tg.get("name")][:8]
area_hint = f"Talkgroups include: {', '.join(tg_names)}" if tg_names else "Unknown area"
terms = await asyncio.to_thread(_sync_bootstrap, system_name, system_type, area_hint)
if not terms:
@@ -279,9 +293,20 @@ async def _run_induction_pass() -> None:
async def _induct_system(system_id: str, system_doc: dict) -> None:
"""Sample random transcripts for a system and propose new vocabulary."""
system_name = system_doc.get("name", "Unknown")
existing_vocab: list[str] = system_doc.get("vocabulary") or []
"""
Sample recent transcripts per TALKGROUP and propose local knowledge there.
Proposals used to land at system level, which is the wrong blast radius
(server-26#37). A wrong term on a talkgroup misleads one channel; the same
term at system level misleads every channel on that system — including one
400km away on a statewide system, which is exactly the context poisoning the
scope rule exists to prevent. If a term really does apply system-wide,
carrying it on several talkgroups costs almost nothing, while auto-promoting
a wrong one is expensive to notice. So: talkgroup-level pending terms only,
and nothing here ever promotes upward or approves itself.
"""
system_name = system_doc.get("name", "Unknown")
system_area = system_doc.get("area_context") or {}
# Fetch calls from the last 7 days only — avoids scanning the entire history.
# Active calls have ended_at=None and are excluded by the range filter automatically.
@@ -294,57 +319,87 @@ async def _induct_system(system_id: str, system_doc: dict) -> None:
if not all_calls:
return
# Random sample up to the token budget (4 chars ≈ 1 token)
random.shuffle(all_calls)
char_budget = settings.vocabulary_induction_sample_tokens * 4
by_tg: dict[Any, list[dict]] = {}
for call in all_calls:
tgid = call.get("talkgroup_id")
if tgid is None:
continue
by_tg.setdefault(tgid, []).append(call)
# The sample budget is per system, split across the talkgroups that have
# traffic — a channel with 400 calls should not starve one with 12.
char_budget = max(
(settings.vocabulary_induction_sample_tokens * 4) // max(len(by_tg), 1), 800
)
for talkgroup_id, calls in by_tg.items():
try:
await _induct_talkgroup(
system_id, system_doc, system_name, system_area,
talkgroup_id, calls, char_budget,
)
except Exception as e:
logger.warning(
f"Induction failed for talkgroup {talkgroup_id} on system {system_id}: {e}"
)
async def _induct_talkgroup(
system_id: str,
system_doc: dict,
system_name: str,
system_area: dict,
talkgroup_id: Any,
calls: list[dict],
char_budget: int,
) -> None:
tg_entry = area_context.talkgroup_entry(system_doc, talkgroup_id)
tg_area = tg_entry.get("area_context") or {}
area = area_context.effective(system_area, tg_area)
talkgroup_name = (
tg_entry.get("name")
or calls[0].get("talkgroup_name")
or f"TGID {talkgroup_id}"
)
known = area_context._known_terms(tg_entry, system_doc)
random.shuffle(calls)
transcript_block = ""
sampled_call_docs: list[dict] = []
sampled = 0
for call in all_calls:
for call in calls:
text = call.get("transcript_corrected") or call.get("transcript") or ""
if not text:
continue
if len(transcript_block) + len(text) > char_budget:
break
tg = call.get("talkgroup_name") or f"TGID {call.get('talkgroup_id', '?')}"
transcript_block += f"[{tg}] {text}\n"
transcript_block += f"{text}\n"
sampled_call_docs.append(call)
sampled += 1
if sampled < 3:
return # not enough data to learn from yet
if len(sampled_call_docs) < 3:
return # not enough data on this channel to learn from yet
new_terms = await asyncio.to_thread(
_sync_induct, system_name, existing_vocab, transcript_block
place = ", ".join(str(area[f]) for f in area_context.PLACE_FIELDS if area.get(f))
proposed = await asyncio.to_thread(
_sync_induct,
system_name, talkgroup_name, place or "not set",
sorted(known)[:80], transcript_block,
)
if not new_terms:
if not proposed:
return
now = datetime.now(timezone.utc).isoformat()
existing_pending: list[dict] = system_doc.get("vocabulary_pending") or []
pending_lower = {p["term"].lower() for p in existing_pending}
vocab_lower = {t.lower() for t in existing_vocab}
to_queue = []
for t in new_terms:
if t.lower() in vocab_lower or t.lower() in pending_lower:
continue
to_queue.append({
"term": t,
entries = [
{
"term": p["term"],
"meaning": p.get("meaning"),
"source": "induction",
"added_at": now,
"source_call_ids": _find_source_calls(t, sampled_call_docs),
})
if not to_queue:
return
await fstore.doc_set("systems", system_id, {
"vocabulary_pending": existing_pending + to_queue,
})
logger.info(
f"Vocabulary induction: {len(to_queue)} new term(s) proposed for "
f"system {system_id} ({system_name}): {[p['term'] for p in to_queue]}"
)
"source_call_ids": _find_source_calls(p["term"], sampled_call_docs),
}
for p in proposed
if p.get("term") and p["term"].lower() not in known
]
if entries:
await area_context.add_pending(system_id, talkgroup_id, entries)
# ─────────────────────────────────────────────────────────────────────────────
@@ -441,8 +496,13 @@ def _sync_bootstrap(system_name: str, system_type: str, area_hint: str) -> list[
def _sync_induct(
system_name: str, existing_vocab: list[str], transcript_block: str
) -> list[str]:
system_name: str,
talkgroup_name: str,
area_hint: str,
existing_vocab: list[str],
transcript_block: str,
) -> list[dict]:
"""Returns [{term, meaning}] — a bare string is still accepted from the model."""
from app.config import settings as cfg
from openai import OpenAI
@@ -452,6 +512,8 @@ def _sync_induct(
vocab_str = ", ".join(existing_vocab[:80]) if existing_vocab else "(none yet)"
prompt = _INDUCTION_PROMPT.format(
system_name=system_name,
talkgroup_name=talkgroup_name,
area_hint=area_hint,
existing_vocab=vocab_str,
transcript_block=transcript_block[:8000],
)
@@ -463,8 +525,7 @@ def _sync_induct(
response_format={"type": "json_object"},
)
data = json.loads(response.choices[0].message.content)
terms = data.get("new_terms") or []
return [str(t).strip() for t in terms if str(t).strip()]
return area_context.normalize_local_knowledge(data.get("new_terms") or [])
except Exception as e:
logger.warning(f"Vocabulary induction GPT call failed: {e}")
return []