Live ops wall + pluggable camera providers with licence gating (server-26#198)

Backend (c2-core):
- app/internal/cameras/: provider interface, licence model, per-provider
  catalog cache; one adapter per source in design-canvas/camera-sources.md.
  Implemented: 511NY (legacy shape + tolerant XML inventory parser, 511x
  endpoints UNVERIFIED), Caltrans, DelDOT, Ontario 511, DriveBC, Calgary,
  TfL, NSW, Fintraffic. Stubs: NYC DOT, Westchester DPW, TRANSCOM, CTroads,
  511NJ, Austin, TxDOT, MD CHART, WSDOT, Tallinn, Tarktee, Warendorf,
  TrafficLand, Windy, EarthCam.
- Every provider defaults OFF (config/camera_providers). Gated licences
  (agreement required / unverified / non-commercial / paid) need an explicit
  acknowledgement stored with who/when and written to audit_log; the audit
  write is a precondition. Stubs and prohibited sources cannot be enabled.
  Ack is bound to the licence status it was given for.
- GET /cameras (signed-in), GET/PUT /admin/camera-providers (admin only).
- /traffic/511 follows the ny511 provider switch and returns the DAA credit
  + NYSDOT as-is disclaimer; 511NY event descriptions no longer truncated
  ("may not alter the substance of a report").

Frontend:
- LiveView becomes the ops wall: Region > Incident > Camera drill-down,
  worst-first rail (MapView), KPIs, transcript ticker, scrubber incident
  bands, auto-tour, deep links (#inc=&cam=&tour), Esc steps up.
- Incident panel: nearest cameras (path-weighted), fused situational context
  (ADS-B, AIS, 511NY events, NEXRAD tile sample), units, call spine.
- MapView keeps every layer; adds Camera Viewsheds (approx.) overlay and a
  nearest-camera layer for the selected incident.
- CameraFeed hygiene: hidden-tab/off-screen gating, 2 concurrent HLS cap,
  last good frame + stale badge, frozen-frame signature, HLS stall watchdog,
  provider attribution rendered on the imagery.
- Camera/event popups show source credit, update time and the disclaimer.
- Admin > Camera providers tab with licence status, health, gated confirm.
- Camera detection is not built (blocked on #199/#200); seam only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Logan Cusano
2026-10-04 17:56:06 -04:00
co-authored by Claude Opus 5.5
parent d57738a8f6
commit aa5b71d853
28 changed files with 3342 additions and 132 deletions
+102 -4
View File
@@ -9,6 +9,25 @@ makes no 511 calls at all.
A failed refresh keeps serving the last good data and reports the error and
its age to the caller -- an empty layer must never be the only symptom of a
dead feed (the AI-silent-failures lesson).
**Feed status (server-26#198), 2026-10-04:** 511ny.org cut over to the CARS
"511x" platform around 2026-09-24..30 and the legacy ``/api/getcameras`` and
``/api/getevents`` endpoints now answer HTTP 200 with the SPA's HTML. The new
developer feed is granted only after the NYSDOT Developer Access Agreement
request is approved ("If approved for use, you will receive instructions for
accessing the data feeds & documentation" -- 511NY "For Developers" page), so
its URL shape is NOT public. This client therefore keeps the LEGACY request and
record shape behind ``settings.ny511_api_base`` (default: the legacy base) and
``settings.ny511_api_key``. UNVERIFIED until the grant arrives:
* the new base URL and whether it still takes ``?key=&format=json``;
* whether cameras still come as JSON (the public feed list says
"Camera inventory (XML)") and keep the legacy field names;
* the still/HLS URL shape and its CORS headers.
Never point this at the new site's own backend (api-511x-nysdot.carsprogram.org):
the 511NY Terms of Use ban robots and reverse engineering outside the DAA.
Access is also gated by the camera-provider registry (``ny511`` provider,
default OFF, DAA acknowledgement required) -- see app/internal/cameras/.
"""
import asyncio
import time
@@ -20,11 +39,14 @@ import httpx
from app.config import settings
from app.internal.logger import logger
_BASE = "https://511ny.org/api"
LEGACY_BASE = "https://511ny.org/api"
CAMERAS_TTL_S = 60 * 60 # camera list is near-static
EVENTS_TTL_S = 2 * 60 # accidents/closures change minute to minute
RETRY_AFTER_FAILURE_S = 60
_DESCRIPTION_MAX = 500
# The DAA forbids altering "the substance of a report", so descriptions are
# shown as published. This cap only guards against a pathological record; no
# real 511NY description comes near it, and a cut one is marked with an ellipsis.
_DESCRIPTION_MAX = 4000
class _Feed:
@@ -61,9 +83,22 @@ async def _fetch(path: str, normalize) -> List[Dict[str, Any]]:
if settings.ny511_api_key:
params["key"] = settings.ny511_api_key
async with httpx.AsyncClient(timeout=20.0) as client:
r = await client.get(f"{_BASE}/{path}", params=params)
base = (settings.ny511_api_base or LEGACY_BASE).rstrip("/")
r = await client.get(f"{base}/{path}", params=params)
r.raise_for_status()
raw = r.json()
text = getattr(r, "text", None)
if isinstance(text, str) and text.lstrip()[:1] == "<":
# The 511x camera inventory is published as XML; HTML is the dead
# legacy endpoint serving the SPA shell.
raw = parse_inventory_xml(text)
else:
try:
raw = r.json()
except ValueError as e:
raise ValueError(
"511NY answered neither JSON nor XML (the legacy API has served the 511x "
"site's HTML since the cutover, server-26#198)"
) from e
if not isinstance(raw, list):
raise ValueError(f"expected a JSON list, got {type(raw).__name__}")
out = [n for n in (normalize(x) for x in raw) if n is not None]
@@ -73,6 +108,69 @@ async def _fetch(path: str, normalize) -> List[Dict[str, Any]]:
return out
# ── XML camera inventory (511x) -- UNVERIFIED shape ──────────────────────────
# The 511NY "For Developers" page lists the camera feed as "Camera inventory
# (XML)" but publishes no schema (it arrives with the DAA grant). This parser is
# deliberately tolerant: any element carrying latitude + longitude children (or
# attributes) is a record, and child names are matched case-insensitively onto
# the legacy JSON field names so normalize_camera() works unchanged. Replace the
# alias table with the real element names once the grant's documentation is in hand.
_XML_ALIASES = {
"ID": ("id", "cameraid", "camera_id", "deviceid", "device_id"),
"Latitude": ("latitude", "lat"),
"Longitude": ("longitude", "lon", "lng", "long"),
"Name": ("name", "cameraname", "location", "description", "title"),
"RoadwayName": ("roadwayname", "roadway", "road", "route"),
"DirectionOfTravel": ("directionoftravel", "direction"),
"Url": ("url", "imageurl", "image_url", "snapshoturl", "image", "stillurl"),
"VideoUrl": ("videourl", "video_url", "streamurl", "video", "hlsurl"),
"Disabled": ("disabled",),
"Blocked": ("blocked",),
}
_XML_LOOKUP = {alias: key for key, aliases in _XML_ALIASES.items() for alias in aliases}
_XML_MAX_BYTES = 32 * 1024 * 1024
def _local(tag: str) -> str:
return tag.rsplit("}", 1)[-1].lower()
def _xml_bool(v: Any) -> Any:
if isinstance(v, str) and v.strip().lower() in ("true", "false", "1", "0"):
return v.strip().lower() in ("true", "1")
return v
def parse_inventory_xml(text: str) -> List[Dict[str, Any]]:
"""Map an XML camera inventory onto legacy-shaped dicts. Raises on HTML or
on any DTD/entity declaration (stdlib expat would expand entities)."""
head = text.lstrip()[:512].lower()
if head.startswith("<!doctype html") or head.startswith("<html"):
raise ValueError("511NY answered HTML, not a feed (legacy API dead since the 511x cutover, server-26#198)")
if len(text) > _XML_MAX_BYTES:
raise ValueError("511NY XML inventory larger than the 32 MB safety cap")
if "<!doctype" in text.lower() or "<!entity" in text.lower():
raise ValueError("511NY XML carries a DTD/entity declaration -- refused")
import xml.etree.ElementTree as ET
root = ET.fromstring(text)
out: List[Dict[str, Any]] = []
for el in root.iter():
fields: Dict[str, Any] = {}
for k, v in el.attrib.items():
key = _XML_LOOKUP.get(_local(k))
if key:
fields.setdefault(key, v)
for child in el:
key = _XML_LOOKUP.get(_local(child.tag))
if key and len(child) == 0 and child.text is not None:
fields.setdefault(key, child.text.strip())
if "Latitude" in fields and "Longitude" in fields:
fields["Disabled"] = _xml_bool(fields.get("Disabled", False))
fields["Blocked"] = _xml_bool(fields.get("Blocked", False))
out.append(fields)
return out
def _coords(x: Dict[str, Any]) -> Optional[tuple]:
try:
lat, lon = float(x["Latitude"]), float(x["Longitude"])