Live ops wall + pluggable camera providers with licence gating (server-26#198)
Backend (c2-core):
- app/internal/cameras/: provider interface, licence model, per-provider
catalog cache; one adapter per source in design-canvas/camera-sources.md.
Implemented: 511NY (legacy shape + tolerant XML inventory parser, 511x
endpoints UNVERIFIED), Caltrans, DelDOT, Ontario 511, DriveBC, Calgary,
TfL, NSW, Fintraffic. Stubs: NYC DOT, Westchester DPW, TRANSCOM, CTroads,
511NJ, Austin, TxDOT, MD CHART, WSDOT, Tallinn, Tarktee, Warendorf,
TrafficLand, Windy, EarthCam.
- Every provider defaults OFF (config/camera_providers). Gated licences
(agreement required / unverified / non-commercial / paid) need an explicit
acknowledgement stored with who/when and written to audit_log; the audit
write is a precondition. Stubs and prohibited sources cannot be enabled.
Ack is bound to the licence status it was given for.
- GET /cameras (signed-in), GET/PUT /admin/camera-providers (admin only).
- /traffic/511 follows the ny511 provider switch and returns the DAA credit
+ NYSDOT as-is disclaimer; 511NY event descriptions no longer truncated
("may not alter the substance of a report").
Frontend:
- LiveView becomes the ops wall: Region > Incident > Camera drill-down,
worst-first rail (MapView), KPIs, transcript ticker, scrubber incident
bands, auto-tour, deep links (#inc=&cam=&tour), Esc steps up.
- Incident panel: nearest cameras (path-weighted), fused situational context
(ADS-B, AIS, 511NY events, NEXRAD tile sample), units, call spine.
- MapView keeps every layer; adds Camera Viewsheds (approx.) overlay and a
nearest-camera layer for the selected incident.
- CameraFeed hygiene: hidden-tab/off-screen gating, 2 concurrent HLS cap,
last good frame + stale badge, frozen-frame signature, HLS stall watchdog,
provider attribution rendered on the imagery.
- Camera/event popups show source credit, update time and the disclaimer.
- Admin > Camera providers tab with licence status, health, gated confirm.
- Camera detection is not built (blocked on #199/#200); seam only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
d57738a8f6
commit
aa5b71d853
@@ -9,6 +9,25 @@ makes no 511 calls at all.
|
||||
A failed refresh keeps serving the last good data and reports the error and
|
||||
its age to the caller -- an empty layer must never be the only symptom of a
|
||||
dead feed (the AI-silent-failures lesson).
|
||||
|
||||
**Feed status (server-26#198), 2026-10-04:** 511ny.org cut over to the CARS
|
||||
"511x" platform around 2026-09-24..30 and the legacy ``/api/getcameras`` and
|
||||
``/api/getevents`` endpoints now answer HTTP 200 with the SPA's HTML. The new
|
||||
developer feed is granted only after the NYSDOT Developer Access Agreement
|
||||
request is approved ("If approved for use, you will receive instructions for
|
||||
accessing the data feeds & documentation" -- 511NY "For Developers" page), so
|
||||
its URL shape is NOT public. This client therefore keeps the LEGACY request and
|
||||
record shape behind ``settings.ny511_api_base`` (default: the legacy base) and
|
||||
``settings.ny511_api_key``. UNVERIFIED until the grant arrives:
|
||||
* the new base URL and whether it still takes ``?key=&format=json``;
|
||||
* whether cameras still come as JSON (the public feed list says
|
||||
"Camera inventory (XML)") and keep the legacy field names;
|
||||
* the still/HLS URL shape and its CORS headers.
|
||||
Never point this at the new site's own backend (api-511x-nysdot.carsprogram.org):
|
||||
the 511NY Terms of Use ban robots and reverse engineering outside the DAA.
|
||||
|
||||
Access is also gated by the camera-provider registry (``ny511`` provider,
|
||||
default OFF, DAA acknowledgement required) -- see app/internal/cameras/.
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
@@ -20,11 +39,14 @@ import httpx
|
||||
from app.config import settings
|
||||
from app.internal.logger import logger
|
||||
|
||||
_BASE = "https://511ny.org/api"
|
||||
LEGACY_BASE = "https://511ny.org/api"
|
||||
CAMERAS_TTL_S = 60 * 60 # camera list is near-static
|
||||
EVENTS_TTL_S = 2 * 60 # accidents/closures change minute to minute
|
||||
RETRY_AFTER_FAILURE_S = 60
|
||||
_DESCRIPTION_MAX = 500
|
||||
# The DAA forbids altering "the substance of a report", so descriptions are
|
||||
# shown as published. This cap only guards against a pathological record; no
|
||||
# real 511NY description comes near it, and a cut one is marked with an ellipsis.
|
||||
_DESCRIPTION_MAX = 4000
|
||||
|
||||
|
||||
class _Feed:
|
||||
@@ -61,9 +83,22 @@ async def _fetch(path: str, normalize) -> List[Dict[str, Any]]:
|
||||
if settings.ny511_api_key:
|
||||
params["key"] = settings.ny511_api_key
|
||||
async with httpx.AsyncClient(timeout=20.0) as client:
|
||||
r = await client.get(f"{_BASE}/{path}", params=params)
|
||||
base = (settings.ny511_api_base or LEGACY_BASE).rstrip("/")
|
||||
r = await client.get(f"{base}/{path}", params=params)
|
||||
r.raise_for_status()
|
||||
raw = r.json()
|
||||
text = getattr(r, "text", None)
|
||||
if isinstance(text, str) and text.lstrip()[:1] == "<":
|
||||
# The 511x camera inventory is published as XML; HTML is the dead
|
||||
# legacy endpoint serving the SPA shell.
|
||||
raw = parse_inventory_xml(text)
|
||||
else:
|
||||
try:
|
||||
raw = r.json()
|
||||
except ValueError as e:
|
||||
raise ValueError(
|
||||
"511NY answered neither JSON nor XML (the legacy API has served the 511x "
|
||||
"site's HTML since the cutover, server-26#198)"
|
||||
) from e
|
||||
if not isinstance(raw, list):
|
||||
raise ValueError(f"expected a JSON list, got {type(raw).__name__}")
|
||||
out = [n for n in (normalize(x) for x in raw) if n is not None]
|
||||
@@ -73,6 +108,69 @@ async def _fetch(path: str, normalize) -> List[Dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
# ── XML camera inventory (511x) -- UNVERIFIED shape ──────────────────────────
|
||||
# The 511NY "For Developers" page lists the camera feed as "Camera inventory
|
||||
# (XML)" but publishes no schema (it arrives with the DAA grant). This parser is
|
||||
# deliberately tolerant: any element carrying latitude + longitude children (or
|
||||
# attributes) is a record, and child names are matched case-insensitively onto
|
||||
# the legacy JSON field names so normalize_camera() works unchanged. Replace the
|
||||
# alias table with the real element names once the grant's documentation is in hand.
|
||||
_XML_ALIASES = {
|
||||
"ID": ("id", "cameraid", "camera_id", "deviceid", "device_id"),
|
||||
"Latitude": ("latitude", "lat"),
|
||||
"Longitude": ("longitude", "lon", "lng", "long"),
|
||||
"Name": ("name", "cameraname", "location", "description", "title"),
|
||||
"RoadwayName": ("roadwayname", "roadway", "road", "route"),
|
||||
"DirectionOfTravel": ("directionoftravel", "direction"),
|
||||
"Url": ("url", "imageurl", "image_url", "snapshoturl", "image", "stillurl"),
|
||||
"VideoUrl": ("videourl", "video_url", "streamurl", "video", "hlsurl"),
|
||||
"Disabled": ("disabled",),
|
||||
"Blocked": ("blocked",),
|
||||
}
|
||||
_XML_LOOKUP = {alias: key for key, aliases in _XML_ALIASES.items() for alias in aliases}
|
||||
_XML_MAX_BYTES = 32 * 1024 * 1024
|
||||
|
||||
|
||||
def _local(tag: str) -> str:
|
||||
return tag.rsplit("}", 1)[-1].lower()
|
||||
|
||||
|
||||
def _xml_bool(v: Any) -> Any:
|
||||
if isinstance(v, str) and v.strip().lower() in ("true", "false", "1", "0"):
|
||||
return v.strip().lower() in ("true", "1")
|
||||
return v
|
||||
|
||||
|
||||
def parse_inventory_xml(text: str) -> List[Dict[str, Any]]:
|
||||
"""Map an XML camera inventory onto legacy-shaped dicts. Raises on HTML or
|
||||
on any DTD/entity declaration (stdlib expat would expand entities)."""
|
||||
head = text.lstrip()[:512].lower()
|
||||
if head.startswith("<!doctype html") or head.startswith("<html"):
|
||||
raise ValueError("511NY answered HTML, not a feed (legacy API dead since the 511x cutover, server-26#198)")
|
||||
if len(text) > _XML_MAX_BYTES:
|
||||
raise ValueError("511NY XML inventory larger than the 32 MB safety cap")
|
||||
if "<!doctype" in text.lower() or "<!entity" in text.lower():
|
||||
raise ValueError("511NY XML carries a DTD/entity declaration -- refused")
|
||||
import xml.etree.ElementTree as ET
|
||||
root = ET.fromstring(text)
|
||||
out: List[Dict[str, Any]] = []
|
||||
for el in root.iter():
|
||||
fields: Dict[str, Any] = {}
|
||||
for k, v in el.attrib.items():
|
||||
key = _XML_LOOKUP.get(_local(k))
|
||||
if key:
|
||||
fields.setdefault(key, v)
|
||||
for child in el:
|
||||
key = _XML_LOOKUP.get(_local(child.tag))
|
||||
if key and len(child) == 0 and child.text is not None:
|
||||
fields.setdefault(key, child.text.strip())
|
||||
if "Latitude" in fields and "Longitude" in fields:
|
||||
fields["Disabled"] = _xml_bool(fields.get("Disabled", False))
|
||||
fields["Blocked"] = _xml_bool(fields.get("Blocked", False))
|
||||
out.append(fields)
|
||||
return out
|
||||
|
||||
|
||||
def _coords(x: Dict[str, Any]) -> Optional[tuple]:
|
||||
try:
|
||||
lat, lon = float(x["Latitude"]), float(x["Longitude"])
|
||||
|
||||
Reference in New Issue
Block a user