Updates to reduce firestore calls to try and stay in free tier

### Firestore read reductions

**1. `doc_get_cached()` in `firestore.py` — new 5-min TTL cache**
One place, benefits everything. System and node config documents almost never change during a monitoring session.

**2. System doc: 4 reads → 1 per call**
| Before | After |
|---|---|
| `upload.py` — `doc_get("systems")` for ai_flags | `doc_get_cached` |
| `transcription.py` — `get_vocabulary()` → `doc_get("systems")` | cache hit |
| `intelligence.py` — `get_vocabulary()` → `doc_get("systems")` | cache hit |
| `intelligence.py` — `doc_get("systems")` again for ten_codes | eliminated (reads same cached doc) |

**3. Node doc: cached in `_on_call_start` and `intelligence.py`**
The node is read every call event to get `assigned_system_id` and lat/lon for geocoding. Both now use the cache — node assignments and positions essentially never change at runtime.

**4. Node sweeper: 30s → 90s interval**
The sweeper was doing a full node collection scan 3× more often than necessary — the offline threshold is already 90s. Cuts sweeper reads by 66%.

**5. Vocabulary induction: scans all-time calls → last 7 days**
Previously fetched every ended call for a system (could be thousands). Now scoped to the last 7 days.

> **Note:** The vocabulary induction query `(system_id == X, ended_at >= cutoff)` needs a Firestore
> composite index on `(system_id ASC, ended_at ASC)`. When the induction loop first fires it will log
> an error with a Firebase Console link to create it in one click.
This commit is contained in:
Logan
2026-05-04 02:05:00 -04:00
parent 97f4286810
commit 7e1b01a275
6 changed files with 43 additions and 15 deletions
+10 -4
View File
@@ -196,8 +196,8 @@ async def remove_term(system_id: str, term: str) -> None:
async def get_vocabulary(system_id: str) -> dict:
"""Return vocabulary and pending terms for a system."""
doc = await fstore.doc_get("systems", system_id)
"""Return vocabulary and pending terms for a system (TTL-cached, 5 min)."""
doc = await fstore.doc_get_cached("systems", system_id)
if not doc:
return {"vocabulary": [], "vocabulary_pending": [], "vocabulary_bootstrapped": False}
return {
@@ -281,8 +281,14 @@ async def _induct_system(system_id: str, system_doc: dict) -> None:
system_name = system_doc.get("name", "Unknown")
existing_vocab: list[str] = system_doc.get("vocabulary") or []
# Fetch recent ended calls for this system
all_calls = await fstore.collection_list("calls", system_id=system_id, status="ended")
# Fetch calls from the last 7 days only — avoids scanning the entire history.
# Active calls have ended_at=None and are excluded by the range filter automatically.
# Needs a composite index on (system_id ASC, ended_at ASC).
cutoff = datetime.now(timezone.utc) - timedelta(days=7)
all_calls = await fstore.collection_where("calls", [
("system_id", "==", system_id),
("ended_at", ">=", cutoff),
])
if not all_calls:
return