Cut AI pipeline cost: evaluate cheaper / alternative models for STT, scene extraction, correlation #296

New Issue

2026-10-27T19:27:14-04:00

logan commented

2026-10-27 19:27:14 -04:00

Follow-up to #196 (closed 2026-10-27 as billing, not a bug). Prod AI went dark when the Gemini + OpenAI credits ran out, and the owner wants to know whether the same system can run cheaper.

Candidates to measure via Admin Replay (#170) against the answer keys:

STT: self-hosted Whisper (faster-whisper / whisper.cpp on CPU or GPU) instead of the OpenAI API. Ties into the #163 accuracy work.
Scene/entity extraction: smaller or cheaper Gemini tiers, or a local model.
Correlation: lean more on the rules tier and call the LLM less (#35: the LLM already decides only 3.5% of links).

Can't price anything per call until #45 (no per-call/org cost accounting) is done, so that comes first or alongside.

Follow-up to #196 (closed 2026-10-27 as billing, not a bug). Prod AI went dark when the Gemini + OpenAI credits ran out, and the owner wants to know whether the same system can run cheaper. Candidates to measure via Admin Replay (#170) against the answer keys: - STT: self-hosted Whisper (faster-whisper / whisper.cpp on CPU or GPU) instead of the OpenAI API. Ties into the #163 accuracy work. - Scene/entity extraction: smaller or cheaper Gemini tiers, or a local model. - Correlation: lean more on the rules tier and call the LLM less (#35: the LLM already decides only 3.5% of links). Can't price anything per call until #45 (no per-call/org cost accounting) is done, so that comes first or alongside.

logan added the

type

feature

area/ai-pipeline area/transcription

priority

p1

labels 2026-10-27 19:27:14 -04:00

logan referenced this issue

2026-10-27 20:03:49 -04:00

Transcription: move to an OpenAI-compatible cheaper provider (Groq whisper-large-v3-turbo) with a domain vocabulary prompt #297

logan referenced this issue

2026-10-27 20:03:49 -04:00