logan/server-26
1
0
Fork 0
Code Issues 84 Pull Requests Actions Packages Projects Releases Wiki Activity

Prod AI dark since 09-29/09-30: Gemini and OpenAI credits exhausted (billing, not a bug) #196

New Issue
Closed
opened 2026-10-04 01:15:39 -04:00 by logan · 1 comment
logan commented 2026-10-04 01:15:39 -04:00
Owner
Copy Link
Copy Source

/health/ai checked 2026-10-04 04:50Z:

tier dry since (UTC) problem
correlation_cheap (gemini-3.6-flash) 2026-09-29 15:07:53 Gemini account out of credit
transcription (whisper-1) 2026-09-30 12:15:41 OpenAI account cannot be billed, 9,616 consecutive failures

Effect, confirmed from call docs: LLM correlation decisions drop to 0 from 09-29 15:08Z, so incidents are rules-only. From 09-30 12:15Z there are no transcripts and no new incidents at all. The portal has shown nothing new for about 4 days. alerted: true on both tiers, but nobody acted on the alert.

Fix: owner tops up both accounts (links are in /health/ai). Nothing to debug.

Cost reading: the live run lasted about 48h on Gemini (09-27 15Z to 09-29 15Z), but the starting balance isn't recorded, so no per-day figure can be derived. See #45 (no cost accounting).

Measured result of the healthy window is on #170.

`/health/ai` checked 2026-10-04 04:50Z: | tier | dry since (UTC) | problem | |---|---|---| | correlation_cheap (gemini-3.6-flash) | 2026-09-29 15:07:53 | Gemini account out of credit | | transcription (whisper-1) | 2026-09-30 12:15:41 | OpenAI account cannot be billed, 9,616 consecutive failures | Effect, confirmed from call docs: LLM correlation decisions drop to 0 from 09-29 15:08Z, so incidents are rules-only. From 09-30 12:15Z there are no transcripts and no new incidents at all. The portal has shown nothing new for about 4 days. `alerted: true` on both tiers, but nobody acted on the alert. **Fix:** owner tops up both accounts (links are in /health/ai). Nothing to debug. Cost reading: the live run lasted about 48h on Gemini (09-27 15Z to 09-29 15Z), but the starting balance isn't recorded, so no per-day figure can be derived. See #45 (no cost accounting). Measured result of the healthy window is on #170.
logan added the area/ai-pipelinearea/billing
priority
p0
status
blocked
type
tracking
 labels 2026-10-05 02:05:00 -04:00
logan commented 2026-10-27 19:27:09 -04:00
Author
Owner
Copy Link
Copy Source

Closing per owner (2026-10-27): running out of credits is a billing state, not a bug. The real follow-up (cheaper or alternative ways to run the AI pipeline) is filed as a separate issue.

Closing per owner (2026-10-27): running out of credits is a billing state, not a bug. The real follow-up (cheaper or alternative ways to run the AI pipeline) is filed as a separate issue.
logan closed this issue 2026-10-27 19:27:09 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area/ai-pipeline
Scene/entity extraction, summarizer, AI providers, embeddings
 area/auth
AuthN/Z, roles, tenancy/org scoping, API keys, credentials
 area/billing
Plans, entitlements, AI spend accounting and caps
 area/correlation
Incident correlation, linking, lifecycle, severity, corr_debug
 area/discord
Discord bot, radio bots, voice streams, bot tokens, webhooks
 area/frontend
Web UI: Next.js app and the edge-node local dashboard
 area/governance
Board process, minutes, unattended runner, agent tooling
 area/growth
Go-to-market, outreach, cold email, discovery conversations
 area/infra-deploy
CI/CD, VM, registry, Firestore rules/indexes deploy, Terraform/Caddy
 area/legal
ToS, retention, licences/DAAs, CAN-SPAM, E&O, name suppression
 area/map
Live map: pins, paths, tiles, legend, geocoding
 area/mqtt
Node-C2 MQTT, checkins, commands, node sweeper, dynsec
 area/node-ota
Node install, enrollment, OTA updater, image skew
 area/osint
External feeds: cameras, 511, ADS-B, AIS, outages, and their licences
 area/recording-audio
Call capture, boundaries, PulseAudio, ring buffer, trimming, attribution
 area/sdr-radio
OP25 decode, SDR hardware/tuning, conventional, second SDR, mesh radio
 area/transcription
STT, transcript correction, place verification, name accuracy
 board
Board process: minutes and board-filed items
 board:agenda
Needs board scoping at the next sitting; CEO removes the label once scoped
 minutes:draft
One exec pre-read, closed by CEO
 minutes:final
CEO ruling; open until enacted
 role:ceo
Filed by the CEO
 role:cfo
Filed by the CFO
 role:cio
Filed by the CIO
 role:ciso
Filed by the CISO
 role:cmo
Filed by the CMO
 role:coo
Filed by the COO
 role:cto
Filed by the CTO
kind
behavior-bug
Wrong product behavior; the instrument reads it correctly
kind
measurement-bug
Corrupts the recording/instrument (debug data, evals, health checks), not behavior
priority
p0
Drop everything: prod broken, live exposure, or owner-ruled P0
priority
p1
Next up: blocks a gate, a customer, or core incident quality
priority
p2
Normal: real defect or planned work, schedule in turn
priority
p3
Low: minor, cosmetic, or someday
status
blocked
Waiting on an external party, spend, or another issue
status
in-progress
Branch or PR open against it
status
needs-decision
Waiting on an owner or board ruling before work
status
needs-info
Root cause or repro unknown; needs a reading first
type
bug
Something built is wrong: code, data, or a measurement
type
chore
Maintenance, refactor, tests, pinning, cleanup: no user-visible change
type
decision
A ruling to make or record: board minutes, owner calls, legal reviews
type
docs
Docs, runbooks, CLAUDE.md, skills, SOPs
type
feature
New capability or an extension of an existing one
type
security
Security, privacy, credential or tenant-isolation exposure
type
tracking
Tracker, checklist, initiative or punch-list spanning several changes
No labels area/ai-pipeline area/billing
priority
p0
status
blocked
type
tracking
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logan
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logan/server-26#196