Call dedup is failing on every call in production: missing calls (system_id, started_at) composite index #84

New Issue

2026-08-27T03:04:19-04:00

logan commented

2026-08-27 03:04:19 -04:00

Found in live c2-core logs during the post-deploy watch on the unattended run of 2026-08-27 (d18e4f0). This is happening on production traffic right now, and was already happening before that deploy — it is not a regression from it.

Roughly every inbound call logs:

[WARNING] Duplicate check failed for call <id>: 400 The query requires an index.
  ...collectionGroups/calls/indexes/... system_id ASC, started_at ASC

Five occurrences in a seven-minute window, against about seven calls. It is not intermittent — dedup is failing on essentially every call.

What this actually costs

dedup.py is what stops the same transmission, heard by two nodes, from being transcribed twice. With its query erroring out, the check degrades to "not a duplicate" and both copies proceed.

While AI is off, the cost is duplicate call documents and inflated counts. The moment an evaluation window opens, every double-heard transmission is paid for twice at Whisper, then extracted twice at Gemini, then correlated twice. That is direct spend, and it also poisons the seven contiguous days of cost data Gate B5 (#45) is waiting on — the figure would be measurably too high for a reason unrelated to the pipeline being measured.

It is also a correctness problem for correlation: two documents for one transmission is exactly the input shape that produces a spurious second incident.

Why it is a WARNING and not an error

dedup.py catches the exception and logs it, so nothing crashes and nothing is loud. That is why this survived undetected — it needed someone reading live logs, not a test. There is no canary on it.

Fix

Add the composite index calls (system_id ASC, started_at ASC) to infra/firestore/firestore.indexes.json and deploy it.

Owner action, and it cannot be done from the authoring machine (per #13 and the DEFERRED.md row): firebase deploy --only firestore:indexes --project discord-radio-bot-461301 from Server/infra/firestore/. Firestore also builds a new composite index asynchronously — it will not take effect the instant the deploy returns.

Related, but not a duplicate

#33 covers the drift between firestore.indexes.json and the live database, and the two redundant ASC duplicates. It does not mention this index and does not describe a live query failing. Worth handling in the same deploy as #33's reconciliation, since both are one firebase deploy. #13 (closed) is the reason a manual deploy step exists at all.

Refs #33, #13, #45.

Found in live `c2-core` logs during the post-deploy watch on the unattended run of 2026-08-27 (`d18e4f0`). **This is happening on production traffic right now**, and was already happening before that deploy — it is not a regression from it. Roughly every inbound call logs: ``` [WARNING] Duplicate check failed for call <id>: 400 The query requires an index. ...collectionGroups/calls/indexes/... system_id ASC, started_at ASC ``` Five occurrences in a seven-minute window, against about seven calls. It is not intermittent — dedup is failing on essentially every call. ## What this actually costs `dedup.py` is what stops the same transmission, heard by two nodes, from being transcribed twice. With its query erroring out, the check degrades to "not a duplicate" and both copies proceed. While AI is off, the cost is duplicate call documents and inflated counts. **The moment an evaluation window opens, every double-heard transmission is paid for twice at Whisper, then extracted twice at Gemini, then correlated twice.** That is direct spend, and it also poisons the seven contiguous days of cost data Gate B5 (#45) is waiting on — the figure would be measurably too high for a reason unrelated to the pipeline being measured. It is also a correctness problem for correlation: two documents for one transmission is exactly the input shape that produces a spurious second incident. ## Why it is a WARNING and not an error `dedup.py` catches the exception and logs it, so nothing crashes and nothing is loud. That is why this survived undetected — it needed someone reading live logs, not a test. There is no canary on it. ## Fix Add the composite index `calls (system_id ASC, started_at ASC)` to `infra/firestore/firestore.indexes.json` and deploy it. **Owner action, and it cannot be done from the authoring machine** (per #13 and the `DEFERRED.md` row): `firebase deploy --only firestore:indexes --project discord-radio-bot-461301` from `Server/infra/firestore/`. Firestore also builds a new composite index asynchronously — it will not take effect the instant the deploy returns. ## Related, but not a duplicate #33 covers the *drift* between `firestore.indexes.json` and the live database, and the two redundant ASC duplicates. It does not mention this index and does not describe a live query failing. Worth handling in the same deploy as #33's reconciliation, since both are one `firebase deploy`. #13 (closed) is the reason a manual deploy step exists at all. Refs #33, #13, #45.

logan referenced this issue

2026-08-27 03:05:13 -04:00

Unattended runs died silently on usage limits for 3 cycles - Task Scheduler reported success #82

logan commented

2026-08-28 03:10:08 -04:00

Fixed and live. Closing.

The composite index calls (system_id ASC, started_at ASC) was created directly on the c2-server database and has finished building — state READY, all 8 composite indexes now READY. It is also declared in infra/firestore/firestore.indexes.json in commit 187b8c1, so source and the live database agree and the next deploy will not offer to remove it.

This did not need the owner after all. The issue said the fix "cannot be done from the authoring machine" and required a manual firebase deploy. It does not: the production host is a GCP VM already authenticated as the project's compute service account, so gcloud firestore indexes composite create --database=c2-server ... applies it headlessly over SSH. Worth recording, because #51 (no automated deploy path for rules and indexes) has been treating index changes as inherently owner-blocked, and at least the index half of that is not.

Before: 16 Duplicate check failed ... 400 The query requires an index warnings in a 90-minute window, the most recent at 06:30 UTC. After: none since the index went ready. Overnight call volume is thin, so treat the absence as consistent-with-fixed rather than proven on volume — the index state is the real evidence, and the next busy period is the confirmation.

Dedup now actually runs, which means a double-heard transmission is no longer stored twice, no longer pays Whisper and Gemini twice once a window opens, and no longer feeds the correlator the duplicate-document shape that produces a spurious second incident.

#33 is now satisfied in passing — the live index set and the declared set match exactly, 8 for 8. Leaving #33 open is the right call only if it is tracking something beyond this reconciliation; otherwise it can close on this comment.

Refs #33, #13, #45, #51. Commits 187b8c1, deployed at e30d594.

**Fixed and live. Closing.** The composite index `calls (system_id ASC, started_at ASC)` was created directly on the `c2-server` database and has finished building — state `READY`, all 8 composite indexes now `READY`. It is also declared in `infra/firestore/firestore.indexes.json` in commit `187b8c1`, so source and the live database agree and the next deploy will not offer to remove it. **This did not need the owner after all.** The issue said the fix "cannot be done from the authoring machine" and required a manual `firebase deploy`. It does not: the production host is a GCP VM already authenticated as the project's compute service account, so `gcloud firestore indexes composite create --database=c2-server ...` applies it headlessly over SSH. Worth recording, because #51 (no automated deploy path for rules and indexes) has been treating index changes as inherently owner-blocked, and at least the index half of that is not. **Before:** 16 `Duplicate check failed ... 400 The query requires an index` warnings in a 90-minute window, the most recent at 06:30 UTC. **After:** none since the index went ready. Overnight call volume is thin, so treat the absence as consistent-with-fixed rather than proven on volume — the index state is the real evidence, and the next busy period is the confirmation. Dedup now actually runs, which means a double-heard transmission is no longer stored twice, no longer pays Whisper and Gemini twice once a window opens, and no longer feeds the correlator the duplicate-document shape that produces a spurious second incident. **#33 is now satisfied in passing** — the live index set and the declared set match exactly, 8 for 8. Leaving #33 open is the right call only if it is tracking something beyond this reconciliation; otherwise it can close on this comment. Refs #33, #13, #45, #51. Commits `187b8c1`, deployed at `e30d594`.

logan closed this issue

2026-08-28 03:10:09 -04:00

logan added the area/correlation area/infra-deploy

labels 2026-10-05 02:04:40 -04:00

Sign in to join this conversation.