Correlation: rules first, one cheap LLM call only when ambiguous; remove the cheap/smart consensus tiebreak #299

New Issue

2026-10-27T20:03:49-04:00

logan commented

2026-10-27 20:03:49 -04:00

From the 2026-10-27 review (#296). The consensus path (upload.py _correlate_with_consensus) runs a cheap Gemini call on every scene and a smart one on disagreement. The LLM decides only 3.5% of links (#35), and the tiebreaker manufactures incidents (#115). It is likely the main AI cost.

New shape: the rules tier decides confident links/new incidents, and a single cheap LLM call runs only in the ambiguous band. Section issues: #115, #134, #86, #88, #103, #138, #91, #35, #27, #155, #189, #190.

From the 2026-10-27 review (#296). The consensus path (upload.py _correlate_with_consensus) runs a cheap Gemini call on every scene and a smart one on disagreement. The LLM decides only 3.5% of links (#35), and the tiebreaker manufactures incidents (#115). It is likely the main AI cost. New shape: the rules tier decides confident links/new incidents, and a single cheap LLM call runs only in the ambiguous band. Section issues: #115, #134, #86, #88, #103, #138, #91, #35, #27, #155, #189, #190.

logan added the

type

feature

area/correlation area/ai-pipeline

priority

p1

labels 2026-10-27 20:03:49 -04:00

logan referenced this issue

2026-11-08 01:07:48 -05:00

Correlation: make the P25 radio ID (srcaddr) a first-class link, and measure how often calls carry it #304

logan commented

2026-11-08 21:51:06 -05:00

Fixed in 3246fc9, deployed to prod as 4fab959 on 2026-11-08 (/health confirms).

Fixed in 3246fc9, deployed to prod as 4fab959 on 2026-11-08 (/health confirms).

logan removed the

priority

p1

label 2026-11-08 21:51:06 -05:00

logan closed this issue

2026-11-08 21:51:07 -05:00

logan commented

2026-11-08 22:45:49 -05:00

Replay scoring after deploy 4fab959 (2026-11-08). These are reuse-mode runs, so each one keeps the September run's extraction fixed and re-runs correlation only. Scores are pairwise P/R/F1 against the hand-labelled answer keys.

Window	Before (`e90a73f`, Gemini consensus)	After (`4fab959`, rules-first + gpt-4o-mini)
W1 09-22 10-12 ET	P 0.947 R 0.695 F1 0.801	P 0.849 R 0.664 F1 0.745
W2 09-21 18-20 ET (held out)	P 0.937 R 0.810 F1 0.869	P 0.482 R 0.480 F1 0.481

Cost per run went from about $0.06 to $0.03. There were no AI failures, so these are real decisions and not fallbacks.

Where it went wrong in W2: there were wrong links in both directions.

Ossining Post 4 stops were merged with the knife-menacing incident.
MTA Pelham, Riverdale and Cortlandt were cross-linked.
The 17-call knife incident was split apart.
Most bad links came through llm or fast/disambig.

Not tested: whether the drop comes from the rules-first change (#299) or from gpt-4o-mini replacing Gemini. A plain revert would not restore 0.869 either, because the old path needs Gemini, which is unfunded.

Live AI is still off on prod, so nothing user-facing changed.

Replay scoring after deploy 4fab959 (2026-11-08). These are reuse-mode runs, so each one keeps the September run's extraction fixed and re-runs correlation only. Scores are pairwise P/R/F1 against the hand-labelled answer keys. | Window | Before (e90a73f, Gemini consensus) | After (4fab959, rules-first + gpt-4o-mini) | |---|---|---| | W1 09-22 10-12 ET | P 0.947 R 0.695 F1 0.801 | P 0.849 R 0.664 F1 0.745 | | W2 09-21 18-20 ET (held out) | P 0.937 R 0.810 F1 0.869 | P 0.482 R 0.480 F1 0.481 | Cost per run went from about $0.06 to $0.03. There were no AI failures, so these are real decisions and not fallbacks. Where it went wrong in W2: there were wrong links in both directions. - Ossining Post 4 stops were merged with the knife-menacing incident. - MTA Pelham, Riverdale and Cortlandt were cross-linked. - The 17-call knife incident was split apart. - Most bad links came through llm or fast/disambig. Not tested: whether the drop comes from the rules-first change (#299) or from gpt-4o-mini replacing Gemini. A plain revert would not restore 0.869 either, because the old path needs Gemini, which is unfunded. Live AI is still off on prod, so nothing user-facing changed.

logan reopened this issue

2026-11-08 22:45:56 -05:00

logan added the

priority

p1

label 2026-11-08 22:45:57 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/server-26#299