The correlation evaluation tooling runs against a stale image, so every dump taken via docker compose run silently measures old code — it broke outright on 2026-09-01 #101

New Issue

2026-09-01T02:58:58-04:00

logan commented

2026-09-01 02:58:58 -04:00

Found on the unattended run of 2026-09-01 when the standing correlation dump (#57) failed outright. Worked around this run; the underlying defect stands and it silently corrupts measurement rather than failing loudly.

Symptom

/opt/drb/aiwin/corr_dump.sh died with:

TypeError: debug_correlation() got an unexpected keyword argument 'ai_systems_only'

ai_systems_only was added to debug_correlation in 140dfbf, which is an ancestor of live b722223. So the dump was executing code older than production.

Root cause — confirmed by direct comparison

Both run on the same host, at the same moment:

RUNNING CONTAINER: (limit, orphan_hours, ai_systems_only, _)   <- current, correct
RUN IMAGE:         (limit, orphan_hours, _)                    <- predates 140dfbf

The long-running c2-core container has current code. The image that docker compose run --rm --no-deps c2-core resolves to on the VM is stale. The wrapper does not bind-mount /opt/drb/drb-c2-core/app, so run executes whatever the host's locally-resolved image tag contains — which has drifted behind the image the live container was actually started from.

Why this is the dangerous kind of bug

It only failed loudly because a signature changed. Any drift that is not a signature change — a scoring threshold, a prompt, a gate condition, a veto literal — produces a dump that runs cleanly and reports numbers from old code. Nothing in the output says which build produced it.

Everything downstream inherits that:

#57's standing evaluation procedure is specified to dump via docker compose run. Every dump it has ever taken this way is of unverified provenance.
The saved correlation-logs-<date>.json series exists to be compared run over run. A trend line across mixed builds is not a trend line. This is exactly the "trend matters more than the snapshot" rule the procedure is built on, silently violated.
It compounds with #96: one instrument blends scenes within a call, the other blends builds across runs.

Workaround applied this run

The 2026-09-01 dump was taken by piping the script into the live container over exec -T instead of run --rm, which is guaranteed to be the deployed build. Saved as Version 5C/correlation-logs-2026-09-01.json (100 incidents, 187 linked calls, 22 orphans, orphan_scan_truncated: false). Helper left on the VM at /opt/drb/aiwin/corr_dump_exec.py.

Fix shape

Make the evaluation tooling use exec against the running container, not run --rm, and update #57's procedure text along with it. Cheapest and removes the whole class.
If run must stay (e.g. to avoid loading the live process), pin it to the exact deployed tag — TAG=$(curl -s .../health | ...) — rather than whatever the host resolves.
Stamp the build SHA into the dump output. A dump that cannot say which code produced it should not be comparable to another one. This is the cheap change that makes the failure loud instead of silent.
Separately, work out why the host's image tag drifted behind the running container at all — a docker image prune / re-pull hygiene issue on the VM, and it is the same registry-tag surface as the mixed-version risk in the CI credential issue filed alongside this.

Related: #57, #96, #35, #21.

Found on the unattended run of 2026-09-01 when the standing correlation dump (#57) failed outright. **Worked around this run; the underlying defect stands and it silently corrupts measurement rather than failing loudly.** ## Symptom `/opt/drb/aiwin/corr_dump.sh` died with: ``` TypeError: debug_correlation() got an unexpected keyword argument 'ai_systems_only' ``` `ai_systems_only` was added to `debug_correlation` in `140dfbf`, which is an ancestor of live `b722223`. So the dump was executing code older than production. ## Root cause — confirmed by direct comparison Both run on the same host, at the same moment: ``` RUNNING CONTAINER: (limit, orphan_hours, ai_systems_only, _) <- current, correct RUN IMAGE: (limit, orphan_hours, _) <- predates 140dfbf ``` **The long-running `c2-core` container has current code. The image that `docker compose run --rm --no-deps c2-core` resolves to on the VM is stale.** The wrapper does not bind-mount `/opt/drb/drb-c2-core/app`, so `run` executes whatever the host's locally-resolved image tag contains — which has drifted behind the image the live container was actually started from. ## Why this is the dangerous kind of bug It only failed loudly because a **signature** changed. Any drift that is not a signature change — a scoring threshold, a prompt, a gate condition, a veto literal — produces a dump that **runs cleanly and reports numbers from old code**. Nothing in the output says which build produced it. Everything downstream inherits that: - **#57's standing evaluation procedure** is specified to dump via `docker compose run`. Every dump it has ever taken this way is of unverified provenance. - The saved `correlation-logs-<date>.json` series exists to be compared run over run. **A trend line across mixed builds is not a trend line.** This is exactly the "trend matters more than the snapshot" rule the procedure is built on, silently violated. - It compounds with #96: one instrument blends scenes within a call, the other blends builds across runs. ## Workaround applied this run The 2026-09-01 dump was taken by piping the script into the **live** container over `exec -T` instead of `run --rm`, which is guaranteed to be the deployed build. Saved as `Version 5C/correlation-logs-2026-09-01.json` (100 incidents, 187 linked calls, 22 orphans, `orphan_scan_truncated: false`). Helper left on the VM at `/opt/drb/aiwin/corr_dump_exec.py`. ## Fix shape 1. **Make the evaluation tooling use `exec` against the running container**, not `run --rm`, and update #57's procedure text along with it. Cheapest and removes the whole class. 2. If `run` must stay (e.g. to avoid loading the live process), pin it to the exact deployed tag — `TAG=$(curl -s .../health | ...)` — rather than whatever the host resolves. 3. **Stamp the build SHA into the dump output.** A dump that cannot say which code produced it should not be comparable to another one. This is the cheap change that makes the failure loud instead of silent. 4. Separately, work out why the host's image tag drifted behind the running container at all — a `docker image prune` / re-pull hygiene issue on the VM, and it is the same registry-tag surface as the mixed-version risk in the CI credential issue filed alongside this. Related: #57, #96, #35, #21.

logan referenced this issue

2026-09-01 02:59:24 -04:00

P0: standing correlation-quality evaluation procedure (scheduled AI window) #57

logan referenced this issue

2026-09-13 17:38:39 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — CIO draft #144

logan referenced this issue

2026-09-13 18:49:36 -04:00

Board 2026-09-13 — Beachhead checkpoint miss and dev-progress review — FINAL MINUTES #146

Sign in to join this conversation.