Pin node images to the checked-out source tree — fix v1 code / :latest image skew (#19) #31

2026-10-04T18:10:07-04:00

logan commented

2026-10-04 18:10:07 -04:00

Fixes #19 (install.sh hazard D3).

Problem: nodes check out v1 code and bind-mount it over images tagged :latest/:stable, which CI builds from main. A dependency or op25_addons/tk_drb_conv.py change on main therefore runs under v1 code without any warning.

Fix: pin each image to the source tree it was built from. The tag is src-<git tree hash of the image's build directory>.

CI: every build-*.yml adds the src- tag. :latest, :stable and the version tag are unchanged, so dev and old nodes keep working.
Node: scripts/image-tags.sh computes the same hash from the checkout (git rev-parse <commit>:<dir>) and writes DRB_TAG_* into .env. docker-compose.yml uses ${DRB_TAG_*:-latest|stable}, so manual docker compose / make up-prebuilt on the node use the same pins.
Why a tree hash and not a commit SHA: each image only rebuilds when its own directory changes, so most commits have no image built at that SHA. An unchanged directory keeps the same tree hash, so the tag stays exact.
install.sh: pins right after checkout. After compose pull it checks that every pinned image is present, because compose only warns when a service with a build: section fails to pull. If one is missing it stops with a clear message.
drb-update.sh: in pull mode it pulls the target commit's pinned images before checking out, using the target's own compose file. A missing image therefore leaves the node untouched. Before, uvicorn --reload picked up new code on old images as soon as the checkout landed. After the pull it checks out, writes the pins, recreates the containers and removes src- tags that are no longer used. Build mode tags its local builds with the same pins.
New release-images.yml: runs on v* tag pushes and on manual dispatch. For each image it builds and pushes src-<tree> only if that tag is missing from the registry.

Owner rollout. The v1 tag is not moved by this PR.

Merge.
Run Release images by hand on main. This PR changes none of the four image directories, so main's trees are the same as v1's and every src- image a node will ask for gets published. op25 should mostly come from cache.
When ready, re-cut v1 onto the merge commit. The tag push runs Release images again, and it skips everything already published.
Transition: the first update after the re-cut still runs the old drb-update.sh, which pulls :latest as today. Every update after that is pinned.

Out of scope, existing issues: #17 (no signature or digest verification; this pins by tag, not digest) and #28 (images are arm64-only; release-images.yml keeps linux/arm64).

Verification: bash -n on all scripts. All workflows and the compose file parse as YAML. I ran image-tags.sh against this repo: the tags match git rev-parse HEAD:<dir>, .env keeps mode 0600 and its other keys, and an older commit gives different src- tags. Not verified: the compose runtime and the Gitea Actions runs. There is no docker on the authoring box, so the first real update needs watching. No Python touched. edge-node pytest: 287 passed, 9 skipped. flake8 is clean.

🤖 Generated with Claude Code

Fixes #19 (install.sh hazard D3). **Problem:** nodes check out `v1` code and bind-mount it over images tagged `:latest`/`:stable`, which CI builds from **main**. A dependency or `op25_addons/tk_drb_conv.py` change on main therefore runs under v1 code without any warning. **Fix: pin each image to the source tree it was built from.** The tag is `src-<git tree hash of the image's build directory>`. - **CI:** every `build-*.yml` adds the `src-` tag. `:latest`, `:stable` and the version tag are unchanged, so dev and old nodes keep working. - **Node:** `scripts/image-tags.sh` computes the same hash from the checkout (`git rev-parse <commit>:<dir>`) and writes `DRB_TAG_*` into `.env`. `docker-compose.yml` uses `${DRB_TAG_*:-latest|stable}`, so manual `docker compose` / `make up-prebuilt` on the node use the same pins. - **Why a tree hash and not a commit SHA:** each image only rebuilds when its own directory changes, so most commits have no image built at that SHA. An unchanged directory keeps the same tree hash, so the tag stays exact. - **`install.sh`:** pins right after checkout. After `compose pull` it checks that every pinned image is present, because compose only *warns* when a service with a `build:` section fails to pull. If one is missing it stops with a clear message. - **`drb-update.sh`:** in pull mode it pulls the **target** commit's pinned images *before* checking out, using the target's own compose file. A missing image therefore leaves the node untouched. Before, uvicorn `--reload` picked up new code on old images as soon as the checkout landed. After the pull it checks out, writes the pins, recreates the containers and removes `src-` tags that are no longer used. Build mode tags its local builds with the same pins. - **New `release-images.yml`:** runs on `v*` tag pushes and on manual dispatch. For each image it builds and pushes `src-<tree>` only if that tag is missing from the registry. **Owner rollout. The v1 tag is not moved by this PR.** 1. Merge. 2. Run **Release images** by hand on `main`. This PR changes none of the four image directories, so main's trees are the same as v1's and every `src-` image a node will ask for gets published. op25 should mostly come from cache. 3. When ready, re-cut `v1` onto the merge commit. The tag push runs Release images again, and it skips everything already published. 4. Transition: the first update after the re-cut still runs the *old* `drb-update.sh`, which pulls `:latest` as today. Every update after that is pinned. **Out of scope, existing issues:** #17 (no signature or digest verification; this pins by tag, not digest) and #28 (images are arm64-only; `release-images.yml` keeps `linux/arm64`). **Verification:** `bash -n` on all scripts. All workflows and the compose file parse as YAML. I ran `image-tags.sh` against this repo: the tags match `git rev-parse HEAD:<dir>`, `.env` keeps mode 0600 and its other keys, and an older commit gives different `src-` tags. **Not verified:** the compose runtime and the Gitea Actions runs. There is no docker on the authoring box, so the first real update needs watching. No Python touched. edge-node pytest: 287 passed, 9 skipped. flake8 is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

logan added 1 commit 2026-10-04 18:10:08 -04:00

Pin node images to the checked-out source tree (node-26#19)

CI / lint (push) Successful in 10s

Details

CI / test-op25 (push) Successful in 16s

Details

CI / test (push) Failing after 49s

Details

CI / lint (pull_request) Successful in 8s

Details

CI / test-op25 (pull_request) Successful in 18s

Details

CI / test (pull_request) Failing after 46s

Details

6c8289f3f3

Nodes checked out v1 code but pulled :latest/:stable images built from main.
Each image is now tagged src-<git tree hash of its build directory>: CI adds
that tag in every build-*.yml, and scripts/image-tags.sh computes the same
value on the node and writes it into .env as DRB_TAG_*, which
docker-compose.yml uses (falls back to :latest/:stable for local dev).
Tree hashes stay exact even though each image only rebuilds when its own
directory changes.

- install.sh pins after checkout and fails loudly if a pinned image is
  missing (compose pull only warns for services with a build: section).
- drb-update.sh pulls the target commit's pinned images BEFORE checkout,
  using the target's own compose file, so a missing image changes nothing
  (uvicorn --reload would otherwise run new code on old images). It then
  checks out, writes the pins, recreates, and removes superseded src- tags.
- New "Release images" workflow (v* tags + manual dispatch) builds any
  src- image a release ref is missing; it skips ones already published.

The live v1 tag is not moved.

Refs #19

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

logan referenced this pull request

2026-10-04 18:10:37 -04:00

Code/image skew (install.sh D3): updater checks out v1 code but pulls :latest images built from main #19

logan added 1 commit 2026-10-04 18:17:28 -04:00

Fix updater tests for image pinning; harden helper load (node-26#19)

CI / lint (push) Successful in 9s

Details

CI / lint (pull_request) Successful in 9s

Details

CI / test-op25 (push) Successful in 17s

Details

CI / test-op25 (pull_request) Successful in 18s

Details

CI / test (push) Successful in 56s

Details

CI / test (pull_request) Successful in 57s

Details

e7b4aef142

- test_host_updater: stub git/docker now model cat-file/show/rev-parse
  --verify and compose config/image inspect. The two update-path tests
  were failing on Linux CI (skipped locally on macOS, no flock). Adds
  tests for pull-before-checkout + .env pins, a missing pinned image
  leaving the node unchanged, and a pre-pinning target still updating.
- drb-update.sh: load image-tags.sh via a variable + eval and confirm the
  functions exist; `source <(git show ...)` "succeeded" with nothing
  defined when git show failed, wedging every later update.
- install.sh: skip pinning (warn) when the checked-out ref predates
  scripts/image-tags.sh instead of dying on the missing file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

logan commented

2026-10-04 18:17:54 -04:00

Review pass (adversarial): pushed e7b4aef onto this branch.

CI was red on 6c8289f: CI / test failed — test_host_updater.py's stub git/docker didn't model the new pin path, so the two update-path tests broke (they were skipped on the macOS authoring box, no flock, so the reported 287/9-skipped never exercised drb-update.sh). Stubs fixed; added tests for pull-before-checkout + .env pins, missing pinned image => node unchanged, pre-pinning target still updates.
drb-update.sh: source <(git show ...) reports success even when git show fails, leaving pin=1 with no functions defined (every later update wedges). Now read into a variable, eval, and declare -F checks.
install.sh: a --ref older than #19 has no scripts/image-tags.sh; it used to die there. Now warns and uses that ref's compose defaults.

Local: edge-node pytest 299 passed, 0 skipped (flock shim), flake8 clean.

🤖 Generated with Claude Code

Review pass (adversarial): pushed `e7b4aef` onto this branch. - **CI was red on `6c8289f`**: `CI / test` failed — `test_host_updater.py`'s stub git/docker didn't model the new pin path, so the two update-path tests broke (they were skipped on the macOS authoring box, no `flock`, so the reported 287/9-skipped never exercised `drb-update.sh`). Stubs fixed; added tests for pull-before-checkout + `.env` pins, missing pinned image => node unchanged, pre-pinning target still updates. - `drb-update.sh`: `source <(git show ...)` reports success even when `git show` fails, leaving pin=1 with no functions defined (every later update wedges). Now read into a variable, `eval`, and `declare -F` checks. - `install.sh`: a `--ref` older than #19 has no `scripts/image-tags.sh`; it used to die there. Now warns and uses that ref's compose defaults. Local: edge-node pytest 299 passed, 0 skipped (flock shim), flake8 clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

logan referenced this pull request

2026-10-04 18:17:55 -04:00

install.sh: enrollment token never on the command line (#18) #30

logan added 1 commit 2026-10-04 18:22:45 -04:00

Merge branch 'main' into fix/19-image-tag-skew

CI / lint (push) Successful in 9s

Details

CI / test-op25 (push) Successful in 20s

Details

CI / lint (pull_request) Successful in 11s

Details

CI / test-op25 (pull_request) Successful in 23s

Details

CI / test (push) Successful in 56s

Details

CI / test (pull_request) Successful in 54s

Details

085139e171

logan merged commit 737e7c9bb1 into main

2026-10-04 18:26:01 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: logan/node-26#31