Skip to content

feat(omnigraph): pin 0.14.0, storage format 16 and HTTP contract 0.14 - #573

Open
blarghmatey wants to merge 5 commits into
mainfrom
feat/omnigraph-0.14-pin
Open

blarghmatey wants to merge 5 commits into
mainfrom
feat/omnigraph-0.14-pin

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

What are the relevant tickets?

witan task tk-bump-the-omnigraph-pin-to-0-14-contract-0-14-sto-e7e5c7, the pin-bump step of the omnigraph 0.14 upgrade (spec in docs/internals/design/omnigraph-0-14-upgrade-spec.md, rollout step 3). Follows #564, #568, #569 and #571.

Description (What does it do?)

The installer and both images pin omnigraph 0.13.0 (storage format 14, HTTP contract 0.13). This moves all three to the released v0.14.0 assets (tag commit 60bfd19). The digests were hashed locally and matched against each published .sha256 and the release API. omnigraph version on the binary reads internal-schema 16 (serves v14 to v16) and http-contract 0.14, so _OMNIGRAPH_INTERNAL_SCHEMA and both image labels move 14 to 16 (the release notes say 15) and HTTP_API_CONTRACT moves to 0.14. The data-tier image now selects the patched CLI from #569 (OMNIGRAPH_CLI_SOURCE=patched), which check-omnigraph-pins requires once the release tag equals the patch base. The witan image and witan setup keep the stock release binary.

Merging this is one-way for any graph once 0.14 publishes to it. 0.14 opens a format-14 graph in place, but the first write a 0.14 binary makes stamps that branch 16, and every 0.13 binary refuses it from then on. A 0.13 server and a 0.14 client also refuse each other's contract header. So this has to merge in the cutover window, with the data tier moving before the witan clients in each environment, and there is no rollback short of restoring the pre-cutover backup.

Two client changes ride along, because the 0.14 CLI breaks the existing failure handling:

  • 0.14 prints the resolved target on stderr for every call (target <store> (inline): direct graph …), so stderr is never empty. _cli_failure classified stderr whenever it had text, so every read's JSON error object on stdout (a recovery barrier, a conflict, an unreachable server) was classified from the target line alone and reported as fatal. This is what failed test_apply_schema_upgrades_an_old_store. The error object now wins whenever there is one.
  • _classify_cli_error reads 0.14's exits (command_outcome.rs exit_for at v0.14.0). 2 (partial or unknown effects) is never retried, and a write raises WriteIndeterminate; it is checked before the retryable markers so a dropped connection can't turn it into a retry. 64 (a refused command line) is fatal. 75 now covers any refusal with effects: none, action: retry, so one that carries a 503 gets the unavailable-server budget rather than the admission cap's.
  • The target lines 0.14 prints before every command (target <name> (<provenance>): ..., target write scope: branch=<name>) are kept in the message but left out of the classified text. They carry graph ids, branches and paths, and 0.13's --quiet that suppressed them is gone.

The 0.13-worded markers needed no change: 0.14 prints the same sentences with 0.14 in place of 0.13, checked on the binary and in graph_http.rs. Both servers' witan-core floors move to >=0.50 for the new contract.

Test changes for 0.14:

  • test_binary_contract.py gets a format-16 row, measured on 0.14.0 (every surface matches format 14), and addresses upgrade with --target as witan does.
  • testsupport/omnigraph_server.py applies and force-unlocks with 0.14's --target, so the served tests run against a 0.14 server with the 0.14 contract.
  • The bridge-recreate test used the drop of an indexed property, which 0.14 now does in place. It uses a String? to String change instead, which both 0.13.0 and 0.14.0 refuse in place (OG-MF-106 on 0.14.0; the test passes against both binaries).
  • The s3 export proxy test pins a pre-0.14 CLI, since stock 0.14 opens no s3:// store and witan-core already tests that refusal.

How can this be tested?

Run against the 0.14.0 binaries (a scratch HOME whose ~/.local/bin holds the release's omnigraph and omnigraph-server, TMPDIR on disk):

  • just test-all: witan-core 888 passed, 7 skipped; witan-council 1801 passed, 4 skipped, 1 failed; witan-code 711 passed, 1 failed; agent-config-kit 564 passed; ol-agent-kit 1 passed. The two failures are harness-only: test_pre_upgrade_candidates_exclude_the_current_binary (two omnigraph binaries on PATH under the scratch HOME) and test_rebuild_refuses_a_file_path_with_real_git (a long TMPDIR wraps the asserted phrase). Both pass in the default environment. The served group-commit and branch-view tests ran against a real 0.14 server.
  • The bridge tests in witan-code/tests/test_store.py also pass against the 0.13.0 binary.
  • just check-omnigraph-pins, just check-omnigraph-format, just check-versions, just docs-check and uv run prek run --all-files pass.
  • just check-core-floor fails: the floors say >=0.50 and the workspace witan-core is still 0.49.0. It passes once witan-core is bumped to 0.50.0 (just bump witan-core minor), which this PR leaves to whoever merges it.

An independent review found nothing high or medium. One low it raised is left for tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca: the HTTP path still retries every 409, and a 0.14 server can answer an untyped conflict after an owned write with 409 and disposition: {effects: unknown} (asserted in upstream omnigraph-server/src/lib.rs tests at v0.14.0). The CLI path now refuses to retry the same condition (exit 2). The common race is a typed read_set_conflict with effects: none, so it stays safe, but how often the untyped case happens is unverified.

Not verified here: the Concourse image builds (the patched CLI stage was qualified by omnigraph-cli-patch.yml in #569), and anything against a deployed data tier. The cutover itself is tk-cut-over-ci-qa-then-production-to-omnigraph-0-14-be7663.

Checklist:

  • Bump witan-core to 0.50.0 (just bump witan-core minor) so check-core-floor passes
  • Merge only in the cutover window, with the data tier moving before the witan clients

https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B

Copilot AI balanced review requested due to automatic review settings October 29, 2026 17:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Writes with unknown effects can be replayed, the core-version mismatch leaves dependency checks failing, and paths containing spaces can cause incorrect retries.

3 open issues
What changed in this PR

Upgrades Witan’s Omnigraph integration to 0.14.0, aligning the installer, container images and HTTP client with storage format 16 and contract 0.14. This follows the earlier CLI-addressing, patched-S3 and remote-branch work and requires a coordinated data-tier-first cutover.

Changes:

  • Updates binary pins, checksums and image labels; enables the patched data-tier CLI.
  • Adapts CLI error classification to 0.14 output and exit codes.
  • Raises both servers’ core dependency floors and updates compatibility tests and changelog fragments.
File Description
testsupport/​omnigraph_server.py Uses version-aware cluster targeting in served tests.
packages/​witan-core/​witan_core/​omnigraph.py Handles 0.14 error output, target reports and exit codes.
packages/​witan-core/​witan_core/​omnigraph_install.py Pins 0.14.0 assets and declares storage format 16.
packages/​witan-core/​witan_core/​omnigraph_http.py Selects contract 0.14 and adds an unknown-outcome kind.
packages/​witan-core/​tests/​test_omnigraph.py Covers 0.14 CLI failure classification.
packages/​witan-core/​tests/​test_omnigraph_http.py Updates HTTP contract expectations.
packages/​witan-core/​tests/​test_binary_contract.py Adds format-16 expectations and target-aware upgrade calls.
packages/​witan-core/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Records core compatibility and error-handling changes.
mcp/​servers/​witan/​tests/​test_remote_proxy.py Explicitly selects the legacy dialect for S3 export coverage.
mcp/​servers/​witan/​tests/​test_group_commit_served.py Uses shared HTTP contract constants.
mcp/​servers/​witan/​pyproject.toml Raises the core dependency floor to 0.50.
mcp/​servers/​witan/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Documents the dependency and image updates.
mcp/​servers/​witan-code/​tests/​test_store.py Uses a schema change that still requires bridge recreation.
mcp/​servers/​witan-code/​pyproject.toml Raises the code server’s core dependency floor to 0.50.
mcp/​servers/​witan-code/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Documents the new compatibility floor.
docker/​witan.Dockerfile Updates CLI pins, checksums and storage-format label.
docker/​omnigraph-server.Dockerfile Updates server pins and selects the patched CLI.
docker/​omnigraph_server_entrypoint.py Updates documentation for the default 0.14 image.
.github/​workflows/​omnigraph-cli-patch.yml Updates comments describing patched-image builds.

Review effort: Balanced

Comment thread mcp/servers/witan/pyproject.toml Resolved
The installer and both images pinned omnigraph 0.13.0 (format 14, HTTP
contract 0.13). This moves all three to the released v0.14.0 assets. The
binary reports "internal-schema 16 (serves v14 to v16)" and http-contract
0.14, so `_OMNIGRAPH_INTERNAL_SCHEMA` moves to 16 (the release notes say
15) and `HTTP_API_CONTRACT` to 0.14. The data-tier image selects the
patched CLI, which `check-omnigraph-pins` requires once the release tag
equals the patch base. Both servers' witan-core floors move to 0.50.

Two client changes the 0.14 CLI needs:

- 0.14 prints the resolved target on stderr for every call, so stderr is
  never empty, and `_cli_failure` classified stderr whenever it had text.
  Every read's error object on stdout (recovery barrier, conflict,
  unreachable server) was then classified from the target line alone and
  reported as fatal. The error object now wins whenever there is one.
- `_classify_cli_error` reads the new exits: 2 (partial or unknown
  effects) is never retried and a write raises WriteIndeterminate, 64 (a
  refused command line) is fatal, and 75 from a 503 gets the
  unavailable-server budget rather than the admission cap's.

Tests: a format-16 row in the binary contract table (measured on 0.14.0,
same as 14's), `upgrade` addressed by `--target`, the served-server
helper applies and unlocks with 0.14's `--target`, the bridge-recreate
test uses a type change 0.14 still refuses in place (it now drops an
indexed property in place), and the s3 export test pins a pre-0.14 CLI.

The first write a 0.14 binary makes to a format-14 graph stamps it 16,
and 0.13 binaries then refuse it. This merges only in the cutover window,
with the data tier moving before the witan clients.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
…fication

0.14 prints `target <name> (<provenance>): ...` and `target write scope:
branch=<name>` on stderr before every command, and 0.13's `--quiet`
that suppressed them is gone. Those lines carry the graph id, branch and
paths, so a marker without spaces (`recovery_required`,
`too_many_requests`) could match a name and misroute an ordinary
failure. The message keeps them; only the classified text drops them.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
The witan-policy CI job installs the pinned omnigraph, so with the pin at
0.14.0 policy/check.sh failed at its first cluster command with
"target_required". 0.14 takes the cluster root as --target on cluster apply
(--config is only the source) and replaced --cluster with --target on policy
validate and policy test.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
Both servers' floors move to witan-core>=0.50 in this PR, so check-core-floor needs the workspace at 0.50.0. Collects the 0.14 pin fragment and #572's StoreQuarantined removal.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
@blarghmatey
blarghmatey force-pushed the feat/omnigraph-0.14-pin branch from d95aa41 to 47a9bdf Compare October 29, 2026 19:24
…ects landed

0.14 answers a /load whose fork published before its rows were refused with
HTTP 409 and disposition effects "partial". The 409 rule classified that
RETRYABLE, so the retry loop could replay a write that had already changed
the graph. A retried kind (409/429/503) on a non-idempotent request now
becomes OUTCOME_UNKNOWN when the disposition says partial or unknown, and
the write raises WriteIndeterminate. FATAL responses are left alone,
because 0.14 reports "unknown" for every owned write that isn't a typed
refusal (a plain 400 included) and they aren't retried anyway.

The 0.14 target-report regex matched the label as \S+, so an inline path
with a space left the line in the classified text, where a marker such as
too_many_requests could turn a precondition failure into a retried
admission cap. It now matches up to the provenance delimiter.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B

@feoh feoh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks right. I checked the exit-code mapping against upstream exit_for at v0.14.0, and the digests against the release API. Two merge-time problems are inline. Minor items:

  • The body is out of date: the witan-core bump is already done (47a9bdf), and f391196 fixes the deferred "HTTP still retries every 409" low, so tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca may be closable.
  • Unchanged code, but it bites now: _download_omnigraph replaces any non-pinned binary, so an older witan's setup puts 0.13 back over 0.14. That build then refuses local stores already stamped 16. Recoverable from omnigraph-0.14.0, but there's no warning.

[project]
name = "witan-core"
version = "0.49.0"
version = "0.50.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging this publishes witan-core 0.50.0 to PyPI (publish-witan-core.yml). Published witan-council 0.45.0 and witan-code 0.27.1 allow witan-core>=0.46,<1, and their versions don't change here, so the >=0.50 floors never ship. A fresh install or upgrade resolves 0.50, which sends contract 0.14 to QA and Production while they're still on 0.13 (400 api_contract_mismatch). Suggest moving this bump and the two floor changes to a release PR after Production is cut over. The images build from the workspace, so they don't need it.

# ARG not re-declared in the stage expands to empty, which builds a clean image
# advertising no format at all.
ARG OMNIGRAPH_INTERNAL_SCHEMA=14
ARG OMNIGRAPH_INTERNAL_SCHEMA=16

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging auto-deploys CI for both pulumi-omnigraph and pulumi-witan, in no fixed order. With this label at 16, the omnigraph CI preview fails validate_image_internal_schema (ol-infrastructure omnigraph/storage.py:182; every stack is pinned to 14/fmt14). witan has no such guard, so it deploys a 0.14 client against a 0.13 data tier. I found no open ol-infrastructure PR for the D2 guard change. Land it first, and pause witan's CI deploy until the data tier is on 0.14.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants