Repository navigation
feat(omnigraph): pin 0.14.0, storage format 16 and HTTP contract 0.14 - #573
blarghmatey wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Writes with unknown effects can be replayed, the core-version mismatch leaves dependency checks failing, and paths containing spaces can cause incorrect retries.
3 open issues
What changed in this PR
Upgrades Witan’s Omnigraph integration to 0.14.0, aligning the installer, container images and HTTP client with storage format 16 and contract 0.14. This follows the earlier CLI-addressing, patched-S3 and remote-branch work and requires a coordinated data-tier-first cutover.
Changes:
- Updates binary pins, checksums and image labels; enables the patched data-tier CLI.
- Adapts CLI error classification to 0.14 output and exit codes.
- Raises both servers’ core dependency floors and updates compatibility tests and changelog fragments.
| File | Description |
|---|---|
| testsupport/omnigraph_server.py | Uses version-aware cluster targeting in served tests. |
| packages/witan-core/witan_core/omnigraph.py | Handles 0.14 error output, target reports and exit codes. |
| packages/witan-core/witan_core/omnigraph_install.py | Pins 0.14.0 assets and declares storage format 16. |
| packages/witan-core/witan_core/omnigraph_http.py | Selects contract 0.14 and adds an unknown-outcome kind. |
| packages/witan-core/tests/test_omnigraph.py | Covers 0.14 CLI failure classification. |
| packages/witan-core/tests/test_omnigraph_http.py | Updates HTTP contract expectations. |
| packages/witan-core/tests/test_binary_contract.py | Adds format-16 expectations and target-aware upgrade calls. |
| packages/witan-core/changelog.d/20261029_125338_blarghmatey_omnigraph_0_14_pin.md | Records core compatibility and error-handling changes. |
| mcp/servers/witan/tests/test_remote_proxy.py | Explicitly selects the legacy dialect for S3 export coverage. |
| mcp/servers/witan/tests/test_group_commit_served.py | Uses shared HTTP contract constants. |
| mcp/servers/witan/pyproject.toml | Raises the core dependency floor to 0.50. |
| mcp/servers/witan/changelog.d/20261029_125338_blarghmatey_omnigraph_0_14_pin.md | Documents the dependency and image updates. |
| mcp/servers/witan-code/tests/test_store.py | Uses a schema change that still requires bridge recreation. |
| mcp/servers/witan-code/pyproject.toml | Raises the code server’s core dependency floor to 0.50. |
| mcp/servers/witan-code/changelog.d/20261029_125338_blarghmatey_omnigraph_0_14_pin.md | Documents the new compatibility floor. |
| docker/witan.Dockerfile | Updates CLI pins, checksums and storage-format label. |
| docker/omnigraph-server.Dockerfile | Updates server pins and selects the patched CLI. |
| docker/omnigraph_server_entrypoint.py | Updates documentation for the default 0.14 image. |
| .github/workflows/omnigraph-cli-patch.yml | Updates comments describing patched-image builds. |
Review effort: Balanced
The installer and both images pinned omnigraph 0.13.0 (format 14, HTTP contract 0.13). This moves all three to the released v0.14.0 assets. The binary reports "internal-schema 16 (serves v14 to v16)" and http-contract 0.14, so `_OMNIGRAPH_INTERNAL_SCHEMA` moves to 16 (the release notes say 15) and `HTTP_API_CONTRACT` to 0.14. The data-tier image selects the patched CLI, which `check-omnigraph-pins` requires once the release tag equals the patch base. Both servers' witan-core floors move to 0.50. Two client changes the 0.14 CLI needs: - 0.14 prints the resolved target on stderr for every call, so stderr is never empty, and `_cli_failure` classified stderr whenever it had text. Every read's error object on stdout (recovery barrier, conflict, unreachable server) was then classified from the target line alone and reported as fatal. The error object now wins whenever there is one. - `_classify_cli_error` reads the new exits: 2 (partial or unknown effects) is never retried and a write raises WriteIndeterminate, 64 (a refused command line) is fatal, and 75 from a 503 gets the unavailable-server budget rather than the admission cap's. Tests: a format-16 row in the binary contract table (measured on 0.14.0, same as 14's), `upgrade` addressed by `--target`, the served-server helper applies and unlocks with 0.14's `--target`, the bridge-recreate test uses a type change 0.14 still refuses in place (it now drops an indexed property in place), and the s3 export test pins a pre-0.14 CLI. The first write a 0.14 binary makes to a format-14 graph stamps it 16, and 0.13 binaries then refuse it. This merges only in the cutover window, with the data tier moving before the witan clients. Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
…fication 0.14 prints `target <name> (<provenance>): ...` and `target write scope: branch=<name>` on stderr before every command, and 0.13's `--quiet` that suppressed them is gone. Those lines carry the graph id, branch and paths, so a marker without spaces (`recovery_required`, `too_many_requests`) could match a name and misroute an ordinary failure. The message keeps them; only the classified text drops them. Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
The witan-policy CI job installs the pinned omnigraph, so with the pin at 0.14.0 policy/check.sh failed at its first cluster command with "target_required". 0.14 takes the cluster root as --target on cluster apply (--config is only the source) and replaced --cluster with --target on policy validate and policy test. Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
Both servers' floors move to witan-core>=0.50 in this PR, so check-core-floor needs the workspace at 0.50.0. Collects the 0.14 pin fragment and #572's StoreQuarantined removal. Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
d95aa41 to
47a9bdf
Compare
…ects landed 0.14 answers a /load whose fork published before its rows were refused with HTTP 409 and disposition effects "partial". The 409 rule classified that RETRYABLE, so the retry loop could replay a write that had already changed the graph. A retried kind (409/429/503) on a non-idempotent request now becomes OUTCOME_UNKNOWN when the disposition says partial or unknown, and the write raises WriteIndeterminate. FATAL responses are left alone, because 0.14 reports "unknown" for every owned write that isn't a typed refusal (a plain 400 included) and they aren't retried anyway. The 0.14 target-report regex matched the label as \S+, so an inline path with a space left the line in the classified text, where a marker such as too_many_requests could turn a precondition failure into a retried admission cap. It now matches up to the provenance delimiter. Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
feoh
left a comment
There was a problem hiding this comment.
Code looks right. I checked the exit-code mapping against upstream exit_for at v0.14.0, and the digests against the release API. Two merge-time problems are inline. Minor items:
- The body is out of date: the witan-core bump is already done (
47a9bdf), andf391196fixes the deferred "HTTP still retries every 409" low, sotk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3camay be closable. - Unchanged code, but it bites now:
_download_omnigraphreplaces any non-pinned binary, so an older witan'ssetupputs 0.13 back over 0.14. That build then refuses local stores already stamped 16. Recoverable fromomnigraph-0.14.0, but there's no warning.
| [project] | ||
| name = "witan-core" | ||
| version = "0.49.0" | ||
| version = "0.50.0" |
There was a problem hiding this comment.
Merging this publishes witan-core 0.50.0 to PyPI (publish-witan-core.yml). Published witan-council 0.45.0 and witan-code 0.27.1 allow witan-core>=0.46,<1, and their versions don't change here, so the >=0.50 floors never ship. A fresh install or upgrade resolves 0.50, which sends contract 0.14 to QA and Production while they're still on 0.13 (400 api_contract_mismatch). Suggest moving this bump and the two floor changes to a release PR after Production is cut over. The images build from the workspace, so they don't need it.
| # ARG not re-declared in the stage expands to empty, which builds a clean image | ||
| # advertising no format at all. | ||
| ARG OMNIGRAPH_INTERNAL_SCHEMA=14 | ||
| ARG OMNIGRAPH_INTERNAL_SCHEMA=16 |
There was a problem hiding this comment.
Merging auto-deploys CI for both pulumi-omnigraph and pulumi-witan, in no fixed order. With this label at 16, the omnigraph CI preview fails validate_image_internal_schema (ol-infrastructure omnigraph/storage.py:182; every stack is pinned to 14/fmt14). witan has no such guard, so it deploys a 0.14 client against a 0.13 data tier. I found no open ol-infrastructure PR for the D2 guard change. Land it first, and pause witan's CI deploy until the data tier is on 0.14.


What are the relevant tickets?
witan task
tk-bump-the-omnigraph-pin-to-0-14-contract-0-14-sto-e7e5c7, the pin-bump step of the omnigraph 0.14 upgrade (spec indocs/internals/design/omnigraph-0-14-upgrade-spec.md, rollout step 3). Follows #564, #568, #569 and #571.Description (What does it do?)
The installer and both images pin omnigraph 0.13.0 (storage format 14, HTTP contract 0.13). This moves all three to the released v0.14.0 assets (tag commit
60bfd19). The digests were hashed locally and matched against each published.sha256and the release API.omnigraph versionon the binary readsinternal-schema 16 (serves v14 to v16)andhttp-contract 0.14, so_OMNIGRAPH_INTERNAL_SCHEMAand both image labels move 14 to 16 (the release notes say 15) andHTTP_API_CONTRACTmoves to0.14. The data-tier image now selects the patched CLI from #569 (OMNIGRAPH_CLI_SOURCE=patched), whichcheck-omnigraph-pinsrequires once the release tag equals the patch base. The witan image andwitan setupkeep the stock release binary.Merging this is one-way for any graph once 0.14 publishes to it. 0.14 opens a format-14 graph in place, but the first write a 0.14 binary makes stamps that branch 16, and every 0.13 binary refuses it from then on. A 0.13 server and a 0.14 client also refuse each other's contract header. So this has to merge in the cutover window, with the data tier moving before the witan clients in each environment, and there is no rollback short of restoring the pre-cutover backup.
Two client changes ride along, because the 0.14 CLI breaks the existing failure handling:
target <store> (inline): direct graph …), so stderr is never empty._cli_failureclassified stderr whenever it had text, so every read's JSON error object on stdout (a recovery barrier, a conflict, an unreachable server) was classified from the target line alone and reported as fatal. This is what failedtest_apply_schema_upgrades_an_old_store. The error object now wins whenever there is one._classify_cli_errorreads 0.14's exits (command_outcome.rsexit_forat v0.14.0). 2 (partial or unknown effects) is never retried, and a write raisesWriteIndeterminate; it is checked before the retryable markers so a dropped connection can't turn it into a retry. 64 (a refused command line) is fatal. 75 now covers any refusal witheffects: none, action: retry, so one that carries a 503 gets the unavailable-server budget rather than the admission cap's.target <name> (<provenance>): ...,target write scope: branch=<name>) are kept in the message but left out of the classified text. They carry graph ids, branches and paths, and 0.13's--quietthat suppressed them is gone.The 0.13-worded markers needed no change: 0.14 prints the same sentences with
0.14in place of0.13, checked on the binary and ingraph_http.rs. Both servers' witan-core floors move to>=0.50for the new contract.Test changes for 0.14:
test_binary_contract.pygets a format-16 row, measured on 0.14.0 (every surface matches format 14), and addressesupgradewith--targetas witan does.testsupport/omnigraph_server.pyapplies and force-unlocks with 0.14's--target, so the served tests run against a 0.14 server with the 0.14 contract.String?toStringchange instead, which both 0.13.0 and 0.14.0 refuse in place (OG-MF-106on 0.14.0; the test passes against both binaries).s3://store and witan-core already tests that refusal.How can this be tested?
Run against the 0.14.0 binaries (a scratch
HOMEwhose~/.local/binholds the release'somnigraphandomnigraph-server,TMPDIRon disk):just test-all: witan-core 888 passed, 7 skipped; witan-council 1801 passed, 4 skipped, 1 failed; witan-code 711 passed, 1 failed; agent-config-kit 564 passed; ol-agent-kit 1 passed. The two failures are harness-only:test_pre_upgrade_candidates_exclude_the_current_binary(twoomnigraphbinaries on PATH under the scratch HOME) andtest_rebuild_refuses_a_file_path_with_real_git(a longTMPDIRwraps the asserted phrase). Both pass in the default environment. The served group-commit and branch-view tests ran against a real 0.14 server.witan-code/tests/test_store.pyalso pass against the 0.13.0 binary.just check-omnigraph-pins,just check-omnigraph-format,just check-versions,just docs-checkanduv run prek run --all-filespass.just check-core-floorfails: the floors say>=0.50and the workspace witan-core is still 0.49.0. It passes once witan-core is bumped to 0.50.0 (just bump witan-core minor), which this PR leaves to whoever merges it.An independent review found nothing high or medium. One low it raised is left for
tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca: the HTTP path still retries every 409, and a 0.14 server can answer an untyped conflict after an owned write with 409 anddisposition: {effects: unknown}(asserted in upstreamomnigraph-server/src/lib.rstests at v0.14.0). The CLI path now refuses to retry the same condition (exit 2). The common race is a typedread_set_conflictwitheffects: none, so it stays safe, but how often the untyped case happens is unverified.Not verified here: the Concourse image builds (the patched CLI stage was qualified by
omnigraph-cli-patch.ymlin #569), and anything against a deployed data tier. The cutover itself istk-cut-over-ci-qa-then-production-to-omnigraph-0-14-be7663.Checklist:
just bump witan-core minor) socheck-core-floorpasseshttps://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B