Skip to content

Build the relational engine for macOS, Windows and 3.14 - #42

Open
Manushpm8 wants to merge 1 commit into
mainfrom
engine-wheels-all-platforms
Open

Manushpm8 wants to merge 1 commit into
mainfrom
engine-wheels-all-platforms

Conversation

@Manushpm8

@Manushpm8 Manushpm8 commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

structured-data-models-client[relational] installs on Linux and nowhere else. The engine is a native build that ships no source distribution, so a platform with no wheel has nothing to fall back on:

ERROR: Could not find a version that satisfies the requirement
structured-data-models-relational-engine==1.0.0  (from versions: none)

A developer on a Mac can install the client but not the model it fronts. Found by installing 1.0.0 from PyPI across platforms after the release.

It was never the code

The sampler is 564 lines of standard-library C++ with no POSIX headers, and CMakeLists.txt has carried a macOS deployment target all along. What was Linux-only was the wheel matrix: [tool.cibuildwheel.linux] was the only platform table, and both the CI job and the release step ran cibuildwheel on Ubuntu runners, which can only produce manylinux wheels.

What changes

25 wheels where there were 8.

before after
Linux x86-64 / aarch64 3.10–3.13 3.10–3.14
macOS arm64 / x86-64 — 3.10–3.14
Windows AMD64 — 3.10–3.14

requires-python moves to <3.15 together with the classifiers, as the comment above it requires. -O3 is now guarded by if(NOT MSVC).

The release workflow built the engine inside the job that builds everything else, on one Linux machine. A wheel is native to the machine that builds it, so that job becomes three: build for the pure-Python distributions, engine-wheels across five runners, and assemble, which joins them and runs the checks build used to run. One runner per platform and architecture, native rather than emulated, so each wheel's smoke test runs on the machine it targets. That retires the QEMU step and with it a privileged container image in the release path. release is still the only job behind the protected pypi environment.

macos-15-intel replaces macos-13, which GitHub retired in December 2025.

Two things CI caught that review would not

The Windows wheels were built by the wrong compiler. An earlier revision set CMAKE_GENERATOR=Ninja to get past an MSBuild MAX_PATH failure. Ninja then picked up the runner's MinGW: The CXX compiler identification is GNU 14.2.0, C:/mingw64/bin/c++.exe, with libgcc_s_seh-1.dll, libstdc++-6.dll and libwinpthread-1.dll repaired into every Windows wheel — and the job passed. Green CI was not evidence the artifact was right.

Ninja is gone. The default Visual Studio generator selects MSVC on its own, and the actual problem — MSBuild's .tlog paths exceeding MAX_PATH under this repository's long directory names — is fixed in setup.py with a short build_base on win32, keyed by a digest of the package path so two checkouts never share a build tree. Windows now reports MSVC 19.44.35229.0 and ships msvcp140.dll.

A guard now checks the artifact, not the build. Because the failure mode was a passing build, every Windows wheel is inspected for MinGW runtimes, and an empty wheelhouse fails rather than reporting nothing found.

Also corrected

  • publish.yml accepted a release whose engine matrix was incomplete: it reduced filenames to distribution names, so four pure wheels plus a single engine wheel satisfied it. It now derives interpreters from the engine's own [tool.cibuildwheel] build and requires all five platform families.
  • .gitlab-ci.yml pinned cp310–cp313 in both matrix rows and in wheels-check.
  • The shipped diagnostic in rfm/backend/local/__init__.py, the engine README, RELEASING.md, the root README and the architecture/installation/prerequisites docs all named the old matrix. Corrected, and they now say CPython where only cp wheels exist — there is no sdist, so PyPy cannot install the engine.
  • test_an_extra_narrower_than_its_parent_says_so asserted '3.14' in source; once 3.14 became supported that string appears for an unrelated reason and the test proved nothing. It now reads the engine's upper bound and requires the client's prose to name it.

Verification

  • CI green on all 20 checks, including all five engine-wheels rows.
  • The 25 wheels CI produced were downloaded and inventoried; the Windows wheel contains the MSVC runtime and no MinGW.
  • CI-built macOS arm64 wheels for cp312 and cp314 were installed locally and the compiled sampler exercised.
  • The release matrix check was run against the real artifacts: it passes on the full set, and fails on a dropped platform (['cp310 win_amd64', ...]) and on a single dropped wheel (['cp314 macosx_.*_arm64']).
  • publish.yml only runs on a tag, so its wiring was checked statically: every artifact download resolves to an upload, and the job graph is build → engine-wheels → assemble → release.

Not a release

No version bump and no tag. This ships with whatever the next release is; the changelog entry sits under Unreleased. 1.0.0 is already published and its files are immutable, so the platforms it offers do not change.

🤖 Generated with Claude Code

@Manushpm8
Manushpm8 force-pushed the engine-wheels-all-platforms branch 5 times, most recently from 528c7ba to a290711 Compare October 18, 2026 05:59
`structured-data-models-client[relational]` installed on Linux and nowhere else. The engine
is a native build that ships no source distribution, so a platform with no
wheel has nothing to fall back on: pip reports "No matching distribution
found for structured-data-models-relational-engine" and stops. A developer on
a Mac could install the client but not the model it fronts.

Nothing about the engine required that. The sampler is 564 lines of standard
library C++ with no POSIX headers, and `CMakeLists.txt` has carried a macOS
deployment target all along. What was Linux-only was the wheel matrix:
`[tool.cibuildwheel.linux]` was the only platform table, and both the CI job
and the release step ran cibuildwheel on Ubuntu runners, which can only
produce manylinux wheels.

So the engine now builds for macOS on both architectures, for 64-bit Windows,
and for CPython 3.14 everywhere, alongside the manylinux wheels it already
built. 25 wheels where there were 8. `requires-python` moves to `<3.15` with
the classifiers, as the comment above it requires.

Two things this needed:

`-O3` was set unconditionally, which MSVC rejects; its release default is
already `/O2`. That made the file buildable by Clang and GCC alone, a limit
of the flag rather than of the code.

The release workflow built the engine inside the job that builds everything
else, on one Linux machine. A wheel is native to the machine that builds it,
so that job becomes three: `build` for the pure-Python distributions,
`engine-wheels` across five runners, and `assemble`, which joins them and runs
the checks `build` used to run, unchanged. One runner per platform and
architecture, native rather than emulated, so each wheel's smoke test runs on
the machine it targets. That retires the QEMU step, and with it a privileged
container image in the release path.

The matrix is the one CI runs, so a tag uploads the set a pull request proved.
It uses `macos-15-intel` rather than `macos-13`, which GitHub retired in
December 2025.

Verified by building and smoke-testing cp312 and cp314 macOS arm64 wheels from
this tree, and by reading back the matrix cibuildwheel now selects. Windows is
the one platform not proven locally; CI is the first machine to compile it.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Manush Murali <manushm@nvidia.com>
@Manushpm8
Manushpm8 force-pushed the engine-wheels-all-platforms branch from a290711 to 53cf671 Compare October 18, 2026 06:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant