From 53cf671d77df6c2326374e3e710a77db9a4230be Mon Sep 17 00:00:00 2001 From: Manush Murali Date: Sat, 17 Oct 2026 21:46:14 -0700 Subject: [PATCH] Build the relational engine for macOS, Windows and 3.14 `structured-data-models-client[relational]` installed on Linux and nowhere else. The engine is a native build that ships no source distribution, so a platform with no wheel has nothing to fall back on: pip reports "No matching distribution found for structured-data-models-relational-engine" and stops. A developer on a Mac could install the client but not the model it fronts. Nothing about the engine required that. The sampler is 564 lines of standard library C++ with no POSIX headers, and `CMakeLists.txt` has carried a macOS deployment target all along. What was Linux-only was the wheel matrix: `[tool.cibuildwheel.linux]` was the only platform table, and both the CI job and the release step ran cibuildwheel on Ubuntu runners, which can only produce manylinux wheels. So the engine now builds for macOS on both architectures, for 64-bit Windows, and for CPython 3.14 everywhere, alongside the manylinux wheels it already built. 25 wheels where there were 8. `requires-python` moves to `<3.15` with the classifiers, as the comment above it requires. Two things this needed: `-O3` was set unconditionally, which MSVC rejects; its release default is already `/O2`. That made the file buildable by Clang and GCC alone, a limit of the flag rather than of the code. The release workflow built the engine inside the job that builds everything else, on one Linux machine. A wheel is native to the machine that builds it, so that job becomes three: `build` for the pure-Python distributions, `engine-wheels` across five runners, and `assemble`, which joins them and runs the checks `build` used to run, unchanged. One runner per platform and architecture, native rather than emulated, so each wheel's smoke test runs on the machine it targets. That retires the QEMU step, and with it a privileged container image in the release path. The matrix is the one CI runs, so a tag uploads the set a pull request proved. It uses `macos-15-intel` rather than `macos-13`, which GitHub retired in December 2025. Verified by building and smoke-testing cp312 and cp314 macOS arm64 wheels from this tree, and by reading back the matrix cibuildwheel now selects. Windows is the one platform not proven locally; CI is the first machine to compile it. Co-authored-by: Claude Opus 5 (1M context) Signed-off-by: Manush Murali --- .github/workflows/ci.yml | 69 +++++- .github/workflows/publish.yml | 219 ++++++++++++++---- .gitlab-ci.yml | 6 +- CHANGELOG.md | 18 ++ README.md | 13 +- RELEASING.md | 3 +- docs/about/architecture.md | 7 +- docs/get-started/installation.md | 2 +- docs/get-started/prerequisites.md | 23 +- .../pyproject.toml | 13 +- .../CMakeLists.txt | 5 +- .../README.md | 5 +- .../pyproject.toml | 21 +- .../setup.py | 20 ++ .../rfm/backend/local/__init__.py | 11 +- tests/test_release_contract.py | 17 +- 16 files changed, 358 insertions(+), 94 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c55267dbb..7479667ce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -221,20 +221,32 @@ jobs: name: wheels path: dist/ - # The engine's manylinux wheels, from the same `[tool.cibuildwheel]` table - # the release reads, each architecture on a runner of its own rather than - # under emulation. Databricks serverless compute can run on either - # architecture, and each wheel is smoke-tested on the architecture it targets. + # The engine's wheels, from the same `[tool.cibuildwheel]` table the release + # reads. One runner per platform and architecture, so every wheel is + # smoke-tested on the machine it targets. engine-wheels: needs: build strategy: fail-fast: false matrix: include: - - arch: x86_64 + - name: linux-x86_64 runner: ubuntu-24.04 - - arch: aarch64 + archs: CIBW_ARCHS_LINUX=x86_64 + - name: linux-aarch64 runner: ubuntu-24.04-arm + archs: CIBW_ARCHS_LINUX=aarch64 + - name: macos-arm64 + runner: macos-15 + archs: CIBW_ARCHS_MACOS=arm64 + # macos-13 was retired in December 2025. x86_64 macOS ends with the + # macOS 15 image in autumn 2027, and this row ends with it. + - name: macos-x86_64 + runner: macos-15-intel + archs: CIBW_ARCHS_MACOS=x86_64 + - name: windows-amd64 + runner: windows-2022 + archs: CIBW_ARCHS_WINDOWS=AMD64 runs-on: ${{ matrix.runner }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -244,15 +256,54 @@ jobs: with: name: wheels path: _sibling_wheels + # Only Linux runs in a container, where the engine's + # `[tool.cibuildwheel.linux]` already names /project/_sibling_wheels. + - name: Point the wheel test environments at the sibling wheels + shell: bash + run: | + echo "${{ matrix.archs }}" >> "$GITHUB_ENV" + case "${{ runner.os }}" in + Linux) ;; + # cibuildwheel layers its own `environment` over the host's, so a + # plain variable reaches the build and test environments without + # displacing what the engine's pyproject sets per platform. + *) echo "PIP_FIND_LINKS=${{ github.workspace }}/_sibling_wheels" >> "$GITHUB_ENV" ;; + esac + - uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1 - env: - CIBW_ARCHS_LINUX: ${{ matrix.arch }} with: package-dir: packages/structured-data-models-relational-engine output-dir: wheelhouse + + # A wheel built by the wrong compiler passes every other check here, so + # the evidence has to come from the wheel. MinGW's runtime travelling + # inside one means CMake found GCC where MSVC was intended. Every wheel + # is read, and an empty wheelhouse fails rather than reporting nothing + # found. + - name: No MinGW runtime rode along in a Windows wheel + if: runner.os == 'Windows' + shell: bash + run: | + set -euo pipefail + shopt -s nullglob + wheels=(wheelhouse/*.whl) + if [ ${#wheels[@]} -eq 0 ]; then + echo "::error::the Windows row produced no wheel to check" + exit 1 + fi + for wheel in "${wheels[@]}"; do + names=$(unzip -Z1 "$wheel") + if printf '%s\n' "$names" \ + | grep -qiE "lib(gcc_s_seh-1|stdc\+\+-6|winpthread-1)[^/]*\.dll"; then + echo "::error::$(basename "$wheel") was built with MinGW, not MSVC" + printf '%s\n' "$names" | grep -iE "\.dll$" + exit 1 + fi + done + echo "${#wheels[@]} Windows wheels, no MinGW runtime in any of them" - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: engine-wheels-${{ matrix.arch }} + name: engine-wheels-${{ matrix.name }} path: wheelhouse/ # The suite has only ever run against the source tree. What users install is diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index d241f34ff..22d6a5606 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -32,9 +32,10 @@ # `build` has `contents: read` to check the tree out and `actions: read` to ask # whether CI passed, and nothing more, so a compromised build dependency cannot # publish anything itself. It can still decide what gets published, because it -# is what produces the distributions. `release` holds `contents: write` to -# create the Release and attach to it, and is the only job behind the -# protected environment. +# is what produces the distributions, and `engine-wheels` and `assemble` can +# too, for the same reason; both hold `contents: read` alone. `release` holds +# `contents: write` to create the Release and attach to it, and is the only job +# behind the protected environment. name: publish on: @@ -119,47 +120,137 @@ jobs: ( cd "packages/$package" && python -m build --outdir "$GITHUB_WORKSPACE/dist" . ) done - # The engine pins its siblings by exact version, and at this point in - # a release those versions exist only in dist/, not on PyPI yet, and - # will not be until someone publishes them. cibuildwheel's test - # environment resolves from PyPI unless told otherwise, so without this - # the smoke test fails with "structured-data-models-api was not found" and the release stops - # before it can publish anything. cibuildwheel copies the directory it - # runs from, the repository root, into the container as /project, and - # the engine's `[tool.cibuildwheel.linux]` points pip at - # /project/_sibling_wheels. - - name: Stage the sibling wheels where the engine's test env can see them - run: | - mkdir -p _sibling_wheels - cp dist/*.whl _sibling_wheels/ - - # aarch64 is built under emulation on the x86-64 runner, the same way the - # GitLab `wheels` job builds it, so the two cannot disagree on the set. - # The image is pinned as well as the action. Left to its default, this - # step pulls `tonistiigi/binfmt:latest` and runs it with `--privileged` - # on the runner that then builds the native wheels, so a moving tag on - # that image reaches the release even though the action cannot. Nothing - # updates this digest automatically, since dependabot does not read a - # workflow input; it is raised by hand with the action. - - uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0 + # The engine's wheels cannot be built here: each is native to the + # machine that builds it, and this job is one Linux machine. + - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - platforms: arm64 - image: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0 - - # The engine is a native build and needs one wheel per interpreter and - # platform. It is not optional: structured-data-models-client[relational] pins it by exact - # version, so publishing the other four without it makes that extra - # uninstallable for the release it names. What is built, and the smoke - # test each wheel must pass, is the engine's `[tool.cibuildwheel]`. - - name: Build the relational engine's wheels - uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1 + name: pure-distributions + path: dist/ + + # The same matrix CI runs, so a tag uploads the set a pull request proved. + # Native on every row, which retires the QEMU step that built aarch64 here + # and the privileged container image it pulled into the release path. + # + # The engine is not optional. `structured-data-models-client[relational]` pins it by exact + # version, so publishing the other four without it makes that extra + # uninstallable for the release it names. + engine-wheels: + needs: build + strategy: + fail-fast: false + matrix: + include: + - name: linux-x86_64 + runner: ubuntu-24.04 + archs: CIBW_ARCHS_LINUX=x86_64 + - name: linux-aarch64 + runner: ubuntu-24.04-arm + archs: CIBW_ARCHS_LINUX=aarch64 + - name: macos-arm64 + runner: macos-15 + archs: CIBW_ARCHS_MACOS=arm64 + # macos-13 was retired in December 2025. x86_64 macOS ends with the + # macOS 15 image in autumn 2027, and this row ends with it. + - name: macos-x86_64 + runner: macos-15-intel + archs: CIBW_ARCHS_MACOS=x86_64 + - name: windows-amd64 + runner: windows-2022 + archs: CIBW_ARCHS_WINDOWS=AMD64 + runs-on: ${{ matrix.runner }} + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # The engine pins its siblings by exact version, and at this point in a + # release those versions exist only as the artifact `build` uploaded. + # cibuildwheel's test environment resolves from PyPI unless told + # otherwise, so without this the smoke test cannot find them. + - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: pure-distributions + path: _sibling_wheels + + # Only Linux runs in a container, where the engine's + # `[tool.cibuildwheel.linux]` already names /project/_sibling_wheels. + - name: Point the wheel test environments at the sibling wheels + shell: bash + run: | + echo "${{ matrix.archs }}" >> "$GITHUB_ENV" + case "${{ runner.os }}" in + Linux) ;; + # cibuildwheel layers its own `environment` over the host's, so a + # plain variable reaches the build and test environments without + # displacing what the engine's pyproject sets per platform. + *) echo "PIP_FIND_LINKS=${{ github.workspace }}/_sibling_wheels" >> "$GITHUB_ENV" ;; + esac + + + - uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1 with: package-dir: packages/structured-data-models-relational-engine - output-dir: dist + output-dir: wheelhouse - - name: Unstage them, so nothing but distributions reaches the upload - if: always() - run: rm -rf _sibling_wheels + # A wheel built by the wrong compiler passes every other check here, so + # the evidence has to come from the wheel. MinGW's runtime travelling + # inside one means CMake found GCC where MSVC was intended. Every wheel + # is read, and an empty wheelhouse fails rather than reporting nothing + # found. + - name: No MinGW runtime rode along in a Windows wheel + if: runner.os == 'Windows' + shell: bash + run: | + set -euo pipefail + shopt -s nullglob + wheels=(wheelhouse/*.whl) + if [ ${#wheels[@]} -eq 0 ]; then + echo "::error::the Windows row produced no wheel to check" + exit 1 + fi + for wheel in "${wheels[@]}"; do + names=$(unzip -Z1 "$wheel") + if printf '%s\n' "$names" \ + | grep -qiE "lib(gcc_s_seh-1|stdc\+\+-6|winpthread-1)[^/]*\.dll"; then + echo "::error::$(basename "$wheel") was built with MinGW, not MSVC" + printf '%s\n' "$names" | grep -iE "\.dll$" + exit 1 + fi + done + echo "${#wheels[@]} Windows wheels, no MinGW runtime in any of them" + + - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: engine-wheels-${{ matrix.name }} + path: wheelhouse/ + + # Every distribution in one directory, checked as a set. Split out of `build` + # only because the engine's wheels now come from five machines; the steps + # below are the ones that used to run there, unchanged. + assemble: + needs: [build, engine-wheels] + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.12' + + - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: pure-distributions + path: dist + + # Every engine row merged into the same directory. A pattern rather than + # one step per platform, so adding a row to the matrix above does not + # need a matching step here to be published. + - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + pattern: engine-wheels-* + merge-multiple: true + path: dist # The tag is the release. If it does not name the version the packages # carry, the wrong thing is about to be published under the right name. @@ -195,6 +286,52 @@ jobs: if missing: sys.exit(f'refusing to publish without {sorted(missing)}') + # Naming the engine is not the same as building it everywhere. One + # wheel satisfies the check above, so a row of `engine-wheels` that + # produced a subset would publish an engine installable on fewer + # platforms than the release claims, and nothing downstream would + # notice. The interpreters are read from the engine's own + # `[tool.cibuildwheel] build`; the platforms are the matrix that + # built them. + import tomllib + + engine = pathlib.Path( + 'packages/structured-data-models-relational-engine/pyproject.toml' + ) + selector = tomllib.loads(engine.read_text())['tool']['cibuildwheel'] + interpreters = { + part.split('-')[0] for part in selector['build'].split() + } + platforms = { + 'manylinux.*_x86_64', + 'manylinux.*_aarch64', + 'macosx_.*_arm64', + 'macosx_.*_x86_64', + 'win_amd64', + } + engine_wheels = [ + name + for name in wheels + if name.startswith('structured_data_models_relational_engine-') + ] + absent = sorted( + f'{interpreter} {platform}' + for interpreter in interpreters + for platform in platforms + if not any( + re.fullmatch( + rf'.*-{interpreter}-{interpreter}-{platform}\.whl', name + ) + for name in engine_wheels + ) + ) + if absent: + sys.exit(f'the engine wheel matrix is incomplete: {absent}') + print( + f'{len(engine_wheels)} engine wheels, ' + f'{len(interpreters)} interpreters on {len(platforms)} platforms' + ) + # Every file, not only the wheels: an sdist carrying a different # version passes a wheels-only check and is published anyway. versions = { @@ -260,7 +397,7 @@ jobs: # asset URLs are private and Kitmaker could not fetch them. Declining the # approval therefore means nothing was created at all. release: - needs: build + needs: assemble runs-on: ubuntu-latest environment: pypi # a protected environment, so a release is reviewable permissions: diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index d766d1d13..ef8865635 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -251,9 +251,9 @@ wheels: parallel: matrix: - CIBW_ARCHS_LINUX: x86_64 - CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-*" + CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*" - CIBW_ARCHS_LINUX: aarch64 - CIBW_BUILD: [cp310-*, cp311-*, cp312-*, cp313-*] + CIBW_BUILD: [cp310-*, cp311-*, cp312-*, cp313-*, cp314-*] script: - apt-get update -qq && apt-get install -y -qq docker.io - for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done @@ -308,7 +308,7 @@ wheels-check: built.add(match.groups()) expected = { (f'cp3{minor}', arch) - for minor in (10, 11, 12, 13) + for minor in (10, 11, 12, 13, 14) for arch in ('x86_64', 'aarch64') } if built != expected: diff --git a/CHANGELOG.md b/CHANGELOG.md index 5eb62fc07..6f8734742 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,24 @@ versioned by [semantic versioning](https://semver.org) as [`docs/reference/stability.md`](docs/reference/stability.md) defines it for these packages. All five distributions share one version and one entry. +## Unreleased + +### Added + +- The relational engine publishes wheels for macOS (Apple silicon and Intel) + and 64-bit Windows, and for CPython 3.14, alongside the `manylinux_2_28` + x86-64 and aarch64 wheels it already built, so `structured-data-models-client[relational]` + installs on a laptop rather than on Linux only. The engine ships no source + distribution, so those wheels remain the full set of platforms it supports: + `structured-data-models-client` itself is pure Python and stays broader. + +### Changed + +- Every engine wheel is built on a machine of its own, native rather than + emulated, and each is smoke-tested on the architecture it targets. The + release no longer builds aarch64 under QEMU, which also takes a privileged + container image out of the release path. + ## 1.0.0 - 2026-10-15 First public release. The five distributions are versioned and published diff --git a/README.md b/README.md index d0bbf2dd3..93b26df2d 100644 --- a/README.md +++ b/README.md @@ -77,12 +77,13 @@ differs on Databricks. Python 3.10 through 3.14. The client, the language and the connectors are pure Python and install anywhere. -The relational engine is a native build. It publishes `manylinux_2_28` wheels -for x86-64 and aarch64 on **CPython 3.10 through 3.13** and **no source -distribution**, so `[relational]` resolves on a Linux with glibc 2.28 or newer -and nowhere else: not on musl, not on an older glibc, and not on macOS or -Windows, where pip has nothing to fall back to. `kumo-tabular` works on any -laptop. +The relational engine is a native build, so it ships wheels rather than a +source distribution: `manylinux_2_28` for x86-64 and aarch64, macOS 11 or newer +for Apple silicon and Intel, and 64-bit Windows, each on **CPython 3.10 through +3.14**. `[relational]` resolves on all of those and nowhere else. There is **no +source distribution**, so anywhere without a matching wheel pip has nothing to +fall back to: musl, an older glibc, a 32-bit interpreter, or a Python newer +than the engine has built for yet. ## Packages diff --git a/RELEASING.md b/RELEASING.md index 5d4b0f7a7..c3975cbd9 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -72,7 +72,8 @@ git tag v1.2.0 && git push origin v1.2.0 That starts the release. `.github/workflows/publish.yml` builds the four pure-Python distributions plus the engine's wheel matrix through `cibuildwheel` -(cp310 to cp313, manylinux_2_28 x86-64 and aarch64, each smoke-tested, as set in +(cp310 to cp314 on manylinux_2_28 x86-64 and aarch64, macOS arm64 and x86-64, +and Windows AMD64, one runner each and every wheel smoke-tested, as set in the engine's `[tool.cibuildwheel]`), and then refuses to continue unless **all five** are present and agree with the tag. That check is not ceremony: `structured-data-models-client[relational]` pins the engine by exact version, so a diff --git a/docs/about/architecture.md b/docs/about/architecture.md index 9123db87b..80cdd88b9 100644 --- a/docs/about/architecture.md +++ b/docs/about/architecture.md @@ -122,9 +122,10 @@ structured-data-models-pql ◄──┘ │ `structured-data-models-api` depends on nothing. -**Only `structured-data-models-relational-engine` is native.** Everything else installs anywhere, -so `kumo-tabular` works on a Mac; `kumo-relational` needs Linux (x86-64 or -aarch64). +**Only `structured-data-models-relational-engine` is native.** Everything else installs anywhere; +the engine installs where it has published a wheel, which is Linux (x86-64 or +aarch64), macOS (Apple silicon or Intel) and 64-bit Windows. Both models work +on a Mac. ```bash pip install structured-data-models-client # tabular, PQL, transport everywhere diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 577cd6ffd..ea8b96eb4 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -3,7 +3,7 @@ | Command | You get | |---|---| | `pip install structured-data-models-client` | The client, the language, and the tabular model. Runs anywhere. | -| `pip install 'structured-data-models-client[relational]'` | Adds the relational model and native engine. **Python 3.10 through 3.13 only**. See [prerequisites](prerequisites.md). | +| `pip install 'structured-data-models-client[relational]'` | Adds the relational model and native engine. Wheels only, for **Linux, macOS and Windows on CPython 3.10 through 3.14**. See [prerequisites](prerequisites.md). | | `pip install 'structured-data-models-client[snowflake]'` | Reads tables from a warehouse. Also `[sqlite]`, `[duckdb]`, `[postgres]`, `[databricks]`, `[lakebase]`, `[s3]`. | | `pip install 'structured-data-models-client[all]'` | The relational model and every warehouse backend. | | `pip install 'structured-data-models-client[databricks-serving]'` | Reaches a model served on Databricks Model Serving, through `SDMClient.for_databricks_serving`. Not in `[all]`: it is a way to reach a model, not a backend. | diff --git a/docs/get-started/prerequisites.md b/docs/get-started/prerequisites.md index cf5873f53..9d7ee0703 100644 --- a/docs/get-started/prerequisites.md +++ b/docs/get-started/prerequisites.md @@ -1,11 +1,13 @@ # Prerequisites -- **Python** 3.10 to 3.14 for the client, the language, the wire types and the - connectors. **3.10 to 3.13 for the relational engine**, which is a native - build with a shorter wheel matrix. `pip install 'structured-data-models-client[relational]'` does - not resolve on 3.14, while plain `pip install structured-data-models-client` does. It is - reported as `No matching distribution found for structured-data-models-relational-engine`, - which names the package and not the reason. +- **Python** 3.10 to 3.14, for every distribution. The relational engine is a + native build publishing CPython wheels and no source distribution, so it + supports an interpreter only once it has built wheels for it, and PyPy not + at all: on a Python newer than that, + `pip install 'structured-data-models-client[relational]'` does not resolve while plain + `pip install structured-data-models-client` does. It is reported as `No matching + distribution found for structured-data-models-relational-engine`, which + names the package and not the reason. - **A reachable NIM** serving `kumo-tabular` or `kumo-relational`. The client runs no model and downloads no weights. - **No GPU** on the client. The NIM owns that. @@ -14,10 +16,11 @@ The client, the language and the connectors are pure Python and install anywhere. The relational engine is a native build: it compiles a neighbour -sampler, and published wheels are `manylinux_2_28` for x86-64 and aarch64 on -CPython only, so -they need a Linux with glibc 2.28 or newer. `kumo-tabular` works on a laptop; -`kumo-relational` needs that Linux. +sampler, and published wheels are `manylinux_2_28` for x86-64 and aarch64, +macOS 11 or newer for Apple silicon and Intel, and 64-bit Windows, on CPython +only. Both models work on a laptop. What has no wheel has nothing to fall back +on, since the engine ships no source distribution: musl, a glibc older than +2.28, a 32-bit interpreter, or Windows on Arm. ## Getting a NIM diff --git a/packages/structured-data-models-client/pyproject.toml b/packages/structured-data-models-client/pyproject.toml index a3da90598..5a7c18eba 100644 --- a/packages/structured-data-models-client/pyproject.toml +++ b/packages/structured-data-models-client/pyproject.toml @@ -57,12 +57,13 @@ Changelog = "https://github.com/NVIDIA/structured-data-models-client/blob/main/C # A model that does heavy client-side work brings its own runtime; a warehouse # driver is opt-in the same way, through the shared connectors package. # -# `relational` is deliberately narrower than this distribution: the engine is a -# native build that publishes wheels for 3.10-3.13, so `structured-data-models-client[relational]` -# does not resolve on 3.14 while plain `structured-data-models-client` does. pip reports that as -# "No matching distribution found for structured-data-models-relational-engine", which names the -# symptom and not the reason, so it is written down here and in the -# installation guide. +# `relational` stays narrower than this distribution. The engine is a +# native build, publishing CPython wheels for 3.10-3.14 on Linux, macOS and +# Windows and no sdist, so the extra stops at 3.15 and on PyPy, while this +# distribution is pure Python and admits each later interpreter as it +# arrives. pip reports the gap as "No matching distribution found for +# structured-data-models-relational-engine", which names the symptom and not +# the reason, so it is written down here and in the installation guide. relational = ["structured-data-models-relational-engine==1.0.0"] sqlite = ["structured-data-models-connectors[sqlite]==1.0.0"] duckdb = ["structured-data-models-connectors[duckdb]==1.0.0"] diff --git a/packages/structured-data-models-relational-engine/CMakeLists.txt b/packages/structured-data-models-relational-engine/CMakeLists.txt index 98102960b..e07475fcc 100644 --- a/packages/structured-data-models-relational-engine/CMakeLists.txt +++ b/packages/structured-data-models-relational-engine/CMakeLists.txt @@ -15,7 +15,10 @@ if(NOT pybind11_FOUND) FetchContent_MakeAvailable(pybind11) endif() -set(CMAKE_CXX_FLAGS "-O3") +# MSVC rejects `-O3`; its release default is already `/O2`. +if(NOT MSVC) + set(CMAKE_CXX_FLAGS "-O3") +endif() set(CMAKE_OSX_DEPLOYMENT_TARGET "11.0" CACHE STRING "Minimum macOS deployment version" FORCE) pybind11_add_module(relationallib src/sdm_relational_engine/csrc/neighbor_sampler.cpp) diff --git a/packages/structured-data-models-relational-engine/README.md b/packages/structured-data-models-relational-engine/README.md index e61ea60f4..ccd01a793 100644 --- a/packages/structured-data-models-relational-engine/README.md +++ b/packages/structured-data-models-relational-engine/README.md @@ -28,8 +28,9 @@ pip install "structured-data-models-relational-engine[explain]" # natural-la pip install "structured-data-models-relational-engine[relbench]" # Graph.from_relbench() dataset loading ``` -Release wheels are built for CPython 3.10, 3.11, 3.12 and 3.13 -(`manylinux_2_28` x86-64 and aarch64); no source distribution is published. +Release wheels are built for CPython 3.10 through 3.14 on `manylinux_2_28` +x86-64 and aarch64, macOS 11 and newer for arm64 and x86-64, and 64-bit +Windows; no source distribution is published. Application code should import the client's neutral surface, not this package directly: diff --git a/packages/structured-data-models-relational-engine/pyproject.toml b/packages/structured-data-models-relational-engine/pyproject.toml index a7c6cd75b..cbd7e1c31 100644 --- a/packages/structured-data-models-relational-engine/pyproject.toml +++ b/packages/structured-data-models-relational-engine/pyproject.toml @@ -16,9 +16,9 @@ readme="README.md" # must fail resolution rather than fail to find a distribution. The bound and # the classifiers below name the same interpreters and move together. # -# The wheel jobs build the Linux matrix these classifiers name, from +# The wheel jobs build the matrix these classifiers name, from # `[tool.cibuildwheel]` below. The bound states what the extension supports. -requires-python=">=3.10,<3.14" +requires-python=">=3.10,<3.15" keywords=[ "deep-learning", "graph-neural-networks", @@ -34,6 +34,7 @@ classifiers=[ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Programming Language :: Python :: 3 :: Only", "Topic :: Scientific/Engineering :: Artificial Intelligence", "Typing :: Typed", @@ -122,7 +123,7 @@ editable.verbose=false # PIP_FIND_LINKS. uv reads no PIP_* variable, and the uv the manylinux images # shipped had no environment form of --find-links at all, so under `build[uv]` # the smoke test could never resolve structured-data-models-api. -build = "cp310-* cp311-* cp312-* cp313-*" +build = "cp310-* cp311-* cp312-* cp313-* cp314-*" # musllinux is not a libc this project has ever built or tested against, and # leaving it on would silently double the Linux wheel count. skip = "*-musllinux_*" @@ -141,8 +142,22 @@ manylinux-aarch64-image = "manylinux_2_28" # only as wheels on disk. cibuildwheel copies the directory it is run from into # the container as /project, and every workflow runs it from the repository # root, so this is `_sibling_wheels/` at that root. +# Only Linux builds in a container, so only Linux has a path that is the same +# on every machine; macOS and Windows are given the checkout's absolute path by +# their workflows instead. environment = { PIP_FIND_LINKS = "/project/_sibling_wheels" } +[tool.cibuildwheel.macos] +# The workflows narrow this to one arch per runner, so every wheel is built and +# smoke-tested natively rather than cross-compiled from the other arch. +archs = ["arm64", "x86_64"] +# Match CMAKE_OSX_DEPLOYMENT_TARGET above. cibuildwheel's default for x86_64 is +# 10.9, and delocate refuses a wheel tagged older than the binary inside it. +environment = { MACOSX_DEPLOYMENT_TARGET = "11.0" } + +[tool.cibuildwheel.windows] +archs = ["AMD64"] + [tool.pytest.ini_options] xfail_strict = true addopts = [ diff --git a/packages/structured-data-models-relational-engine/setup.py b/packages/structured-data-models-relational-engine/setup.py index e3e4abacb..0f289c9a9 100644 --- a/packages/structured-data-models-relational-engine/setup.py +++ b/packages/structured-data-models-relational-engine/setup.py @@ -2,7 +2,10 @@ # All rights reserved. # SPDX-License-Identifier: Apache-2.0 +import hashlib import os +import sys +import tempfile from setuptools import setup @@ -16,6 +19,23 @@ else: kwargs = dict() +if sys.platform == 'win32': + # MSBuild tracks file access through `.tlog` files written beneath the + # build directory, and their names are long. Under this package's own path + # they pass Windows' 260-character limit, and the compiler check fails + # before any source here is read. Build somewhere short instead. + # + # Named for where it is building from, because `build/` is per checkout + # and this is not: without the digest, two checkouts on one machine would + # share a CMake cache and a set of `.tlog` files. + _here = os.path.dirname(os.path.abspath(__file__)) + _tag = hashlib.sha256(_here.encode('utf-8')).hexdigest()[:8] + kwargs['options'] = { + 'build': { + 'build_base': os.path.join(tempfile.gettempdir(), f'sdm-{_tag}') + } + } + setup( install_requires=[ 'numpy', diff --git a/packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/local/__init__.py b/packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/local/__init__.py index d2b7bd4be..f79e765f0 100644 --- a/packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/local/__init__.py +++ b/packages/structured-data-models-relational-engine/src/sdm_relational_engine/rfm/backend/local/__init__.py @@ -16,16 +16,17 @@ glibc version: {platform.libc_ver()[1]} ✅ Prebuilt wheels are published for: -* Python versions: 3.10, 3.11, 3.12, 3.13 +* CPython versions: 3.10, 3.11, 3.12, 3.13, 3.14 * Linux (x86_64 and arm64), glibc >=2.28 +* macOS 11 and newer (arm64 and x86_64) +* Windows (x86_64) No source distribution is published, so `pip install structured-data-models-relational-engine` resolves on those -platforms alone. The extension also builds from a source checkout on macOS -(arm64) and Windows (x86_64). +platforms alone. ❌ Not supported: -* Python versions below 3.10, or 3.14 and later -* macOS (x86_64), Windows (arm64) +* CPython below 3.10, or 3.15 and later; any PyPy +* Windows (arm64), musl-based Linux * glibc versions: <2.28 Please create a feature request at 'https://github.com/NVIDIA/structured-data-models-client'.""" diff --git a/tests/test_release_contract.py b/tests/test_release_contract.py index 385894190..4d16c7041 100644 --- a/tests/test_release_contract.py +++ b/tests/test_release_contract.py @@ -1646,9 +1646,20 @@ def test_an_extra_narrower_than_its_parent_says_so(self): source = ( PACKAGES / 'structured-data-models-client' / 'pyproject.toml' ).read_text() - assert '3.14' in source and 'native build' in source, ( - 'structured-data-models-client admits interpreters its [relational] extra cannot ' - 'install on, and its pyproject does not explain that anywhere' + # The engine's own bound, not a literal: this read `'3.14' in source` + # while the engine stopped at 3.13, so once 3.14 became supported the + # comment named it for a different reason and the test proved nothing. + # What has to be written down is where the extra now stops. + bound = engine['project']['requires-python'] + edge = re.search(r'<\s*(\d+\.\d+)', bound) + assert edge is not None, ( + f'the engine no longer carries an upper bound ({bound!r}), so what ' + f'this test reads as the edge of the matrix is gone' + ) + assert edge.group(1) in source and 'native build' in source, ( + f'structured-data-models-client admits interpreters its [relational] extra cannot ' + f'install on, and its pyproject does not say the extra stops below ' + f'{edge.group(1)}' ) install_doc = ( REPO / 'docs' / 'get-started' / 'installation.md'