Skip to content

Build the relational engine for macOS, Windows and 3.14 #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Manushpm8 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Build the relational engine for macOS, Windows and 3.14 #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 60 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,20 +221,32 @@ jobs:
name: wheels
path: dist/

# The engine's manylinux wheels, from the same `[tool.cibuildwheel]` table
# the release reads, each architecture on a runner of its own rather than
# under emulation. Databricks serverless compute can run on either
# architecture, and each wheel is smoke-tested on the architecture it targets.
# The engine's wheels, from the same `[tool.cibuildwheel]` table the release
# reads. One runner per platform and architecture, so every wheel is
# smoke-tested on the machine it targets.
engine-wheels:
needs: build
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
- name: linux-x86_64
runner: ubuntu-24.04
- arch: aarch64
archs: CIBW_ARCHS_LINUX=x86_64
- name: linux-aarch64
runner: ubuntu-24.04-arm
archs: CIBW_ARCHS_LINUX=aarch64
- name: macos-arm64
runner: macos-15
archs: CIBW_ARCHS_MACOS=arm64
# macos-13 was retired in December 2025. x86_64 macOS ends with the
# macOS 15 image in autumn 2027, and this row ends with it.
- name: macos-x86_64
runner: macos-15-intel
archs: CIBW_ARCHS_MACOS=x86_64
- name: windows-amd64
runner: windows-2022
archs: CIBW_ARCHS_WINDOWS=AMD64
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -244,15 +256,54 @@ jobs:
with:
name: wheels
path: _sibling_wheels
# Only Linux runs in a container, where the engine's
# `[tool.cibuildwheel.linux]` already names /project/_sibling_wheels.
- name: Point the wheel test environments at the sibling wheels
shell: bash
run: |
echo "${{ matrix.archs }}" >> "$GITHUB_ENV"
case "${{ runner.os }}" in
Linux) ;;
# cibuildwheel layers its own `environment` over the host's, so a
# plain variable reaches the build and test environments without
# displacing what the engine's pyproject sets per platform.
*) echo "PIP_FIND_LINKS=${{ github.workspace }}/_sibling_wheels" >> "$GITHUB_ENV" ;;
esac

- uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1
env:
CIBW_ARCHS_LINUX: ${{ matrix.arch }}
with:
package-dir: packages/structured-data-models-relational-engine
output-dir: wheelhouse

# A wheel built by the wrong compiler passes every other check here, so
# the evidence has to come from the wheel. MinGW's runtime travelling
# inside one means CMake found GCC where MSVC was intended. Every wheel
# is read, and an empty wheelhouse fails rather than reporting nothing
# found.
- name: No MinGW runtime rode along in a Windows wheel
if: runner.os == 'Windows'
shell: bash
run: |
set -euo pipefail
shopt -s nullglob
wheels=(wheelhouse/*.whl)
if [ ${#wheels[@]} -eq 0 ]; then
echo "::error::the Windows row produced no wheel to check"
exit 1
fi
for wheel in "${wheels[@]}"; do
names=$(unzip -Z1 "$wheel")
if printf '%s\n' "$names" \
| grep -qiE "lib(gcc_s_seh-1|stdc\+\+-6|winpthread-1)[^/]*\.dll"; then
echo "::error::$(basename "$wheel") was built with MinGW, not MSVC"
printf '%s\n' "$names" | grep -iE "\.dll$"
exit 1
fi
done
echo "${#wheels[@]} Windows wheels, no MinGW runtime in any of them"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: engine-wheels-${{ matrix.arch }}
name: engine-wheels-${{ matrix.name }}
path: wheelhouse/

# The suite has only ever run against the source tree. What users install is
Expand Down
219 changes: 178 additions & 41 deletions .github/workflows/publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,10 @@
# `build` has `contents: read` to check the tree out and `actions: read` to ask
# whether CI passed, and nothing more, so a compromised build dependency cannot
# publish anything itself. It can still decide what gets published, because it
# is what produces the distributions. `release` holds `contents: write` to
# create the Release and attach to it, and is the only job behind the
# protected environment.
# is what produces the distributions, and `engine-wheels` and `assemble` can
# too, for the same reason; both hold `contents: read` alone. `release` holds
# `contents: write` to create the Release and attach to it, and is the only job
# behind the protected environment.
name: publish

on:
Expand Down Expand Up @@ -119,47 +120,137 @@ jobs:
( cd "packages/$package" && python -m build --outdir "$GITHUB_WORKSPACE/dist" . )
done

# The engine pins its siblings by exact version, and at this point in
# a release those versions exist only in dist/, not on PyPI yet, and
# will not be until someone publishes them. cibuildwheel's test
# environment resolves from PyPI unless told otherwise, so without this
# the smoke test fails with "structured-data-models-api was not found" and the release stops
# before it can publish anything. cibuildwheel copies the directory it
# runs from, the repository root, into the container as /project, and
# the engine's `[tool.cibuildwheel.linux]` points pip at
# /project/_sibling_wheels.
- name: Stage the sibling wheels where the engine's test env can see them
run: |
mkdir -p _sibling_wheels
cp dist/*.whl _sibling_wheels/

# aarch64 is built under emulation on the x86-64 runner, the same way the
# GitLab `wheels` job builds it, so the two cannot disagree on the set.
# The image is pinned as well as the action. Left to its default, this
# step pulls `tonistiigi/binfmt:latest` and runs it with `--privileged`
# on the runner that then builds the native wheels, so a moving tag on
# that image reaches the release even though the action cannot. Nothing
# updates this digest automatically, since dependabot does not read a
# workflow input; it is raised by hand with the action.
- uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
# The engine's wheels cannot be built here: each is native to the
# machine that builds it, and this job is one Linux machine.
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
platforms: arm64
image: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0

# The engine is a native build and needs one wheel per interpreter and
# platform. It is not optional: structured-data-models-client[relational] pins it by exact
# version, so publishing the other four without it makes that extra
# uninstallable for the release it names. What is built, and the smoke
# test each wheel must pass, is the engine's `[tool.cibuildwheel]`.
- name: Build the relational engine's wheels
uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1
name: pure-distributions
path: dist/

# The same matrix CI runs, so a tag uploads the set a pull request proved.
# Native on every row, which retires the QEMU step that built aarch64 here
# and the privileged container image it pulled into the release path.
#
# The engine is not optional. `structured-data-models-client[relational]` pins it by exact
# version, so publishing the other four without it makes that extra
# uninstallable for the release it names.
engine-wheels:
needs: build
strategy:
fail-fast: false
matrix:
include:
- name: linux-x86_64
runner: ubuntu-24.04
archs: CIBW_ARCHS_LINUX=x86_64
- name: linux-aarch64
runner: ubuntu-24.04-arm
archs: CIBW_ARCHS_LINUX=aarch64
- name: macos-arm64
runner: macos-15
archs: CIBW_ARCHS_MACOS=arm64
# macos-13 was retired in December 2025. x86_64 macOS ends with the
# macOS 15 image in autumn 2027, and this row ends with it.
- name: macos-x86_64
runner: macos-15-intel
archs: CIBW_ARCHS_MACOS=x86_64
- name: windows-amd64
runner: windows-2022
archs: CIBW_ARCHS_WINDOWS=AMD64
runs-on: ${{ matrix.runner }}
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The engine pins its siblings by exact version, and at this point in a
# release those versions exist only as the artifact `build` uploaded.
# cibuildwheel's test environment resolves from PyPI unless told
# otherwise, so without this the smoke test cannot find them.
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: pure-distributions
path: _sibling_wheels

# Only Linux runs in a container, where the engine's
# `[tool.cibuildwheel.linux]` already names /project/_sibling_wheels.
- name: Point the wheel test environments at the sibling wheels
shell: bash
run: |
echo "${{ matrix.archs }}" >> "$GITHUB_ENV"
case "${{ runner.os }}" in
Linux) ;;
# cibuildwheel layers its own `environment` over the host's, so a
# plain variable reaches the build and test environments without
# displacing what the engine's pyproject sets per platform.
*) echo "PIP_FIND_LINKS=${{ github.workspace }}/_sibling_wheels" >> "$GITHUB_ENV" ;;
esac


- uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1
with:
package-dir: packages/structured-data-models-relational-engine
output-dir: dist
output-dir: wheelhouse

- name: Unstage them, so nothing but distributions reaches the upload
if: always()
run: rm -rf _sibling_wheels
# A wheel built by the wrong compiler passes every other check here, so
# the evidence has to come from the wheel. MinGW's runtime travelling
# inside one means CMake found GCC where MSVC was intended. Every wheel
# is read, and an empty wheelhouse fails rather than reporting nothing
# found.
- name: No MinGW runtime rode along in a Windows wheel
if: runner.os == 'Windows'
shell: bash
run: |
set -euo pipefail
shopt -s nullglob
wheels=(wheelhouse/*.whl)
if [ ${#wheels[@]} -eq 0 ]; then
echo "::error::the Windows row produced no wheel to check"
exit 1
fi
for wheel in "${wheels[@]}"; do
names=$(unzip -Z1 "$wheel")
if printf '%s\n' "$names" \
| grep -qiE "lib(gcc_s_seh-1|stdc\+\+-6|winpthread-1)[^/]*\.dll"; then
echo "::error::$(basename "$wheel") was built with MinGW, not MSVC"
printf '%s\n' "$names" | grep -iE "\.dll$"
exit 1
fi
done
echo "${#wheels[@]} Windows wheels, no MinGW runtime in any of them"

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: engine-wheels-${{ matrix.name }}
path: wheelhouse/

# Every distribution in one directory, checked as a set. Split out of `build`
# only because the engine's wheels now come from five machines; the steps
# below are the ones that used to run there, unchanged.
assemble:
needs: [build, engine-wheels]
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'

- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: pure-distributions
path: dist

# Every engine row merged into the same directory. A pattern rather than
# one step per platform, so adding a row to the matrix above does not
# need a matching step here to be published.
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: engine-wheels-*
merge-multiple: true
path: dist

# The tag is the release. If it does not name the version the packages
# carry, the wrong thing is about to be published under the right name.
Expand Down Expand Up @@ -195,6 +286,52 @@ jobs:
if missing:
sys.exit(f'refusing to publish without {sorted(missing)}')

# Naming the engine is not the same as building it everywhere. One
# wheel satisfies the check above, so a row of `engine-wheels` that
# produced a subset would publish an engine installable on fewer
# platforms than the release claims, and nothing downstream would
# notice. The interpreters are read from the engine's own
# `[tool.cibuildwheel] build`; the platforms are the matrix that
# built them.
import tomllib

engine = pathlib.Path(
'packages/structured-data-models-relational-engine/pyproject.toml'
)
selector = tomllib.loads(engine.read_text())['tool']['cibuildwheel']
interpreters = {
part.split('-')[0] for part in selector['build'].split()
}
platforms = {
'manylinux.*_x86_64',
'manylinux.*_aarch64',
'macosx_.*_arm64',
'macosx_.*_x86_64',
'win_amd64',
}
engine_wheels = [
name
for name in wheels
if name.startswith('structured_data_models_relational_engine-')
]
absent = sorted(
f'{interpreter} {platform}'
for interpreter in interpreters
for platform in platforms
if not any(
re.fullmatch(
rf'.*-{interpreter}-{interpreter}-{platform}\.whl', name
)
for name in engine_wheels
)
)
if absent:
sys.exit(f'the engine wheel matrix is incomplete: {absent}')
print(
f'{len(engine_wheels)} engine wheels, '
f'{len(interpreters)} interpreters on {len(platforms)} platforms'
)

# Every file, not only the wheels: an sdist carrying a different
# version passes a wheels-only check and is published anyway.
versions = {
Expand Down Expand Up @@ -260,7 +397,7 @@ jobs:
# asset URLs are private and Kitmaker could not fetch them. Declining the
# approval therefore means nothing was created at all.
release:
needs: build
needs: assemble
runs-on: ubuntu-latest
environment: pypi # a protected environment, so a release is reviewable
permissions:
Expand Down
6 changes: 3 additions & 3 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,9 +251,9 @@ wheels:
parallel:
matrix:
- CIBW_ARCHS_LINUX: x86_64
CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-*"
CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-* cp314-*"
- CIBW_ARCHS_LINUX: aarch64
CIBW_BUILD: [cp310-*, cp311-*, cp312-*, cp313-*]
CIBW_BUILD: [cp310-*, cp311-*, cp312-*, cp313-*, cp314-*]
script:
- apt-get update -qq && apt-get install -y -qq docker.io
- for i in $(seq 1 30); do docker info >/dev/null 2>&1 && break; sleep 2; done
Expand Down Expand Up @@ -308,7 +308,7 @@ wheels-check:
built.add(match.groups())
expected = {
(f'cp3{minor}', arch)
for minor in (10, 11, 12, 13)
for minor in (10, 11, 12, 13, 14)
for arch in ('x86_64', 'aarch64')
}
if built != expected:
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ versioned by [semantic versioning](https://semver.org) as
[`docs/reference/stability.md`](docs/reference/stability.md) defines it for
these packages. All five distributions share one version and one entry.

## Unreleased

### Added

- The relational engine publishes wheels for macOS (Apple silicon and Intel)
and 64-bit Windows, and for CPython 3.14, alongside the `manylinux_2_28`
x86-64 and aarch64 wheels it already built, so `structured-data-models-client[relational]`
installs on a laptop rather than on Linux only. The engine ships no source
distribution, so those wheels remain the full set of platforms it supports:
`structured-data-models-client` itself is pure Python and stays broader.

### Changed

- Every engine wheel is built on a machine of its own, native rather than
emulated, and each is smoke-tested on the architecture it targets. The
release no longer builds aarch64 under QEMU, which also takes a privileged
container image out of the release path.

## 1.0.0 - 2026-10-15

First public release. The five distributions are versioned and published
Expand Down
Loading
Loading