Add s390x wheels to the build and verification matrix by Devarsh010 · Pull Request #2360 · confluentinc/confluent-kafka-python

Devarsh Patel (Devarsh010) · 2026-09-16T12:20:51Z

What

Adds linux-s390x (IBM Z) to the Python wheel build and verification matrix, at
parity with the existing Linux arm64/x64 lanes, so pip install confluent-kafka
works on s390x with a prebuilt manylinux wheel.

tools/wheels/install-librdkafka.sh: on s390x, use the plain librdkafka.so
(same as arm64). The s390x librdkafka.redist runtime ships only the plain
build, not a centos8- variant.
tools/wheels/build-wheels.sh: set CIBW_MANYLINUX_S390X_IMAGE="manylinux_2_28"
so cibuildwheel builds s390x wheels in the same manylinux image as the other
Linux arches.
.semaphore/semaphore.yml: add Wheels: Linux s390x and
Wheel Verification: Linux s390x blocks (native s1-ubuntu-24-s390x-4 agent,
mirroring the arm64 blocks) and wire the verification into the
Source Distribution dependencies.

The Schema Registry serdes are pure Python and already ship in the wheel, so no
serdes code changes are required.

Checklist

Contains customer facing changes? Including API/behavior changes
- New s390x wheels published on release; no API or behavior changes. Existing
  arches are unaffected (the s390x branch/env var/blocks are additive).
Did you add sufficient unit test and/or integration test coverage for this PR?
- This is CI/packaging . It is exercised by the existing
  Wheel Verification blocks (tools/test-wheels.sh), which run the built
  wheel's smoke tests in clean distro containers — the s390x block runs the
  same verification.

Test & Review

Built the s390x wheel on a native IBM Z host via this exact path
(export ARCH=s390x → tools/wheels/build-wheels.sh → cibuildwheel →
confluent_kafka-<ver>-cp312-cp312-manylinux_2_28_s390x.whl) and verified:

Portability: installs + imports + loads librdkafka in clean s390x containers
across the glibc range manylinux_2_28 targets — AlmaLinux 8 (glibc 2.28, the
floor), RHEL/Rocky 9 (2.34), Ubuntu 24.04 (2.39).
Functional: produce/consume round-trip against a single-node broker

confluent-cla-assistant · 2026-09-16T12:21:03Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

Copilot

🟢 Approval recommended

All reviewed changes are additive and have no unresolved blocking issues.

Pull request overview

Adds native Linux s390x wheel building, verification, and release support.

Changes:

Selects the correct s390x librdkafka runtime.
Configures manylinux 2.28 s390x builds.
Adds Semaphore build and verification lanes.

File summaries

File	Description
`tools/wheels/install-librdkafka.sh`	Handles s390x runtime library selection.
`tools/wheels/build-wheels.sh`	Configures the s390x manylinux image.
`.semaphore/semaphore.yml`	Adds s390x build and verification blocks.

Review details

Files reviewed: 3/3 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

# Conflicts: # .semaphore/semaphore.yml

Since the free-threading work, build-wheels.sh builds a cp314t wheel on every arch, s390x included, but verify-free-threaded-job.sh only handled x64 and arm64, so the s390x cp314t wheel would have been published without verification. - tools/verify-free-threaded-job.sh: add a linux-s390x case. On s390x, install requirements-tests-install-nogil-s390x.txt and skip tests/schema_registry: the schema-registry stack pulls cryptography (via authlib and trivup's jwcrypto), which publishes no s390x wheels. Same scoping as the s390x smoke test. - requirements/requirements-tests-install-nogil-s390x.txt: the nogil test set minus schemaregistry and trivup. - .semaphore/semaphore.yml: add the "Verify free threaded" job to the Linux s390x Wheel Verification block, mirroring arm64.

The container stage of the wheel verification (tools/test-manylinux.sh) runs the smoke test on bare ubuntu images. On s390x several test and extra dependencies publish no s390x wheels and are built from source: fastavro, psutil, librt (via mypy) and cryptography (via the schema-registry deps of the [avro]/[protobuf]/[json] extras). The bare images have no compiler or headers, so the s390x verification failed there. x64 and arm64 are unaffected: these all have wheels there. - On s390x, install gcc, python3.9-dev, libssl-dev and pkg-config in the container before the smoke test. - On s390x, test on ubuntu:22.04 and 24.04 instead of 20.04: cryptography needs OpenSSL >= 3.0 to build and ubuntu:20.04 ships 1.1.1. - Correct the comments in smoketest.sh and requirements-tests-install-s390x.txt: the extras do pull the schema-registry deps, so cryptography is still built from source.

Ojasva Jain (ojasvajain) · 2026-09-25T07:59:04Z

Devarsh Patel (@Devarsh010) Free threading related changes look fine. Just a couple of nits.

Ojasva Jain (ojasvajain) · 2026-09-25T07:54:09Z

+# s390x: the schema-registry stack's deps (cryptography, via authlib) publish
+# no s390x wheels, so the s390x test installs leave them out (see
+# requirements/requirements-tests-install-s390x.txt). Everything under
+# tests/schema_registry needs them: its conftest.py imports the
+# schema-registry client, and with it authlib, at import time. So skip
+# collecting that directory on s390x. The exclusion lives here because that
+# conftest.py would fail to import before it could exclude anything itself.


nit: can we please shorten this?

Ojasva Jain (ojasvajain) · 2026-09-25T07:57:06Z

+# s390x: the schema-registry stack's deps (cryptography, via authlib and trivup's
+# jwcrypto) publish no s390x wheels, so install the reduced set (see
+# requirements/requirements-tests-install-nogil-s390x.txt). tests/conftest.py
+# skips collecting tests/schema_registry on s390x to match.


nit: You can shorten it to

# s390x: cryptography (via authlib, trivup's jwcrypto) ships no s390x wheels, # so this installs the reduced requirements-tests-install-nogil-s390x.txt; # tests/conftest.py skips tests/schema_registry there to match.

Pranav Rathi (pranavrth) · 2026-09-30T18:19:27Z

Build is failing. Also, please run the wheel generation as well as part of this PR.

Copilot

Copilot review overview

🟡 Changes recommended

The clean-container verification lacks dependencies needed to compile cryptography, and its reduced test coverage conflicts with the stated parity.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Smoke test lacks Rust and libffi build dependencies · New
s390x lane skips the entire Schema Registry suite · New

+    # smoke test installs) publish no s390x wheels and are built from source,
+    # which needs a C compiler plus the Python and OpenSSL headers.
+    if [[ $(uname -m) == s390x ]]; then
+        apt-get install -y -q gcc python3.9-dev libssl-dev pkg-config


+if platform.machine() == "s390x":
+    collect_ignore = ["schema_registry"]


Pranav Rathi (pranavrth) · 2026-10-01T09:12:33Z

+          commands:
+            - export ARCH=s390x
+            # sem-version only provides a pip to bootstrap uv; the wheel runs in the 3.14t venv uv creates.
+            - sem-version python 3.9


Can we increase this?

Pranav Rathi (pranavrth) · 2026-10-01T09:21:08Z

+        tests_install_reqs="requirements/requirements-tests-install.txt"
+        if [[ "$(uname -m)" == "s390x" ]]; then
+            tests_install_reqs="requirements/requirements-tests-install-s390x.txt"
+        fi


use if else for readability

Pranav Rathi (pranavrth) · 2026-10-01T09:22:42Z

+    # mypy, and cryptography via the schema-registry deps of the extras the
+    # smoke test installs) publish no s390x wheels and are built from source,
+    # which needs a C compiler plus the Python and OpenSSL headers.
+    if [[ $(uname -m) == s390x ]]; then


Suggested change

if [[ $(uname -m) == s390x ]]; then

if [[ $(uname -m) == "s390x" ]]; then

Pranav Rathi (pranavrth) · 2026-10-01T09:24:49Z

+    # no s390x wheel, so it is built against the system OpenSSL, and it needs
+    # OpenSSL >= 3.0 (ubuntu:20.04 ships 1.1.1).
+    if [[ -z $DOCKER_IMAGES && $(uname -m) == s390x ]]; then
+        DOCKER_IMAGES="ubuntu:22.04 ubuntu:24.04"


Use this everywhere. We don't need to run on ubuntu 20 anymore. Try adding ubuntu 26 as well.

Copilot

Copilot review overview

🔵 Needs a closer look

The clean-container smoke test lacks required source-build tooling, and Schema Registry coverage is disabled too broadly.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Smoke test lacks Rust and libffi build dependencies
s390x lane skips the entire Schema Registry suite

sonarqube-confluent · 2026-10-01T12:49:44Z

Quality Gate passed

Issues
2 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

Devarsh Patel (Devarsh010) · 2026-10-01T15:45:07Z

Copilot review overview

🔵 Needs a closer look

The clean-container smoke test lacks required source-build tooling, and Schema Registry coverage is disabled too broadly.

Review effort: Balanced Findings: 1 · 1

Open (2)

Smoke test lacks Rust and libffi build dependencies

s390x lane skips the entire Schema Registry suite

Not needed and adding cargo might break this stage. When cargo isn't on PATH, cryptography's build backend (maturin) downloads a current Rust toolchain itself. Ubuntu's apt cargo on 22.04/24.04 is 1.75 which is too old for cryptography 50: with it installed the build fails with failed to parse lock file … lock file version 4. I reproduced both on s390x. cffi installs from its s390x wheel, so libffi-dev isn't needed.
The s390x lanes skip tests/schema_registry entirely. Its conftest imports authlib (and through it cryptography) at load time, and cryptography has no s390x wheel, so the folder can't be collected there. This is the approach agreed earlier in this review, and it logs a RuntimeWarning. Once the missing dependencies (cryptography, google-re2, tink) publish s390x wheels, we can drop the s390x test sets and this skip to restore the full suite.

Pranav Rathi (pranavrth)

We need to decrease the build time. Take it as a separate item.

LGTM! Thanks Devarsh Patel (@Devarsh010) !.

Add s390x wheels to the build and verification matrix

fdba92e

Copilot AI lite review requested due to automatic review settings September 16, 2026 12:20

Devarsh Patel (Devarsh010) requested review from a team and Matthew Seal (MSeal) as code owners September 16, 2026 12:20

Copilot started reviewing on behalf of Devarsh Patel (Devarsh010) September 16, 2026 12:21 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Devarsh Patel (Devarsh010) added 4 commits September 21, 2026 13:06

Scope the s390x smoke-test deps to avoid no-wheel source builds

79242d5

Merge remote-tracking branch 'origin/master' into s390x-wheels

3321fba

# Conflicts: # .semaphore/semaphore.yml

Ojasva Jain (ojasvajain) reviewed Sep 24, 2026

View reviewed changes

Comment thread tools/verify-free-threaded-job.sh Outdated

Skip tests/schema_registry on s390x from tests/conftest.py

44e9c64

Ojasva Jain (ojasvajain) reviewed Sep 25, 2026

View reviewed changes

Devarsh Patel (Devarsh010) added 3 commits September 30, 2026 11:27

Merge master into s390x-wheels

330fbfe

Shorten the s390x comments

9851dc2

Add the s390x wheels to the CHANGELOG

d15b154

Pranav Rathi (pranavrth) requested a balanced review from Copilot September 30, 2026 18:19

Copilot started reviewing on behalf of Pranav Rathi (pranavrth) September 30, 2026 18:20 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Pranav Rathi (pranavrth) requested changes Oct 1, 2026

View reviewed changes

Address review: if/else, quoting, newer Ubuntu and Python

20ae908

Pranav Rathi (pranavrth) requested a balanced review from Copilot October 1, 2026 12:46

Copilot started reviewing on behalf of Pranav Rathi (pranavrth) October 1, 2026 12:46 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Pranav Rathi (pranavrth) approved these changes Oct 1, 2026

View reviewed changes

Devarsh Patel (Devarsh010) merged commit 7f2e72f into master Oct 1, 2026
6 checks passed

Devarsh Patel (Devarsh010) deleted the s390x-wheels branch October 1, 2026 16:09

		if platform.machine() == "s390x":
		collect_ignore = ["schema_registry"]

	if [[ $(uname -m) == s390x ]]; then
	if [[ $(uname -m) == "s390x" ]]; then

Conversation

Devarsh Patel (Devarsh010) commented Sep 16, 2026

What

Checklist

Test & Review

Uh oh!

confluent-cla-assistant Bot commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

Ojasva Jain (ojasvajain) commented Sep 25, 2026

Uh oh!

Ojasva Jain (ojasvajain) Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Ojasva Jain (ojasvajain) Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Pranav Rathi (pranavrth) commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Pranav Rathi (pranavrth) Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Pranav Rathi (pranavrth) Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Pranav Rathi (pranavrth) Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Pranav Rathi (pranavrth) Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

sonarqube-confluent Bot commented Oct 1, 2026

Quality Gate passed

Uh oh!

Devarsh Patel (Devarsh010) commented Oct 1, 2026

Copilot review overview

🔵 Needs a closer look

Uh oh!

Pranav Rathi (pranavrth) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants