feat(testing): overhaul test harness with docker compose and restructure kazoo.testing package - #802
Open
ceache wants to merge 10 commits into
Open
feat(testing): overhaul test harness with docker compose and restructure kazoo.testing package#802ceache wants to merge 10 commits into
ceache wants to merge 10 commits into
Conversation
…ure kazoo.testing package - Replace legacy ZookeeperCluster process driver with testcontainers 4.x using base+overlay compose layouts (docker-compose.base.yml). - Add auth matrix overlays for digest, sasl-digest, tls, and sasl-gssapi with certgen and alpine KDC sidecars. - Scaffold docker-compose.capture.yml feature overlay and tshark sidecar for pcapng network packet capture, splitting ensemble into netns-holder + service pairs. - Emit TLS keylog files and context certificates from tls-secrets-agent for Wireshark/pcapng TLS decryption. - Restructure kazoo.testing package into kazoo.testing.common (core logic, helpers) and kazoo.testing.fixtures (pytest plugins), retiring harness.py. - Move compose overlays, JAAS configs, and Dockerfile resources into kazoo.testing package using importlib.resources. - Migrate integration tests off KazooTestCase to zkclient/zkensemble pytest fixtures and retire legacy root compose files. - Add registered pytest markers (zk_version, zk_auth, zk_features) with collection-time evaluation and skip reasons. - Introduce ZK axis enums (backports.strenum / enum.StrEnum) and bump default ZK version to 3.9.5. - Add contract & unit tests reaching 93%+ coverage on kazoo.testing pure helpers and subprocess functions. - Optimize GitHub Actions workflows with test matrices, docker layer caching, and Windows WSL2 runner support.
KazooClient.command() used getpeername()[1], which is the remote port, as the TLS hostname for SNI. Under TLS, ssl._encode_hostname rejects the integer, so every server_version()/command() call failed (e.g. the tls integ suite: test_queue._skip_unless_zk34, test_client server_version and command tests). Use the peer host address instead. Regression test: kazoo/tests/unit/test_client_command.py Docs: CHANGES.md note
- Unskip and stabilize test_request_queuing_session_expired by synchronizing on async result resolution and eliminating state listener race conditions with queue assertions. - Harden test_request_queuing_session_recovered to verify queued async creation and znode existence without race conditions. - Replace fragile polling loop in test_add_auth_on_reconnect and fixed sleep in test_update_host_list with event-driven state listener synchronization. - Increase bounded event wait timeout in test_bad_session_expire for test reliability under load. - Record testing bug fixes in CHANGES.md under unreleased changes.
…nable failover retry
When _read_header hit ConnectionDropped during SASL handshake, it previously
raised AuthFailedError("Connection dropped in SASL"). In _connect_attempt,
AuthFailedError transitioned the client state to AUTH_FAILED and returned
STOP_CONNECTING, permanently halting the connection retry loop during cluster
failover instead of retrying available hosts. Propagating ConnectionDropped
allows the retry loop to continue to other cluster hosts.
Also add diagnostic error message to failover assertion in test_update_host_list.
…ASLException Defer client._session_callback(KeeperState.CONNECTED) until after _authenticate_with_sasl and initial auth_data packet dispatch complete. Previously, the client reported CONNECTED before authentication finished, allowing client operations to race auth processing and falsely signalling readiness on servers enforcing authentication (enforce.auth.enabled=true). Also catch SASLException alongside AuthFailedError in _connect_attempt to cleanly transition client state to AUTH_FAILED.
…race
In test_connect_auth, client2 creates node /1 with ACLs restricting read
access. Follower reads in ZooKeeper are sequentially consistent, but can
briefly lag behind the leader until the commit is applied locally; if
client1 connects to a lagging follower, get("/1") could intermittently raise
NoNodeError instead of NoAuthError.
Calling client1.sync("/1") flushes the replication channel from the leader
to client1's connected follower before reading, eliminating transient
NoNodeError without polling loops.
…adlock An unconnected TCP socket created with socket.socket(AF_INET, SOCK_STREAM) is reported as writable in Linux epoll (TCP_CLOSE state), but BSD/XNU kqueue on macOS requires an established connection (SS_ISCONNECTED) to signal EVFILT_WRITE. Calling selector_select with timeout=None on an unconnected socket caused a permanent deadlock on Darwin. Using socket.socketpair() provides a connected stream socket pair supported on Unix and Windows that is immediately writable on all platforms, eliminating the macOS deadlock and removing the need for a platform skip marker while preserving the list-mutation assertion.
ceache
requested review from
StephenSorriaux,
a-ungurianu,
bbangert and
jeffwidman
September 10, 2026 07:12
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #802 +/- ##
==========================================
- Coverage 96.65% 92.27% -4.38%
==========================================
Files 27 30 +3
Lines 3554 4259 +705
==========================================
+ Hits 3435 3930 +495
- Misses 119 329 +210 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ceache
marked this pull request as ready for review
September 10, 2026 07:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #
Why is this needed?
The legacy test harness (
kazoo.testing.harness) relied on local JVM/ZooKeeper binary installations, custom environment setup scripts (ensure-zookeeper-env.sh), and hardcoded port configurations. This created multiple maintenance and developer experience challenges:Proposed Changes
This PR introduces a modular, containerized test harness powered by Docker Compose and pytest fixtures, separates offline unit tests, and hardens client/protocol reliability:
Docker Compose Test Harness (
kazoo.testing):harness.pywith modular packages:kazoo.testing.common(axis resolution, mount handling, ensemble state) andkazoo.testing.fixtures(pytest fixtures).zoo1,zoo2,zoo3).superDigest), SASL Digest (DIGEST-MD5), SASL GSSAPI (with embedded MIT Kerberos KDC sidecar), and mutual TLS (withcertgenandtls-secrets-agentsidecars).tshark) generating.pcapngtraces and TLS keylog material.2181/2281), eliminating static port math and port collisions.zkchroot) shared cleanly acrosszkclientandzksuperadmin_client.docker_envsnapshot and restoration on session teardown.KAZOO_TESTING_ZK_*environment variables (with fallback toZK_*in CLI option parsing).zk_version,zk_auth,zk_features) and CLI options (--zk-version,--zk-auth,--zk-features).Offline Unit Test Separation (
kazoo/tests/unit):pytest kazoo/tests/unitruns 192 tests in ~12 seconds with zero network or Docker dependencies.kazoo/tests/unit/test_testing.pyproviding comprehensive coverage for harness logic and fixtures.Style & Python 3.12+ AST Compatibility:
black==24.10.0and commit registered in.git-blame-ignore-revs.Client & Protocol Bug Fixes:
fix(client): Pass peer hostname instead of port as TLS SNI server name inKazooClient.command().fix(protocol): PropagateConnectionDroppedduring SASL handshake to enable cluster failover retry instead of stopping connection attempts.fix(protocol): DeferCONNECTEDstate notification until SASL and initial auth data dispatch complete, and handleSASLExceptioncleanly.test(client): Eliminate flakiness in request queuing and reconnect tests by awaiting explicit server state and synchronization events.test(integ): Flush replication viaclient1.sync("/1")intest_connect_authto prevent replication lag assertions.test(unit): Usesocket.socketpair()intest_select_mutatedto eliminate macOS pipe deadlock.Does this PR introduce any breaking change?
kazoo.testing.harness.KazooTestCasetest base class is replaced by standard pytest fixtures (zkensemble,zkclient,zksuperadmin_client,zkchroot).KAZOO_TESTING_ZK_*prefix (legacyZK_*variable names remain supported as fallbacks in CLI resolution).