Scalable Topics: producer end-to-end tests + run pulsar::st tests in CI#603
Open
merlimat wants to merge 5 commits into
Open
Scalable Topics: producer end-to-end tests + run pulsar::st tests in CI#603merlimat wants to merge 5 commits into
pulsar::st tests in CI#603merlimat wants to merge 5 commits into
Conversation
Validate the producer against a real scalable-topics broker and run the st tests in CI (they were built but never executed before). - tests/st/StProducerE2ETest.cc: end-to-end producer tests against a standalone broker — keyed + keyless publishing that fans out to segments, asserting each send returns a segment-qualified MessageId, lastSequenceId advances, and flush/close succeed; plus an async-batch case exercising the in-flight tracking. Gated on PULSAR_ST_E2E so the ordinary broker-free run skips them. - tests/st/docker-compose.yml: a plain apachepulsar/pulsar:latest (5.0.0-M1) standalone broker; it ships the scalable-topics controller and wire protocol with no extra config. - run-unit-tests.sh: bring the broker up, create the scalable topic the producer publishes to (scalable topics are a managed construct — not auto-created on lookup like regular topics), then run the full pulsar-st-tests (88 broker-free cases + the e2e), and tear the broker down. Verified locally: the whole produce path works against the M1 broker (DAG-watch lookup, key routing, per-segment producer creation, MessageId minting); 89 tests pass with the broker up, and the e2e cases skip cleanly without it.
The scalable topic is pre-created not because scalable topics are inherently managed/non-auto-created, but because the pinned 5.0.0-M1 broker image has a bug where create_if_missing does not auto-create on lookup — already fixed in master. Reword the comment so the pre-create can be dropped once the image carries the fix.
The existing e2e cases produce to a single-segment topic, so every key routes to that one segment and cross-segment routing is never exercised. Add a case that publishes to a topic the harness has split into two active half-range segments and asserts the produced message ids span both segments. The harness creates the topic and splits segment 0 (which seals it and yields two active children) before the run.
Add two more producer e2e cases so split/merge topology changes are exercised against a real broker: - testProduceAfterMerge: publish to a topic the harness split then merged back. A merge seals both children and creates one full-range active segment, so every key routes to it — this drives the layout parser and router on a DAG that carries several sealed segments. - testProduceContinuesAcrossLiveSplit: warm up a per-segment producer, then seal that segment (split it) out from under the producer while a burst of async sends is in flight. Sends that hit the sealed segment fail with TopicTerminated and the producer retries + re-routes onto the layout the DAG watch delivers; every send must ultimately succeed and publishing must reach the post-split segments. The split is triggered through the admin command the harness passes in PULSAR_ST_E2E_SPLIT_CMD, keeping the container name out of the test. The harness creates the merge topic (create -> split -> merge) and the live-split topic, and exports the split command.
CI pulled apachepulsar/pulsar:latest, which is a stable release whose pulsar-admin
does not know the scalable-topics command ("Unmatched arguments"), so topic setup
failed. The moving :latest tag does not carry scalable topics yet; the published
5.0.0-M1 milestone does. Pin to it. (A local :latest tagged as an M1 snapshot is
what made this pass locally — that tag is not what CI pulls.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #601 (the
pulsar::stproducer). That PR shipped the producer with broker-free unit tests; this one validates it end-to-end against a real scalable-topics broker and runs thepulsar::sttests in CI for the first time (they were built but never executed).Modifications
tests/st/StProducerE2ETest.cc— end-to-end producer tests against a standalone broker. A keyed + keyless case that fans out across segments, asserting every send returns a segment-qualifiedMessageId,lastSequenceIdadvances, andflush()/close()succeed; and an async-batch case that fires manysendAsync()calls across keys and awaits them all (exercising the in-flight tracking and flush). Both are gated on thePULSAR_ST_E2Eenvironment variable, so the ordinary broker-free unit-test run skips them.tests/st/docker-compose.yml— a single standaloneapachepulsar/pulsar:latestbroker for the e2e run. The image ships the scalable-topics controller and wire protocol; no extra broker config is needed to enable them.run-unit-tests.sh— a scalable-topics phase: bring the broker up, create the scalable topic the producer publishes to, run the fullpulsar-st-tests(the 88 broker-free cases plus the e2e), then tear the broker down. This is the first place anypulsar-st-testsrun in CI.What this validates
The whole producer data path works against the real broker: the DAG-watch lookup resolves the scalable topic, keys route to segments, a per-segment producer is created on the
segment://…topic, and each publish comes back with a segment-qualifiedMessageId.Notes
apachepulsar/pulsar:latestdoes not auto-create a scalable topic on lookup (a bug fixed in later releases), so the harness pre-creates it withpulsar-admin scalable-topics create. That step is harmless once the image carries the fix — the producer'screate_if_missinglookup finds the topic either way — and can be dropped then.MessageIdon each send is the landing check (the scalable-topic admin stats expose only DAG structure, not per-segment message counts).Verifying this change
Documentation
docdoc-not-needed