Repository navigation
Conversation
Tests written first (TDD red phase): - merge_async_iterables: merging multiple async generators, error propagation, closing inner generators on consumer close. - MultiBroker: constructor validation, kick routing by queue_name label, merged listen over all inner brokers, startup/shutdown delegation, process flag propagation, task registry aggregation, result backend propagation, end-to-end Receiver test proving one worker consumes multiple queues.
MultiBroker combines several real brokers so a single worker consumes messages from all of them at the same time: - listen() merges listen generators of all inner brokers via the new merge_async_iterables util (errors propagate, inner generators are closed on shutdown); - kick() routes messages by the queue_name label when a routing mapping is configured, otherwise the default broker is used; - startup/shutdown delegate to all inner brokers, a failing shutdown of one broker doesn't prevent others from stopping; - is_worker_process/is_scheduler_process flags propagate to inner brokers so they fire correct lifecycle events; - find_task/get_all_tasks aggregate task registries of inner brokers, so tasks can be registered on any of them; - with_result_backend() propagates the backend to inner brokers to keep client-side polling and worker-side saving in sync. Also refines the inner-broker-error test to assert the contract at the MultiBroker.listen level: Receiver wraps any broker error in an anyio ExceptionGroup, which is existing taskiq behavior.
Previously an error escaping an inner broker's listen() killed the whole merged listening (fail-fast), taking down all remaining healthy brokers with it. For multi-cluster setups (e.g. two RabbitMQ clusters or RabbitMQ + Kafka) that's the wrong trade-off: one dead cluster should not stop processing of the others. Now MultiBroker.listen() wraps every inner broker in a guard generator: - an inner broker failure is logged (with traceback) and the failed source is excluded from listening; remaining brokers keep working; - only when every inner broker is down does the listening end, so the worker process exits and is restarted by the supervisor (default --max-fails -1 restarts indefinitely); - shutdown paths are unchanged: cancellation and aclose still close inner generators deterministically (guard's finally acloses the inner iterator). merge_async_iterables keeps its fail-fast contract; the degradation policy lives in MultiBroker where the logging context is.
Restructures the MultiBroker docs section and spells out the contract: - How it works: merged listening, kick paths, lifecycle delegation, flag propagation, deterministic shutdown cleanup. - Failure handling: connection flaps are handled by inner brokers themselves (e.g. aio-pika RobustConnection); if listening does fail, the broker is excluded (logged with traceback) and the worker keeps consuming the rest; exclusion is permanent until process restart; all brokers down -> worker exits and the supervisor restarts it (default --max-fails -1 restarts indefinitely); in-flight messages complete; kicks are unaffected by worker-side degradation. - Guarantees table: what the composite provides (merge, routing, degradation, graceful shutdown) vs what must come from inner brokers (reconnection, delivery guarantees, redelivery/dead-lettering, per-queue concurrency limits, ordering). Also mirrors the failure summary in the class docstring.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #723 +/- ##
==========================================
+ Coverage 85.49% 86.02% +0.52%
==========================================
Files 68 69 +1
Lines 2662 2790 +128
==========================================
+ Hits 2276 2400 +124
- Misses 386 390 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The pre-commit mypy hook type-checks tests as well, while only 'taskiq' was checked locally: - annotate the merged list in the empty-iterables test and cast the generator, since the TypeVar cannot be inferred from an empty argument list; - parameterize InmemoryResultBackend instantiation; - align DyingBroker.listen signature with the parent broker (AsyncGenerator[AckableMessage, None]) and yield AckableMessage.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Core taskiq binds one worker process to a single broker with a single
listen()generator. Running multiple queues today means either multipleworker deployments or relying on per-broker multi-queue support (which
only some broker packages have). Multi-queue support in core is still an
open discussion (TRFC-2, org discussion #448), with draft PRs #486/#624.
Solution
Additive
MultiBroker: a composite broker wrapping N real brokers(any
AsyncBrokerimplementations — taskiq-aio-pika, taskiq-redis,taskiq-aio-kafka, etc.) so a single
taskiq workerconsumes all ofthem. No changes to the CLI, the Receiver, or external packages.
What it does
listen()merges listen-generators of all inner brokers into a singlestream (new
merge_async_iterablesutil intaskiq.utils);kick()routes by thequeue_namelabel when aroutingmapping isconfigured, otherwise uses the default broker;
startup()/shutdown()delegate to all inner brokers; one failingshutdown doesn't prevent the rest from stopping;
is_worker_process/is_scheduler_processpropagate, so innerbrokers fire their WORKER_* lifecycle events;
find_task()/get_all_tasks()aggregate task registries, so tasksmay be registered on any inner broker;
with_result_backend()propagates the backend to inner brokers to keepclient-side polling and worker-side saving in sync.
Notes
task_queuesof taskiq-aio-pika >= 0.6.0 keeps a single connectionand remains preferable;
MultiBrokertargets the multi-broker /multi-cluster case (documented, including the one-connection-per-broker
trade-off).
Queuedataclass / ABCcontract changes) — it solves the "one worker, many queues/brokers"
use case additively and can serve as a building block for the full
TRFC-2 design.