diff --git a/content/topics/messaging-implementations-and-hosting/fanout-choices.svg b/content/topics/messaging-implementations-and-hosting/fanout-choices.svg new file mode 100644 index 0000000..6373139 --- /dev/null +++ b/content/topics/messaging-implementations-and-hosting/fanout-choices.svg @@ -0,0 +1,20 @@ + + Independent event subscribers + SNS plus SQS gives AWS subscriber applications separate durable queues, at the cost of per-application queues and policies. GCP Pub/Sub is managed messaging with service-specific APIs and subscription policies; workers sharing one subscription can also distribute jobs. + + + + Independent event subscribers + AWS · durable subscriber queues + → SNS + SQS + Choose: each app gets the event. + Accept: a queue + delivery policy + per subscriber application. + + GCP · managed messaging + → Pub/Sub + Choose: fan-out; share jobs within + one subscription. + Accept: GCP APIs + policies. + + diff --git a/content/topics/messaging-implementations-and-hosting/queue-choices.svg b/content/topics/messaging-implementations-and-hosting/queue-choices.svg new file mode 100644 index 0000000..f61430b --- /dev/null +++ b/content/topics/messaging-implementations-and-hosting/queue-choices.svg @@ -0,0 +1,20 @@ + + Distribute jobs among workers + SQS Standard is a managed AWS queue service; accept duplicates and out-of-order delivery. RabbitMQ is self-managed or hosted; choose for an existing RabbitMQ platform or specific broker/protocol needs and accept setup and capacity decisions. + + + + Distribute jobs among workers + AWS · managed queue service + → SQS Standard + Choose: independent worker jobs. + Accept: duplicates and delivery + that may be out of order. + + Self-managed or managed hosting + → RabbitMQ + Choose: existing RabbitMQ or its + specific broker/protocol features. + Accept: setup + capacity decisions. + + diff --git a/content/topics/messaging-implementations-and-hosting/replay-choice.svg b/content/topics/messaging-implementations-and-hosting/replay-choice.svg new file mode 100644 index 0000000..5c7144a --- /dev/null +++ b/content/topics/messaging-implementations-and-hosting/replay-choice.svg @@ -0,0 +1,14 @@ + + Retained history and replay + Choose Apache Kafka for retained history, replay, or a partitioned log. Partitioning, retention, and consumer progress still need design with managed hosting. + + + + Retained history and replay + Self-managed or managed hosting + → Apache Kafka + Choose: history, replay, log model. + Accept: partitioning, retention, + and consumer-progress design. + + diff --git a/content/topics/messaging-implementations-and-hosting/topic.md b/content/topics/messaging-implementations-and-hosting/topic.md index 75d8c8f..5b73729 100644 --- a/content/topics/messaging-implementations-and-hosting/topic.md +++ b/content/topics/messaging-implementations-and-hosting/topic.md @@ -2,7 +2,7 @@ id: 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' title: 'Messaging implementations and hosting' status: 'draft' -summary: 'Name an engine or cloud service that fits the messaging model and operational constraints.' +summary: 'Narrow an interview scenario to five messaging options using workload, platform, and one tradeoff.' iconKey: 'queue' domains: - 'system-design' @@ -11,94 +11,21 @@ childTopicIds: [] relatedTopicIds: [] --- -## Narrow the candidates +![Worker-job choices: SQS Standard for a managed AWS queue, accepting duplicates and out-of-order delivery; RabbitMQ for an existing platform or its broker features, accepting setup and capacity decisions.](./queue-choices.svg) -- **Background jobs:** RabbitMQ; AWS SQS; GCP Pub/Sub with workers sharing a subscription -- **Independent event subscribers:** RabbitMQ exchanges + queues; AWS SNS + SQS; GCP Pub/Sub -- **Retained log and replay:** Apache Kafka; AWS MSK; GCP Managed Service for Apache Kafka -- **Existing JMS broker integration:** ActiveMQ Classic; Amazon MQ for ActiveMQ Classic +![Fan-out choices: SNS plus SQS on AWS needs a queue and policy per subscriber application; GCP Pub/Sub uses service-specific APIs and subscription policies and also supports workers sharing one subscription.](./fanout-choices.svg) -These are starting points, not exclusive capabilities. Choose the model first, then check ordering, replay, failure handling, and existing platform constraints. +![History and replay: Apache Kafka, self-managed or hosted, still needs partitioning, retention, and consumer-progress design.](./replay-choice.svg) -## Engines: self-managed or hosted +> **Thumbnail jobs:** compare **SQS Standard** and **RabbitMQ** first; **Pub/Sub** is a GCP option. Retained-history or replay requirements give Kafka a stronger reason. -### RabbitMQ +**Broker setup ≠ self-managed hosting.** Defining queues, routing rules, and retries also applies to a hosted broker. Self-management adds servers, upgrades, availability, and recovery. -- **Fit:** job queues and routed pub/sub through exchanges and queues. -- **Tradeoff:** topology, durability, acknowledgements, and capacity still need design. RabbitMQ also supports streams; it is not limited to queues. -- **Hosting:** run it yourself or use a managed provider, including Amazon MQ. - -### ActiveMQ Classic - -- **Fit:** existing JMS applications and broker/protocol compatibility. -- **Tradeoff:** select and operate the broker topology and verify client/version compatibility. -- **Hosting:** self-managed or Amazon MQ. Specify **Classic**; do not assume Amazon MQ hosts Artemis. - -### Apache Kafka - -- **Fit:** retained event history, replay, and the Kafka processing ecosystem. -- **Tradeoff:** partition keys, retention, consumer lag, and ordering remain design responsibilities even with managed hosting. -- **Hosting:** self-managed, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or another provider. - -## AWS services - -### SQS Standard - -Independent jobs; no strict ordering needed. - -**Tradeoff:** At-least-once delivery; duplicates and out-of-order delivery are possible. - -### SQS FIFO - -Ordering within message groups and send deduplication matter. - -**Tradeoff:** Group design constrains parallelism; retries can still repeat business effects. - -### SNS + SQS - -Fan-out with a durable queue for each consumer application. - -**Tradeoff:** More queues, subscriptions, and failure policies to manage. - -### Amazon MQ - -Managed RabbitMQ or ActiveMQ Classic; engine/protocol compatibility matters. - -**Tradeoff:** Broker capacity and engine-specific configuration still matter. - -### MSK - -Managed Kafka and its APIs/ecosystem. - -**Tradeoff:** Kafka data-model and consumer responsibilities remain. - -**Standard and FIFO are SQS queue types, not pricing tiers.** FIFO deduplication is bounded; a received message that is not deleted can become visible again. - -## GCP services - -### Pub/Sub - -Managed fan-out; shared subscriptions can distribute worker jobs. - -**Tradeoff:** Service-specific API and subscription, retention, and ordering policies. - -### Managed Service for Apache Kafka - -Kafka APIs, tooling, and partitioned-log model are required. - -**Tradeoff:** Kafka partitions, retention, and consumer behavior still need design. - -## Who operates what? - -- **Self-managed engine:** your team owns provisioning, upgrades, storage, replication, monitoring, and recovery. -- **Managed engine:** provider operates the broker infrastructure within the service contract; your team still owns topology, configuration, clients, and workload behavior. -- **Managed service API:** no broker cluster to operate directly; accept service-specific APIs, limits, and guarantees. +These are starting points, not exclusive capabilities. Standard is an SQS queue type; ordering requirements may lead to FIFO. Managed hosting reduces infrastructure work, not application delivery responsibilities. ## Sources -- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [streams](https://www.rabbitmq.com/docs/streams) -- [ActiveMQ Classic features](https://activemq.apache.org/components/classic/documentation/features) -- [Amazon MQ engines](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html) -- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery logic](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html) -- [Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) +- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html) +- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [managed RabbitMQ on Amazon MQ](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html) +- [SNS + SQS fan-out](https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html) - [Google Pub/Sub versus managed Kafka](https://docs.cloud.google.com/pubsub/docs/choose-pubsub-kafka)