Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
93 changes: 10 additions & 83 deletions content/topics/messaging-implementations-and-hosting/topic.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
title: 'Messaging implementations and hosting'
status: 'draft'
summary: 'Name an engine or cloud service that fits the messaging model and operational constraints.'
summary: 'Narrow an interview scenario to five messaging options using workload, platform, and one tradeoff.'
iconKey: 'queue'
domains:
- 'system-design'
Expand All @@ -11,94 +11,21 @@ childTopicIds: []
relatedTopicIds: []
---

## Narrow the candidates
![Worker-job choices: SQS Standard for a managed AWS queue, accepting duplicates and out-of-order delivery; RabbitMQ for an existing platform or its broker features, accepting setup and capacity decisions.](./queue-choices.svg)

- **Background jobs:** RabbitMQ; AWS SQS; GCP Pub/Sub with workers sharing a subscription
- **Independent event subscribers:** RabbitMQ exchanges + queues; AWS SNS + SQS; GCP Pub/Sub
- **Retained log and replay:** Apache Kafka; AWS MSK; GCP Managed Service for Apache Kafka
- **Existing JMS broker integration:** ActiveMQ Classic; Amazon MQ for ActiveMQ Classic
![Fan-out choices: SNS plus SQS on AWS needs a queue and policy per subscriber application; GCP Pub/Sub uses service-specific APIs and subscription policies and also supports workers sharing one subscription.](./fanout-choices.svg)

These are starting points, not exclusive capabilities. Choose the model first, then check ordering, replay, failure handling, and existing platform constraints.
![History and replay: Apache Kafka, self-managed or hosted, still needs partitioning, retention, and consumer-progress design.](./replay-choice.svg)

## Engines: self-managed or hosted
> **Thumbnail jobs:** compare **SQS Standard** and **RabbitMQ** first; **Pub/Sub** is a GCP option. Retained-history or replay requirements give Kafka a stronger reason.

### RabbitMQ
**Broker setup ≠ self-managed hosting.** Defining queues, routing rules, and retries also applies to a hosted broker. Self-management adds servers, upgrades, availability, and recovery.

- **Fit:** job queues and routed pub/sub through exchanges and queues.
- **Tradeoff:** topology, durability, acknowledgements, and capacity still need design. RabbitMQ also supports streams; it is not limited to queues.
- **Hosting:** run it yourself or use a managed provider, including Amazon MQ.

### ActiveMQ Classic

- **Fit:** existing JMS applications and broker/protocol compatibility.
- **Tradeoff:** select and operate the broker topology and verify client/version compatibility.
- **Hosting:** self-managed or Amazon MQ. Specify **Classic**; do not assume Amazon MQ hosts Artemis.

### Apache Kafka

- **Fit:** retained event history, replay, and the Kafka processing ecosystem.
- **Tradeoff:** partition keys, retention, consumer lag, and ordering remain design responsibilities even with managed hosting.
- **Hosting:** self-managed, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or another provider.

## AWS services

### SQS Standard

Independent jobs; no strict ordering needed.

**Tradeoff:** At-least-once delivery; duplicates and out-of-order delivery are possible.

### SQS FIFO

Ordering within message groups and send deduplication matter.

**Tradeoff:** Group design constrains parallelism; retries can still repeat business effects.

### SNS + SQS

Fan-out with a durable queue for each consumer application.

**Tradeoff:** More queues, subscriptions, and failure policies to manage.

### Amazon MQ

Managed RabbitMQ or ActiveMQ Classic; engine/protocol compatibility matters.

**Tradeoff:** Broker capacity and engine-specific configuration still matter.

### MSK

Managed Kafka and its APIs/ecosystem.

**Tradeoff:** Kafka data-model and consumer responsibilities remain.

**Standard and FIFO are SQS queue types, not pricing tiers.** FIFO deduplication is bounded; a received message that is not deleted can become visible again.

## GCP services

### Pub/Sub

Managed fan-out; shared subscriptions can distribute worker jobs.

**Tradeoff:** Service-specific API and subscription, retention, and ordering policies.

### Managed Service for Apache Kafka

Kafka APIs, tooling, and partitioned-log model are required.

**Tradeoff:** Kafka partitions, retention, and consumer behavior still need design.

## Who operates what?

- **Self-managed engine:** your team owns provisioning, upgrades, storage, replication, monitoring, and recovery.
- **Managed engine:** provider operates the broker infrastructure within the service contract; your team still owns topology, configuration, clients, and workload behavior.
- **Managed service API:** no broker cluster to operate directly; accept service-specific APIs, limits, and guarantees.
These are starting points, not exclusive capabilities. Standard is an SQS queue type; ordering requirements may lead to FIFO. Managed hosting reduces infrastructure work, not application delivery responsibilities.

## Sources

- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [streams](https://www.rabbitmq.com/docs/streams)
- [ActiveMQ Classic features](https://activemq.apache.org/components/classic/documentation/features)
- [Amazon MQ engines](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html)
- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery logic](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html)
- [Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html)
- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html)
- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [managed RabbitMQ on Amazon MQ](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html)
- [SNS + SQS fan-out](https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html)
- [Google Pub/Sub versus managed Kafka](https://docs.cloud.google.com/pubsub/docs/choose-pubsub-kafka)
Loading