- 01 Terminology Glossary
- 02 Real-Time Chat (WhatsApp)
- 03 Video Streaming (YouTube)
- 04 Low-Latency Systems
- 05 Ordered Message Queues
- 06 URL Shortener
- 07 Rate Limiter
- 08 Notification System
- 09 News Feed
- 10 Offline Messaging
- 11 Circuit Breaker
- 12 Self-Healing Infrastructure
- 13 Scaling to 5M Users
- 14 Handling 5M Requests
- 15 End-to-End Encryption
- 16 Real-Time React Dashboard
- 17 Fast-to-Slow File Transfer
- 18 Ride-Hailing (Uber, Ola)
- 19 Ticket Booking (BookMyShow)
- 20 Unique ID Generator
Blueprint 05 · 4 diagrams
Design for Message Order in Queues
Message order and correct processing in queues
1. Summary
Many systems must process messages in the correct order. Examples: bank transactions, order status changes, and chat messages. A distributed queue does not keep a global order by default. This document explains how to keep the order and process each message correctly.
Important: A global order for all messages is slow and does not scale. Most systems need order only for each key (for example, each account or each order). Design for order per key.
2. Why order breaks
flowchart LR
P["Producer"] -- "M1, M2, M3" --> Q["Queue"]
Q --> C1["Consumer 1: gets M1 (slow)"]
Q --> C2["Consumer 2: gets M2 (fast)"]
Q --> C3["Consumer 3: gets M3"]
C2 --> R["Result: M2 before M1"]
Order breaks for these reasons:
- Many consumers process messages in parallel.
- A retry sends a failed message again after newer messages.
- Many producers send messages at the same time.
- Network delays change the arrival order.
- Messages go to different partitions.
3. Solution: partition by key
flowchart LR
P["Producer"] --> H{"hash(order_id) mod N"}
H --> P0["Partition 0: A1, A2, A3"]
H --> P1["Partition 1: B1, B2"]
H --> P2["Partition 2: C1, C2, C3"]
P0 --> C0["Consumer 0"]
P1 --> C1["Consumer 1"]
P2 --> C2["Consumer 2"]
Procedure
- Choose the ordering key (for example,
account_id). - Send all messages with the same key to the same partition.
- Give each partition to one consumer only.
- Process the messages in each partition one at a time.
Result: messages for one key stay in order. Different keys process in parallel. Thus, the system still scales.
Implementation in common tools
| Tool | Method |
|---|---|
| Apache Kafka | Use a message key. Kafka sends the same key to the same partition. One consumer in a group reads one partition. |
| AWS SQS FIFO | Use MessageGroupId. SQS keeps order in each group. Use MessageDeduplicationId to remove duplicates. |
| RabbitMQ | Use one queue with one consumer for each key range. Or use the consistent-hash exchange. |
| Azure Service Bus | Use sessions (SessionId). |
| Google Pub/Sub | Use ordering keys. |
4. Safe producer settings (Kafka)
- Set
enable.idempotence=true. The broker then removes duplicates from producer retries. - Set
acks=all. The broker confirms the write after all in-sync replicas have the message. - Keep
max.in.flight.requests.per.connectionat 5 or less with idempotence. This keeps order during retries.
5. Sequence numbers
The producer adds a sequence number to each message for each key.
sequenceDiagram
participant Q as Queue
participant C as Consumer
participant S as State store
Q->>C: Message (key=A, seq=5)
C->>S: Read last_seq for A
S-->>C: last_seq = 4
C->>C: seq 5 = last_seq + 1. Process.
C->>S: Save last_seq = 5
Q->>C: Message (key=A, seq=5) duplicate
C->>C: seq 5 is not more than 5. Discard.
Q->>C: Message (key=A, seq=7)
C->>C: Gap. seq 6 is missing. Hold or wait.
The consumer uses the sequence number to:
- Find and discard duplicates.
- Find gaps (missing messages).
- Hold a message until the previous message arrives.
6. Idempotent consumers
Most queues give "at-least-once" delivery. Thus, a consumer can get the same message two times. Make each consumer idempotent. The result of two runs must be the same as the result of one run.
Procedure
- Give each message a unique
message_id. - Start a database transaction.
- Write the
message_idto a "processed messages" table with a unique constraint. - Apply the business change in the same transaction.
- Commit the transaction.
- Commit the queue offset.
If step 3 fails because the ID exists, the consumer skips the message.
7. Transactional outbox pattern
Problem: the service writes to the database and also sends a message. One of these two operations can fail. Then the data is not consistent.
flowchart LR
S["Order service"] -- "one transaction" --> DB[("Database")]
DB --> T1["orders table"]
DB --> T2["outbox table"]
T2 --> R["Relay or CDC Debezium"]
R -- "in commit order" --> K["Kafka"]
K --> C["Consumers"]
Procedure
- Write the business data and the event to the outbox table in one transaction.
- Use a relay process or Change Data Capture to read the outbox table.
- Send the events to the queue in commit order.
- Mark each event as sent.
8. Retries without loss of order
A failed message must not let later messages for the same key go first.
| Strategy | Order | Throughput | Use |
|---|---|---|---|
| Blocking retry | Kept | Lower: the partition stops | Use for short, temporary errors. |
| Retry topic | Broken for that key | High | Use when order is not important. |
| Park the key | Kept for that key | High | Hold all later messages for the failed key. Process other keys. |
| Dead letter queue (DLQ) | Depends on the method | High | Use for messages that always fail. Send an alert. |
Recommended method: retry a small number of times with backoff. Then park the key and send the message to a DLQ. Send an alert to the operations team.
9. Exactly-once processing
- True "exactly-once delivery" across systems is not practical.
- Get an "exactly-once effect" with this formula: at-least-once delivery + idempotent processing.
- Kafka transactions give exactly-once for "read from Kafka, process, write to Kafka" flows.
10. Global order (only when necessary)
If the business needs one total order for all messages:
- Use one partition. This limits throughput to one consumer.
- Or use a single sequencer service that gives a global sequence number. This sequencer can become a bottleneck.
- Or use a consensus log (Raft, for example etcd) for low-volume, critical events.
11. Trade-offs
- More partitions: More partitions give more parallel work. But you cannot easily change the key-to-partition map later.
- Blocking retry: Blocking keeps order. But one bad message stops its partition.
- Hot keys: One very active key can overload one partition. Split the key (for example,
account_id + sub_key) only if order across the parts is not necessary.
12. Interview follow-up questions
- How do you keep order when you increase the number of partitions?
- What happens to order during a consumer group rebalance?
- How do you process a payment exactly one time?
- How do you handle a hot partition?
- What is the difference between Kafka and RabbitMQ for ordered processing?