Message Queues Explained: Async Processing and Kafka Basics

Some work should not happen while a user waits. Sending an email, resizing an image or generating a report can take seconds. A message queue lets a request record the work and return at once, while background workers do it later.

Without a queue:  User --> API --> send email (2 s) --> response
With a queue:     User --> API --> queue --> response (fast)
                                    |
                                    v
                                  Worker --> send email

Why use a queue?#

• Faster responses: users do not wait for slow work.
• Absorb spikes: a queue grows during a burst, and workers catch up.
• Decoupling: the producer does not need to know who consumes.
• Reliability: if a worker crashes, the message is still there.
• Scaling: add workers to process faster.

A burst shows the benefit:

workers_per_minute = 100
arrivals = [50, 400, 400, 50, 50, 50, 50, 50, 50, 50, 50]

backlog = 0
for minute, new in enumerate(arrivals):
    backlog = max(0, backlog + new - workers_per_minute)
    print(f"Minute {minute}: {new:>3} arrived, backlog {backlog}")

Minute 0:  50 arrived, backlog 0
Minute 1: 400 arrived, backlog 300
Minute 2: 400 arrived, backlog 600
Minute 3:  50 arrived, backlog 550
Minute 4:  50 arrived, backlog 500
Minute 5:  50 arrived, backlog 450
Minute 6:  50 arrived, backlog 400
Minute 7:  50 arrived, backlog 350
Minute 8:  50 arrived, backlog 300
Minute 9:  50 arrived, backlog 250
Minute 10:  50 arrived, backlog 200

Nothing was lost: the queue stored the burst, and the backlog drained.

Queue or log#

Publish/subscribe sends one event to many services. An OrderPlaced event might go to email, stock and analytics, which do not need to know about each other.

Kafka in brief#

A topic is split into partitions, which give parallelism. Messages with the same key go to the same partition, so they stay in order:

import hashlib

partitions = 3


def partition_for(key):
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % partitions


for order in ["order-1", "order-1", "order-2", "order-1", "order-3"]:
    print(order, "-> partition", partition_for(order))

order-1 -> partition 1
order-1 -> partition 1
order-2 -> partition 2
order-1 -> partition 1
order-3 -> partition 2

All events for order-1 stay in one partition, in order. Within a consumer group, each partition has one consumer, so more partitions allow more parallel consumers.

Delivery guarantees#

Most systems use at least once, so consumers must handle duplicates. An idempotent consumer gets the same result if a message is processed twice:

processed = set()
balance = 0


def handle(message):
    global balance
    if message["id"] in processed:
        print("Duplicate ignored:", message["id"])
        return
    processed.add(message["id"])
    balance += message["amount"]


for m in [{"id": "t1", "amount": 100}, {"id": "t2", "amount": 50}, {"id": "t1", "amount": 100}]:
    handle(m)
print("Balance:", balance)

Duplicate ignored: t1
Balance: 150

In production, keep processed IDs in a database, in the same transaction as the change.

When messages fail#

• Retries: try again, with backoff.
• Dead-letter queue: after several failures, move the message aside, so that one bad message does not block the rest, and inspect it later.

Backpressure#

If producers are always faster than consumers, the backlog grows without limit. Watch the queue length or consumer lag. Then add workers, slow producers down, or reject work with a clear error.

Saving to a database and publishing#

Saving an order and publishing an event are two separate steps, so one can fail. In the outbox pattern, you write the event to a database table in the same transaction, and a separate process publishes from there.

Common mistakes#

Practice#

import math

per_worker = 60 / 3
print(math.ceil(500 / (per_worker * 0.7)))

36

Key takeaway#

Next: API Design.