Load Balancing Explained: Algorithms and Layer 4 vs Layer 7

A load balancer sits in front of your servers, and spreads requests among them. It lets you add servers, and it keeps traffic away from unhealthy ones.

              +--> Server 1
Users --> LB -+--> Server 2
              +--> Server 3

Users see one address, and servers can come and go behind it.

Algorithms#

Here is a simulation. Some requests are slow, and we compare the busiest server under two methods:

import random

random.seed(1)
requests = [random.choice([1, 1, 1, 1, 20]) for _ in range(1000)]


def simulate(pick):
    active = [[] for _ in range(4)]
    worst = 0
    for t, duration in enumerate(requests):
        active = [[end for end in s if end > t] for s in active]
        server = pick(active, t)
        active[server].append(t + duration)
        worst = max(worst, max(len(s) for s in active))
    return worst


round_robin = simulate(lambda active, t: t % 4)
least_conn = simulate(lambda active, t: min(range(4), key=lambda i: len(active[i])))
print("Most requests on one server at once")
print(f"Round robin:       {round_robin}")
print(f"Least connections: {least_conn}")

Most requests on one server at once
Round robin:       5
Least connections: 3

Least connections adapts when one server gets stuck with slow requests.

Layer 4 and layer 7#

The layers come from the OSI network model.

A layer 7 balancer can route /api to one group of servers and /images to another, and it can end HTTPS, so that app servers do not have to.

Health checks#

The load balancer regularly calls something like GET /health on each server. If a server fails several times in a row, it stops getting traffic, and it returns when it recovers.

Keep it meaningful but cheap. A check that always answers "OK" hides problems, while one that depends on every other service may remove all your servers at once.

Sticky sessions#

Sticky sessions keep a user on one server. They help if sessions are stored in server memory, but they make load uneven, and users lose their session if that server dies. It is better to make servers stateless.

Avoid a new single point of failure#

A single load balancer is itself a single point of failure. Common solutions:

• Run two, with one taking over a shared address.
• Use a managed cloud load balancer, which is already redundant.
• Use DNS to spread users across regions.

Reverse proxy and API gateway#

A reverse proxy is a server in front of your app, which may also cache and compress. A load balancer is one use of it. An API gateway adds features for APIs, such as authentication, rate limiting and routing to microservices.

Global load balancing#

For users worldwide, DNS or anycast sends each user to a nearby region, and regional load balancers then pick a server.

Common mistakes#

Practice#

import itertools

servers = {"a": 3, "b": 1}
order = [name for name, weight in servers.items() for _ in range(weight)]
picks = list(itertools.islice(itertools.cycle(order), 8))
print(picks)

['a', 'a', 'a', 'b', 'a', 'a', 'a', 'b']

Key takeaway#

Next: Caching.