Availability and Reliability in System Design

Servers crash, disks die, networks fail, and people deploy bad code. A well-designed system expects failure, and keeps working.

• Availability: is the system up when you need it?
• Reliability: does it work correctly over time?

A system can be available and unreliable, for example if it responds, but with wrong data.

The nines#

Availability is the percentage of time a system is working:

minutes_per_year = 365 * 24 * 60
for nines in ("99", "99.9", "99.99", "99.999"):
    down = minutes_per_year * (100 - float(nines)) / 100
    print(f"{nines + '%':>8}  {down:>9,.1f} minutes of downtime a year")

     99%    5,256.0 minutes of downtime a year
   99.9%      525.6 minutes of downtime a year
  99.99%       52.6 minutes of downtime a year
 99.999%        5.3 minutes of downtime a year

Each extra nine is much harder and more expensive. Choose a target that suits the product: a payment system needs more than an internal report.

Parts in a chain, parts side by side#

If a request needs every part, the availabilities multiply, and the total falls. If you have spare copies, it rises:

parts = [0.999, 0.999, 0.999]
serial = 1
for p in parts:
    serial *= p
print(f"Three parts in a chain: {serial:.4%}")

one = 0.99
print(f"One server:  {one:.2%}")
print(f"Two servers: {1 - (1 - one) ** 2:.2%}")

Three parts in a chain: 99.7003%
One server:  99.00%
Two servers: 99.99%

This is the maths behind redundancy. It assumes failures are independent, and in reality a shared problem such as a power cut or a bad deployment can take down both copies.

Single points of failure#

A single point of failure (SPOF) is one part whose failure stops everything. Find each one, and add a copy.

Bad:  Users --> One load balancer --> Servers --> One database

Good: Users --> Two load balancers --> Servers --> Primary + Standby
                                                  (in different zones)

Cloud availability zones are separate data centres in a region, so copies placed in different zones survive the loss of one building.

Failover#

Failover switches to a standby when the primary fails.

Test failover regularly. Untested backups and standbys often fail when needed.

Handling failures between services#

Timeouts. Never wait forever. A slow service is often worse than a dead one, because waiting requests pile up.

Retries with backoff. Retry brief failures, waiting longer each time, with some randomness (jitter), so that thousands of clients do not retry at the same moment:

import random

random.seed(7)
for attempt in range(1, 6):
    delay = min(30, 0.5 * 2 ** attempt)
    wait = delay / 2 + random.uniform(0, delay / 2)
    print(f"Attempt {attempt}: wait {wait:.2f}s")

Attempt 1: wait 0.66s
Attempt 2: wait 1.15s
Attempt 3: wait 3.30s
Attempt 4: wait 4.29s
Attempt 5: wait 12.29s

Retrying is safe only for idempotent operations, otherwise a retry could charge a card twice.

Circuit breakers. If a service keeps failing, stop calling it for a while, and fail fast:

class CircuitBreaker:
    def __init__(self, limit=3):
        self.failures = 0
        self.limit = limit

    def call(self, ok):
        if self.failures >= self.limit:
            return "blocked (fail fast)"
        if ok:
            self.failures = 0
            return "ok"
        self.failures += 1
        return "failed"


breaker = CircuitBreaker()
for ok in [True, False, False, False, True]:
    print(breaker.call(ok))

ok
failed
failed
failed
blocked (fail fast)

A real circuit breaker also lets a few trial requests through after a pause, to see whether the service has recovered.

Graceful degradation. If recommendations are down, still show the shop. A reduced service beats no service.

SLI, SLO and SLA#

Disaster recovery#

• RPO (recovery point objective): how much data can you afford to lose? This decides how often you back up.
• RTO (recovery time objective): how long can you be down? This decides how fast you must restore.

Keep backups in another region, and practise restoring them.

Common mistakes#

Practice#

total = 0.9999 * 0.999 * 0.9995
hours = (1 - total) * 365 * 24
print(f"{total:.3%}")
print(f"{hours:.1f} hours a year")

99.840%
14.0 hours a year

Key takeaway#

Next: Consistency and the CAP Theorem.