Monitoring and Observability: Logs, Metrics and Traces

You cannot fix what you cannot see. In a large system, a user complaint should not be how you find out about a problem.

Three pillars#

A typical path: an alert fires on a metric, a trace shows which service is slow, and its logs show why.

The four golden signals#

Measure these for each service:

Why averages mislead#

Averages hide slow requests. Use percentiles: p99 is the time that 99% of requests beat.

import random
import statistics

random.seed(5)
times = [random.gauss(100, 10) for _ in range(990)] + [random.gauss(2000, 200) for _ in range(10)]
times.sort()

print(f"Average: {statistics.mean(times):.0f} ms")
print(f"Median:  {times[500]:.0f} ms")
print(f"p95:     {times[950]:.0f} ms")
print(f"p99:     {times[990]:.0f} ms")
print(f"Max:     {times[-1]:.0f} ms")

Average: 119 ms
Median:  100 ms
p95:     117 ms
p99:     1744 ms
Max:     2105 ms

One request in a hundred takes about two seconds, and neither the average nor the median shows it. If a page makes many calls, most users hit a slow one:

for calls in (1, 10, 50):
    chance = 1 - 0.99 ** calls
    print(f"A page with {calls:>2} calls: {chance:.0%} of views hit a p99-slow call")

A page with  1 calls: 1% of views hit a p99-slow call
A page with 10 calls: 10% of views hit a p99-slow call
A page with 50 calls: 39% of views hit a p99-slow call

Structured logs#

Logs that machines can search are far more useful:

import json

print(json.dumps({"level": "error", "msg": "payment failed", "order": 1042, "trace_id": "a1b2", "ms": 812}))

{"level": "error", "msg": "payment failed", "order": 1042, "trace_id": "a1b2", "ms": 812}

Never log passwords, tokens or card numbers.

Distributed tracing#

In microservices, one request crosses many services. A trace ID travels with it, and each service records a timed span:

GET /checkout                      820 ms
 |- auth service       12 ms
 |- cart service       40 ms
 |- payment service   740 ms    <-- the problem
     |- bank API      700 ms

OpenTelemetry is the standard way to add this.

Alerts#

Too many alerts teach people to ignore them. Good alerts are:

• Actionable: someone needs to do something.
• Based on user impact: "error rate over 1%", rather than "CPU at 80%".
• Urgent only when it cannot wait until morning.

Teams with SLOs often use an error budget. With a 99.9% target, 0.1% is the budget for failures:

slo = 0.999
monthly_requests = 50_000_000
budget = monthly_requests * (1 - slo)
print(f"Error budget: {budget:,.0f} failed requests a month")
print(f"Used so far (12,000 failures): {12_000 / budget:.0%}")

Error budget: 50,000 failed requests a month
Used so far (12,000 failures): 24%

If the budget runs out, the team slows releases, and works on reliability.

After an incident#

Write a blameless postmortem: what happened, why, how it was found, and what will prevent it. Focus on systems, not people.

Common mistakes#

Practice#

times = sorted([12, 15, 11, 200, 14, 13, 16, 18, 12, 500])
print(times)
print("p90:", times[int(len(times) * 0.9) - 1])

[11, 12, 12, 13, 14, 15, 16, 18, 200, 500]
p90: 200

Key takeaway#

Next: Security Basics.