---
title: "Observability and SRE"
chapter: "13"
---

# Observability and SRE

Observability should answer: Are users healthy? What changed? Where is the
bottleneck? Who owns the response?

## Signals

Use metrics, logs, traces, events, and profiles where useful. Start with request
rate, errors, duration, and saturation. CloudWatch Container Insights can
collect cluster/container signals. EKS control-plane logging captures API and
audit evidence. ADOT can export OpenTelemetry data. Amazon Managed Service for
Prometheus and Amazon Managed Grafana support Prometheus-style monitoring and
dashboards.

## SLOs

An SLI is a measurement such as successful checkout requests. An SLO is the
target, such as 99.95% over 30 days. The error budget is the permitted bad
portion. Alert on fast and slow error-budget burn, not every harmless spike.

## Cost and signal quality

Unbounded log volume, metric cardinality, and trace sampling can be expensive.
Define retention, redaction, ownership, sampling, and query needs. Never log
credentials or unnecessary personal data.

## Runbooks

An alert needs an owner, impact, dashboard, first checks, safe mitigation,
escalation, and verification. Practice incident response and record learning
without blame.

## Feynman check

A dashboard is a car dashboard: it shows whether the journey is safe. An alert
is the warning light that asks a person to act. A pile of logs is only raw
material.