Observability and SRE

Observability should answer: Are users healthy? What changed? Where is the bottleneck? Who owns the response?

Signals

Use metrics, logs, traces, events, and profiles where useful. Start with request rate, errors, duration, and saturation. CloudWatch Container Insights can collect cluster/container signals. EKS control-plane logging captures API and audit evidence. ADOT can export OpenTelemetry data. Amazon Managed Service for Prometheus and Amazon Managed Grafana support Prometheus-style monitoring and dashboards.

SLOs

An SLI is a measurement such as successful checkout requests. An SLO is the target, such as 99.95% over 30 days. The error budget is the permitted bad portion. Alert on fast and slow error-budget burn, not every harmless spike.

Cost and signal quality

Unbounded log volume, metric cardinality, and trace sampling can be expensive. Define retention, redaction, ownership, sampling, and query needs. Never log credentials or unnecessary personal data.

Runbooks

An alert needs an owner, impact, dashboard, first checks, safe mitigation, escalation, and verification. Practice incident response and record learning without blame.

Feynman check

A dashboard is a car dashboard: it shows whether the journey is safe. An alert is the warning light that asks a person to act. A pile of logs is only raw material.