---
title: "Design workloads that survive"
chapter: "06"
---

# Design workloads that survive

Kubernetes can replace a Pod only if the application is designed to be
replaceable.

## The safe workload recipe

- Keep web and worker compute stateless; externalize durable state.
- Use immutable image digests, not mutable `latest` tags.
- Run as non-root, drop capabilities, use seccomp, and prefer read-only filesystems.
- Set measured requests and intentional limits.
- Add startup, readiness, and careful liveness probes.
- Handle termination signals and allow enough graceful shutdown time.
- Use rolling, canary, or blue/green delivery with rollback evidence.
- Spread replicas across zones and nodes.
- Use a PodDisruptionBudget that protects service without blocking every drain.

## Configuration and secrets

Keep ordinary environment configuration in ConfigMaps or external configuration.
Store sensitive values in Secrets Manager or Parameter Store and retrieve them
through a controlled integration. Kubernetes Secret values are base64-encoded,
not magically encrypted by that encoding.

## Back-pressure

Queues such as SQS separate bursty intake from slower processing. Workers must
be idempotent because retries and duplicate delivery happen. Bound retries,
send poison messages to a dead-letter queue, and protect downstream systems
with timeouts and concurrency limits.

## Feynman check

If deleting any one Pod causes lost orders, forced user logout, or manual repair,
the application—not EKS—is holding essential state in the wrong place.