Design workloads that survive

Kubernetes can replace a Pod only if the application is designed to be replaceable.

The safe workload recipe

• Keep web and worker compute stateless; externalize durable state.
• Use immutable image digests, not mutable latest tags.
• Run as non-root, drop capabilities, use seccomp, and prefer read-only filesystems.
• Set measured requests and intentional limits.
• Add startup, readiness, and careful liveness probes.
• Handle termination signals and allow enough graceful shutdown time.
• Use rolling, canary, or blue/green delivery with rollback evidence.
• Spread replicas across zones and nodes.
• Use a PodDisruptionBudget that protects service without blocking every drain.

Configuration and secrets

Keep ordinary environment configuration in ConfigMaps or external configuration. Store sensitive values in Secrets Manager or Parameter Store and retrieve them through a controlled integration. Kubernetes Secret values are base64-encoded, not magically encrypted by that encoding.

Back-pressure

Queues such as SQS separate bursty intake from slower processing. Workers must be idempotent because retries and duplicate delivery happen. Bound retries, send poison messages to a dead-letter queue, and protect downstream systems with timeouts and concurrency limits.

Feynman check

If deleting any one Pod causes lost orders, forced user logout, or manual repair, the application—not EKS—is holding essential state in the wrong place.