1. Confirm the context and find the pod
kubectl config current-context
kubectl get pods -n <ns> -o wide | grep -v Running
Make sure you're in the right cluster before you do anything else. The RESTARTS column tells you how long it's been looping.
2. Read the logs of the crashed container
kubectl logs <pod> -n <ns> -c <container> --previous
This is the most commonly missed step. Without --previous you see the logs of the current attempt, which is often empty because it just started. The previous instance holds the stack trace or the "missing env var" message.
3. Decode the exit code
kubectl describe pod <pod> -n <ns> # look at "Last State: Terminated", Reason and Exit Code
| Exit code | Meaning | Look at |
|---|---|---|
| 0 | The process finished successfully | The container's command runs once and exits. A Deployment expects a long-running process. Use a Job instead, or fix the command. |
| 1 (or other app codes) | Application error | The logs from step 2: config, missing secret, failed DB connection, failed migration |
| 126 / 127 | Command not executable / not found | command/args in the manifest, the entrypoint in the image, the shell form vs exec form |
| 137 | SIGKILL | Reason: OOMKilled means the memory limit is too low or there's a leak. Without OOMKilled, it's usually the liveness probe killing the container. |
| 139 | Segfault | Native libraries, an architecture mismatch (an arm64 image on amd64 nodes) |
| 143 | SIGTERM | Something asked it to stop: probes, or the node draining |
4. Check the usual suspects
- Missing config. If a referenced ConfigMap or Secret doesn't exist you'll see
CreateContainerConfigErrorinstead, but a key that's present yet wrong or empty makes the app crash at startup. Check which keys exist without printing the values:kubectl get secret <s> -n <ns> -o jsonpath='{.data}' | jq 'keys' - Dependency unreachable at startup. The app exits if the DB or cache isn't ready yet. The fix is retry with backoff in the app, not an init-container sleep.
- Liveness probe too aggressive. Slow-starting apps (the JVM, big Node bundles) get killed before they're ready. Add a
startupProbe, or raiseinitialDelaySeconds/failureThreshold. Look forLiveness probe failedin the events. - OOMKilled. Compare
kubectl top podwithresources.limits.memory. Runtimes that size their heap from host memory (older JVMs, Node's default heap) need explicit flags such as-XX:MaxRAMPercentageor--max-old-space-size. - Read-only filesystem or wrong user. A
securityContextwithreadOnlyRootFilesystemorrunAsNonRootbreaks apps that write to their working directory. Look for "permission denied" in the logs.
5. Get a shell when the logs aren't enough
You can't exec into a container that keeps dying. Override the command in a copy of the pod instead (this creates a new pod, so don't do it in prod without thinking):
kubectl debug <pod> -n <ns> -it --copy-to=<pod>-debug --container=<container> -- sh
Then run the original entrypoint by hand and watch it fail.
6. Fix, verify, prevent
kubectl rollout status deploy/<name> -n <ns>
kubectl get pods -n <ns> -w
Prevention: set resource requests and limits from real usage, add a startupProbe for slow starters, and validate that referenced Secret and ConfigMap keys exist in CI before deploying.
Let Claude Code do this for you. The free sre-skills plugin includes a k8s-debug skill that works through this checklist using read-only commands and quotes the evidence it finds:
/plugin marketplace add polytheseus/sre-skills
/plugin install sre@sre-skills