Kubernetes Troubleshooting Playbook: Debugging Pods, Deployments, and Cluster Issues

Kubernetes troubleshooting is most effective when you diagnose issues systematically, moving from symptoms to root cause using kubectl, logs, events, and metrics. Many real outages begin as small, preventable configuration errors. The operational goal is to shorten time-to-detect and time-to-recover without introducing new risk in production.

This playbook focuses on practical debugging of pods, deployments, and cluster-wide issues, with modern tools like kubectl debug, multi-pod log tailing, and production-grade monitoring.

Why a Kubernetes Troubleshooting Playbook Matters

Kubernetes is designed for resilience, but it is also highly configurable. Misconfigurations account for a significant share of Kubernetes stability and security incidents, with some ecosystem surveys placing the figure as high as 80%. That makes repeatable troubleshooting steps a core SRE and platform engineering skill.

A strong playbook standardizes four key activities:

• Triage: identify what is broken and the blast radius

• Evidence collection: events, logs, metrics, manifests

• Isolation: app vs platform, node vs network, config vs capacity

• Remediation: safe fixes, rollbacks, and validation

The Layered Workflow: From Pod Status to Root Cause

Start narrow and expand outward. A practical layered workflow looks like this:

1. Triage symptoms: what is failing, where, and since when?

2. Check events and logs: Kubernetes often surfaces the reason a pod is failing

3. Inspect resources: CPU, memory, disk pressure, quotas, limits

4. Validate configuration: selectors, probes, env vars, secrets, RBAC

5. Isolate components: node, CNI, DNS, ingress, registry, control plane

Step 1: Start with What Kubernetes Sees

Begin with a quick inventory:

• Pods and status: kubectl get pods -n <ns>

• Wider context: kubectl get deploy,rs,svc,ingress -n <ns>

• Node placement: kubectl get pods -o wide -n <ns>

Look for common failure states such as CrashLoopBackOff, ImagePullBackOff, Pending, and OOMKilled.

Step 2: Use Describe to Read Events Like a Timeline

kubectl describe is the fastest way to surface scheduling decisions and event errors:

• Pod details and events: kubectl describe pod <pod> -n <ns>

• Deployment details: kubectl describe deploy <deploy> -n <ns>

Events commonly reveal probe failures, image pull errors, missing secrets or configmaps, insufficient resources, and node pressure conditions.

Step 3: Inspect Logs (Single Pod or Many)

Logs should be checked before cluster-level debugging because many failures are application-level: bad config, missing dependencies, migration issues, or an incorrect startup command.

• Container logs: kubectl logs <pod> -n <ns> -c <container>

• Previous crash logs: kubectl logs <pod> -n <ns> -c <container> --previous

• Multi-pod tailing: use tools like stern to tail logs across replicas

For production environments, centralized logging (for example, Fluent Bit or OpenTelemetry collectors shipping to a log store) significantly improves incident response by enabling correlation across pods, nodes, and services.

Step 4: Check Resource Pressure Quickly

Resource exhaustion is a leading trigger for pod instability. Check current consumption with:

• Pod usage: kubectl top pod -n <ns>

• Node usage: kubectl top node

Use these signals to determine whether you are dealing with a noisy neighbor issue, missing requests or limits, insufficient cluster capacity, or a memory leak.

Step 5: Debug Running Containers Safely

Modern Kubernetes clusters increasingly use minimal or distroless images, which makes traditional shell-based debugging difficult. kubectl debug supports ephemeral containers so you can attach tooling without rebuilding images:

• Ephemeral debug container: kubectl debug -n <ns> -it <pod> --image=busybox

• Exec (when available): kubectl exec -n <ns> -it <pod> -- sh

For node-level inspection, container runtime tooling like crictl can help operators inspect containers when kubelet-level symptoms suggest runtime problems.

Debugging Common Pod Failures

The patterns below cover a large percentage of real incidents and each maps to a clear set of checks.

CrashLoopBackOff

What it means: the container starts and crashes repeatedly, triggering restarts.

Primary causes: bad configuration, missing dependencies, failing migrations, incorrect command or args, broken probes that kill a slow-starting application.

Playbook:

1. Check events via kubectl describe pod and read the most recent entries.

2. Read current and previous logs using kubectl logs and the --previous flag.

3. Validate config inputs: env vars, secrets, configmaps, and mounted files.

4. Confirm probes: review liveness, readiness, and startup timing and paths.

5. Check resources: verify CPU and memory requests and limits are realistic.

Typical fix: correct the configuration (for example, a missing secret key), adjust probes for startup time, and right-size requests and limits.

OOMKilled

What it means: the container exceeded its memory limit and was terminated by the kernel.

Playbook:

1. Confirm termination reason in pod status and events via kubectl describe pod.

2. Check memory usage trends with kubectl top pod and dashboards (Prometheus and Grafana are widely used).

3. Compare observed usage with configured limits in the pod spec.

4. Decide whether to increase the memory limit, optimize application memory usage, or both.

Operational guidance: treat repeated OOMKills as either a sizing problem or a memory leak. Consider autoscaling strategies such as Vertical Pod Autoscaler where appropriate, and pair them with alerting to detect regressions quickly.

ImagePullBackOff

What it means: the node cannot pull the container image, blocking the rollout.

Primary causes: registry outage, DNS or network policy restrictions, firewall or proxy issues, incorrect image name or tag, missing imagePullSecrets.

Playbook:

• Check events for the exact error message in kubectl describe pod.

• Verify the image reference and tag exist in the registry.

• Validate registry access from nodes: network route, DNS, and egress policy.

• Confirm authentication: correct imagePullSecrets and service account binding.

Troubleshooting Deployments and Rollouts

When a deployment is unhealthy, focus on rollout mechanics and selector logic.

Deployment Checks

• Rollout status: kubectl rollout status deploy/<name> -n <ns>

• History: kubectl rollout history deploy/<name> -n <ns>

• ReplicaSets: kubectl get rs -n <ns> to compare old vs new

Common deployment mistakes include:

• Selector mismatch between Deployment and Pod labels, resulting in orphaned pods

• Readiness probe failures that block traffic and stall rollouts

• Insufficient resources causing pods to remain Pending during a rollout

Safe remediation: if a new version is failing, use kubectl rollout undo and then debug the failing ReplicaSet using the pod-level steps above.

Cluster-Wide Issues: Nodes, Networking, and Control Plane Signals

If many workloads fail simultaneously, broaden the scope to cluster health.

Node Health and Scheduling Failures

• Node readiness: kubectl get nodes and kubectl describe node

• Common blockers: disk pressure, memory pressure, PID pressure, taints, or exhausted allocatable resources

• Pending pods: events often show Insufficient cpu or Insufficient memory

When capacity is the root cause, remediation options include rebalancing workloads, adding nodes, pruning unused resources, and applying resource quotas to prevent noisy neighbor incidents.

Network and DNS Disruptions

Networking issues are common in multi-cloud, hybrid, and edge deployments. If services cannot reach each other, validate the following:

• CoreDNS health and resolution inside pods

• CNI plugin status and node-to-node connectivity

• NetworkPolicy rules that block egress or service-to-service traffic

eBPF-based observability has become a standard approach for deep network visibility in production, helping teams trace packet drops, latency, and policy enforcement without relying solely on application logs.

Preventing Repeat Incidents: Turning Fixes into Guardrails

A Kubernetes troubleshooting playbook should evolve into a prevention framework. After resolving an incident, reduce recurrence by implementing:

• Centralized logs and metrics: consistent collection and correlation across the cluster

• Prometheus alerting: alerts for CrashLoopBackOff spikes, OOMKills, node pressure, and high error rates

• YAML validation: policy and schema checks in CI pipelines (tools like kubeval and Conftest are commonly used)

• Resilience primitives: PodDisruptionBudgets, priority classes, and realistic requests and limits

• Runbooks and automation: scripted kubectl workflows and CI/CD-integrated diagnostics (GitOps tooling such as Argo CD pairs well here)

Teams building deeper platform competence can benefit from structured training that aligns Kubernetes operations with adjacent skills like cloud security and DevOps practices. Blockchain Council offers programs including Certified Kubernetes Administrator (CKA) training, Certified DevOps Professional, and Certified Cloud Security Professional, which cover troubleshooting, observability, and secure configuration at a professional level.

Conclusion

An effective Kubernetes troubleshooting playbook follows a disciplined workflow: start with pod state, read events, analyze logs, check resource pressure, then expand to deployment and cluster layers only when the evidence points there. This method reduces guesswork, speeds recovery, and makes post-incident prevention actionable.

As Kubernetes operations mature, expect broader adoption of proactive anomaly monitoring, eBPF-driven network insight, and automated runbooks. The fundamentals remain unchanged: collect the right signals, validate configuration, and implement guardrails so the same issue does not recur.