Fix Kubernetes OOMKilled: Pod Killed Due to Out of Memory

Fix Kubernetes OOMKilled: Pod Killed Due to Out of Memory

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

OOMKilled (Out Of Memory Killed) means the Linux kernel's OOM killer terminated the container because it exceeded its memory limits value in the pod spec. Exit code 137 = SIGKILL (128 + 9). This is the kernel acting directly — Kubernetes just reports it.

If the container keeps being OOMKilled and restarted, it will enter CrashLoopBackOff.

Confirm it's OOMKilled

kubectl describe pod <pod-name>

Look for:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Also check recent events:

kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by=.metadata.creationTimestamp

Find how much memory the pod is actually using

Before changing limits, measure actual usage:

# Current memory usage (requires metrics-server)
kubectl top pods
kubectl top pod <pod-name>

# All pods sorted by memory
kubectl top pods --sort-by=memory -A

If metrics-server isn't installed:

# Get memory from inside the container
kubectl exec <pod-name> -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes     # cgroup v1
kubectl exec <pod-name> -- cat /sys/fs/cgroup/memory.current                   # cgroup v2

Cause 1: Memory limit is too low

The most common cause — the app needs more memory than you've allocated.

# Before — limit is too tight
resources:
  requests:
    memory: "128Mi"
  limits:
    memory: "128Mi"

Fix: Raise the limit to give the app headroom. A good starting point is 1.5–2× the average usage observed under normal load:

resources:
  requests:
    memory: "256Mi"    # What the pod needs at baseline
  limits:
    memory: "512Mi"    # Headroom for spikes

Important: requests affects scheduling — the node must have this much free memory to place the pod. limits is the enforcement ceiling. Setting requests == limits (Guaranteed QoS) means the pod is killed at exactly the limit; setting limits > requests (Burstable QoS) gives headroom for spikes.

Cause 2: Memory leak

If memory usage climbs steadily over time until the limit is hit, you have a leak — not a limit problem. Raising the limit just delays the next OOMKill.

How to identify a leak:

# Watch memory usage over time
watch -n 30 kubectl top pod <pod-name>

If memory grows continuously (doesn't plateau), it's a leak.

Fix: Profile the application:

• Node.js: --expose-gc + heap snapshots via v8.writeHeapSnapshot()
• Java/JVM: heap dump with jmap -dump:format=b,file=heap.bin <pid>
• Go: pprof endpoint (net/http/pprof)
• Python: tracemalloc or memory-profiler

As a short-term mitigation, set a memory limit and let Kubernetes restart the pod on schedule — but this is treating the symptom, not the cause.

Cause 3: Traffic spike or batch job peak

Some applications have predictable memory spikes — handling a large request, running a nightly batch, loading a large dataset.

Fix: Size the limit for peak, not average:

resources:
  requests:
    memory: "256Mi"    # Baseline scheduling
  limits:
    memory: "1Gi"      # Peak allowance

For batch jobs that need a lot of memory briefly, consider running them as Kubernetes Jobs with a higher memory limit isolated from the main service.

Cause 4: Multiple containers in a pod sharing node memory

If you have sidecars (log shippers, proxies, init containers), their memory counts against the pod's total. OOMKill applies per-container, not per-pod — but if a sidecar has no limit set, it can consume memory that starves the main container.

# See all containers in a pod and their limits
kubectl get pod <pod-name> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits.memory}{"\n"}{end}'

Fix: Set explicit memory limits on every container, including sidecars.

Verify the fix

After raising the limit and redeploying:

1# Watch for OOMKill events
2kubectl get events -w | grep OOMKilled
3
4# Check restart count has stopped climbing
5kubectl get pod <pod-name>
6
7# Monitor memory usage under load
8kubectl top pod <pod-name>

If restarts stop and memory usage stabilises below the limit, the fix worked.

Prevent future OOMKills

1. Always set both requests and limits — pods without limits can consume all node memory and destabilise other workloads.
2. Set up alerts — Prometheus alert on container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85 (85% of limit) gives you warning before the kill.
3. Use VPA — the Vertical Pod Autoscaler observes actual usage and recommends (or automatically applies) right-sized requests and limits.

See also

• Fix: Kubernetes CrashLoopBackOff — OOMKilled often leads to this
• Kubernetes Resource Requests and Limits — understanding QoS classes and scheduling

OOMKilled surfaces to the runtime as exit code 137 — see what exit code 137 actually means for that side of the same event.

Frequently Asked Questions

How do I confirm it was really OOMKilled?

Check the pod's last terminated state, which names the reason explicitly alongside exit code 137. Do not infer it from the exit code alone — 137 means SIGKILL, and while memory is the usual cause in Kubernetes, something else may have sent the signal.

Is raising the memory limit the right fix?

Only after you know why usage grew. If the workload genuinely needs more, raise it. If usage climbs steadily and never plateaus, that is a leak and a higher limit only delays the kill while wasting more memory. Watch usage over a full traffic cycle before deciding.

Why did my pod get killed when the node had free memory?

Because the limit is per container, not per node. Exceeding your own limit kills you regardless of what else is available. The other case is node-level pressure evicting pods, which shows as an eviction rather than OOMKilled — different symptom, different fix.

How should I size memory requests and limits?

Set the request to steady-state usage so scheduling is accurate, and the limit with enough headroom for legitimate peaks. Setting the limit far above the request lets a pod balloon on a node that looked full, which is how you get OOM kills on a node with apparently spare capacity.

Official References

• Managing resources for containers — requests, limits and how they map to cgroup settings
• Pod Quality of Service classes — how Guaranteed, Burstable and BestEffort affect eviction order