Playbook

Memory Leak Identification

Systematic approach to finding and fixing memory leaks in containers.

Back to playbooks

Initial Detection

Monitor 'container_memory_working_set_bytes' for steady upward trend.

Check if memory usage does not drop after request spikes.

Identify the specific container/process with the highest growth rate.

Diagnosis Steps

Capture a heap dump from the running process.

Analyze GC logs for frequent Full GCs with decreasing reclaimed space.

Use profiling tools (e.g., pprof, clinic.js, VisualVM) to find leaking objects.

Go workloads: expose pprof endpoint and capture heap profile with 'go tool pprof http://localhost:6060/debug/pprof/heap'.

JVM workloads: add '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' to capture a dump automatically on OOM.

Common Root Causes

Unclosed resource handles (DB connections, file descriptors).

Static collections or caches that grow indefinitely.

Long-running event listeners that are never removed.

Goroutine leaks in Go: goroutines blocked on channels or context that are never closed.

Diagnostic Commands

Check Working Set Memory (K8s)

kubectl top pod <pod-name> --containers

Find process Memory map

pmap -x <pid> | sort -rk 3 | head -n 20

Node.js Heap Snapshot

node --inspect index.js
# Use Chrome DevTools to capture snapshot

Go: heap profile via pprof

go tool pprof http://localhost:6060/debug/pprof/heap

Python: memory profiler

pip install memory_profiler
mprof run python app.py
mprof plot

About the Memory Leak Identification

A memory leak in a container rarely announces itself — it shows up as periodic OOMKilled restarts that self-heal, a sawtooth memory graph, and p99 latency creep as the heap fills. Because the restart "fixes" it, leaks routinely survive in production for months until traffic growth shortens the cycle below tolerance.

This playbook covers the identification sequence: confirming it is a leak rather than legitimate growth or misconfigured limits, isolating which workload and which code path, and the language-specific tooling (heap dumps, pprof, tracemalloc) that turns a graph into a culprit.

Frequently asked questions

How do I tell a real leak from normal memory growth?

A leak grows without bound relative to load; a cache grows to a plateau. Check whether memory correlates with request count or with uptime — restart-correlated sawtooth growth that resumes immediately is a leak signature. Also rule out the imposter: JVM/Go runtimes hold freed memory from the OS by design, so container RSS overstates live heap.

What does exit code 137 / OOMKilled actually mean in Kubernetes?

The kernel's cgroup OOM killer terminated the container for exceeding its memory limit — 137 = 128 + SIGKILL(9). It is instant and unloggable from inside the process. Check kubectl describe pod for OOMKilled, and remember limits apply per-container: a sidecar can OOM independently. If limits are simply too low for legitimate usage, that is a sizing bug, not a leak.

How do I capture a heap profile from a production container?

Go: expose net/http/pprof and run go tool pprof against /debug/pprof/heap — compare two snapshots minutes apart. Java: jmap -dump or JFR recordings, analyzed in Eclipse MAT. Node: --inspect with Chrome DevTools heap snapshots. Python: tracemalloc snapshots diffed over time. In all cases, compare two points in time — a single snapshot shows usage, only the diff shows growth.

Need this managed for you, not just automated?

We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.

Explore Our Services