About the Disk Full Incident
A full disk takes down things that appear unrelated to storage: databases stop accepting writes, logging stops, package managers fail, and processes that cannot write a temporary file crash with errors that never mention disk. Recognising the shape early saves a lot of misdirected investigation.
There are two independent ways to run out, and confusing them wastes time. Blocks exhausted means there is no space left for data. Inodes exhausted means there is space but no room for another file entry — the usual result of millions of tiny files, and it presents as a disk-full error while free space still looks fine.
This playbook covers finding what actually consumed the space, the specific case of deleted files still held open by a running process, and the container-specific causes where the pressure is on the image layer store rather than any volume you configured.
Frequently asked questions
Disk shows free space but writes still fail — why?
Almost always inode exhaustion. Every file consumes an inode, and a filesystem has a fixed number set at creation time. Millions of small files — session data, cache entries, unrotated logs — exhaust inodes long before blocks. Check inode usage explicitly; the fix is deleting files rather than freeing bytes, and permanently resolving it may require recreating the filesystem.
I deleted a large file but space was not reclaimed.
A process still has the file open. Unlinking removes the directory entry, but the blocks are only freed when the last file descriptor closes. This is routine with log files deleted while the writing process runs. Find processes holding deleted files and restart them, or truncate the file in place instead of deleting it.
What fills up disk on a Kubernetes node?
Usually the container image store rather than any volume. Unused image layers accumulate until kubelet garbage collection triggers, and its thresholds are often reached later than expected. After that, container logs that are not rotated, emptyDir volumes with no size limit, and evicted pods whose data was not cleaned up. Node disk pressure causes eviction of pods that were not responsible.
How do I find what is using the space?
Work top-down from the filesystem root, checking directory sizes at each level and following the largest, rather than scanning everything at once. On a busy production system a full recursive scan is itself expensive. Remember that a directory mounted from elsewhere reports the other filesystem's usage, which is a common source of confusion.
What is a safe disk usage threshold to alert on?
Alert on the trend, not just the level. A static threshold at 85 percent is fine as a backstop but tells you nothing about urgency — 85 percent stable for a year is not an incident, while 60 percent climbing steadily is. Predicting time-to-full from the recent growth rate gives a far more actionable page. Alert on inode usage separately.
Need this managed for you, not just automated?
We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.