The problem
A full disk breaks whatever needs to write next
At 03:04 the root file system of web1 is full and the shop returns 500 errors. Look before acting.
It is four minutes past three in the morning. The root file system on web1, 40 gigabytes, is completely full. The shop returns server errors, because the app cannot save its session files. The alert wakes the engineer on call. This is the moment people delete the wrong thing, so the first job is to look.
Triage
Narrow it down: how full, then where, one level at a time
df says full; du -x narrows it to /var, then /var/log/app, then one 26 GB file.
df confirms the root file system is full. du, kept on this one file system, shows that /var holds 31 gigabytes. One level deeper, the app's log folder holds 27, and almost all of it is one file: app dot log, 26 gigabytes. Two minutes of reading, and "the disk is full" has become one named file.
The first fix
Deleting the file removed its name, not its space
rm app.log, and df still says 100%. du counts 13 GB: the space has no name.
The tired engineer deletes the log. df still says one hundred percent. du now counts only 13 gigabytes, because it adds up files it can find by name. The space is used by something with no name. And the log that showed what went wrong is gone.
The cause
The app still holds the deleted file open
lsof +L1 shows the shop app, PID 812, still writing to the deleted log. Debug logging at 1.1 GB an hour filled it.
lsof plus L1 lists files that are open but have no name left. The shop app still writes to app dot log through descriptor 4. rm removed the name, but the kernel keeps the blocks until the last descriptor closes. And why was it so big? Debug logging was switched on for yesterday's deploy and never switched off: about 1.1 gigabytes an hour, for 24 hours.
The fix
Empty the file the app still holds, then fix what made it grow
Truncate /proc/812/fd/4, set the log level back to info, restart. The disk drops to 35% and the site answers 200.
The file has no name, but the process's descriptor still points at it, under slash proc. Truncating it there frees the space at once. Then we turn debug logging off and restart the app, which also closes the old descriptor. The disk drops to 35 percent, and the site answers again.
Prevention
Rotate logs, cap the journal, and alert while there is time
logrotate daily or at 1 GB, keep 7; journald capped at 1G; alert at 80% gives 7 hours.
A logrotate rule rotates the app's log every day, or sooner at one gigabyte, keeps 7 compressed copies, and empties the live file in place so the app is never caught out. The journal is capped at one gigabyte. And the alert moves from one hundred percent to 80. At the worst rate we saw, that is 7 hours of warning, in daylight, instead of none.
The runbook
A full disk has a six-step runbook
df, du, lsof +L1, free it safely, restart and verify, then stop it recurring. Each step links to its lesson.
Here is the night as a runbook. How full: df. Where: du, one level at a time. Held open: lsof plus L1. Free it safely: empty the file, never delete a live log. Restart and verify. Then stop it coming back. Step three is the one people skip, and the one that would have saved the log. Each step opens the lesson that teaches its command.
A real case
At GitLab in 2017, the wrong host at night, and backups nobody had restored
GitLab deleted the primary database instead of the replica, and found most backups had failed silently.
On the thirty-first of January 2017, late at night, a GitLab engineer was rebuilding a database replica that had fallen behind. The delete meant for the replica, db2, was run on the primary, db1. Then the backups. The nightly dump had been failing because it ran the wrong version, and its failure emails were being rejected. Disk snapshots were not turned on for the databases. The copy that saved them was a snapshot made six hours earlier for the staging site. About six hours of data were lost, and the site was down for about eighteen hours. GitLab now takes snapshots hourly, tests restores automatically, and has an owner for data. The habits are ours too: check where you are before you delete, and only trust a backup you have restored.
Lesson complete
You finished “The disk filled up at 3 a.m.”
Nicely done.