The problem

A full disk breaks whatever needs to write next

web1nginx + the shop appRoot file system, 40 GB100% usedThe siteHTTP/1.1 500 Internal Server Errorthe app cannot write its session filesAlert, 03:04web1: / is 100% fullcheckout failingA full disk stops every program that needs to write, so the symptom shows up somewhere else first.At 03:04 the on-call engineer is woken by an alert: the root file system of web1 is full, and the shop returns 500 errors because the appcannot save sessions. It is the middle of the night and customers are failing to pay. That is exactly when people delete the wrong thing,so the first job is to look, not to act.

At 03:04 the root file system of web1 is full and the shop returns 500 errors. Look before acting.

Triage

Narrow it down: how full, then where, one level at a time

$ df -h /Filesystem Size Used Avail Use% Mounted on/dev/vda1 40G 40G 0 100% /1$ sudo du -xh --max-depth=1 / | sort -h | tail -231G /var239G /$ sudo du -xh --max-depth=1 /var/log | sort -h | tail -127G /var/log/app3$ ls -lh /var/log/app-rw-r--r-- 1 shop shop 26G ... app.log1How full?df: the whole file system2Where?du, one level at a time3Which file?one log: 26 of 27 GBNarrow down before touching anything: df for how full, du for where, one level at a time.df says the root file system is full. du, kept on one file system with -x, shows /var holding 31 GB, and inside it /var/log/app holds 27 GB.One file, app.log, is 26 GB. Two minutes of reading has turned "the disk is full" into one named file. The next question is why it is sobig, but at 3 a.m. the first instinct is to make room.

df says full; du -x narrows it to /var, then /var/log/app, then one 26 GB file.

The first fix

Deleting the file removed its name, not its space

$ sudo rm /var/log/app/app.log$ df -h /Filesystem Size Used Avail Use% Mounted on/dev/vda1 40G 40G 0 100% /1$ sudo du -xsh /13G /2The file is gone from the folder.The space is not back.du now counts only 13 GB,yet df still says 40 GB are used.Deleting a file removes its name. The space comes back only when no process has it open.The engineer deletes app.log. df still says 100%. du, which adds up the files it can find by name, now counts only 13 GB: the26 GB are used by something with no name. The site is still down, and the log that showed what went wrong has been thrown away.This is the most common mistake with a full disk, and it is why the lesson says to truncate a live log, not delete it.

rm app.log, and df still says 100%. du counts 13 GB: the space has no name.

The cause

The app still holds the deleted file open

$ sudo lsof +L1COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAMEshop 812 shop 4w REG 253,1 27917287424 0 131074 /var/log/app/app.log (deleted)1The nameremoved by rmOpen descriptor 4held by shop, PID 81226 GB of blocksstill in useWhy so big?debug logging left onafter the 03:00 yesterday deploy:1.1 GB an hour × 24 hThe shop app still has the deleted file open, so its 26 GB stay in use until it lets go.lsof +L1 lists open files with no name left (NLINK 0). The shop app, PID 812, still writes to app.log through descriptor 4. The kernel freesthe blocks only when that descriptor closes. And the cause of the growth: debug logging was switched on for yesterday's 03:00 deployand never switched off, at about 1.1 GB an hour. 24 hours later, the disk was full.

lsof +L1 shows the shop app, PID 812, still writing to the deleted log. Debug logging at 1.1 GB an hour filled it.

The fix

Empty the file the app still holds, then fix what made it grow

$ sudo truncate -s 0 /proc/812/fd/41$ sudo sed -i 's/LOG_LEVEL=debug/LOG_LEVEL=info/' /etc/shop.env$ sudo systemctl restart shop2$ df -h /Filesystem Size Used Avail Use% Mounted on/dev/vda1 40G 14G 26G 35% /3$ curl -sI https://shop.example.com | head -1HTTP/2 200Before40 GB, 100%After14 GB, 35%Empty the file the process still holds, fix what made it grow, then check the disk and the site.The file has no name any more, but /proc/812/fd/4 still points at it, so truncate empties it and the space comes back at once. Thenturn debug logging off and restart the app, which also closes the old descriptor. Usage drops from 100% to 35% and the site answers 200.Had the log not been deleted, the same fix would have been : > app.log, keeping the file and its history.

Truncate /proc/812/fd/4, set the log level back to info, restart. The disk drops to 35% and the site answers 200.

Prevention

Rotate logs, cap the journal, and alert while there is time

/etc/logrotate.d/shop/var/log/app/*.log { daily rotate 7 maxsize 1G compress copytruncate}/etc/systemd/journald.conf[Journal]SystemMaxUse=1GAlert earlier80%8 GB left at 1.1 GB an hour = 7 hours to act, in daylightFix the night, then fix the system: rotate logs, cap the journal, and alert while there is still time.logrotate now rotates the app log every day or at 1 GB, keeps 7 compressed copies and empties the live file in place (copytruncate), so aprocess that keeps the file open is never caught out. journald is capped at 1G. And the alert moves from 100% to 80%: at the worst rateseen, that leaves 7 hours to act instead of none. Last, the deploy checklist gets a line: switch debug logging off.

logrotate daily or at 1 GB, keep 7; journald capped at 1G; alert at 80% gives 7 hours.

The runbook

A full disk has a six-step runbook

1 How full?df -h; df -iLesson 10: file systems2 Where?du -xh --max-depth=1 | sort -hLesson 4: pipes and sort3 Held open?lsof +L1Lesson 7: processes4 Free it safely: > file, or /proc/PID/fd/NLesson 2: files and inodes5 Restart, verifysystemctl restart; journalctl -uLesson 8: services6 Stop it recurringlogrotate, journald, alertLesson 14: troubleshootingEach step opens the lesson that teaches its command.A full disk has a six-step runbook. The step people skip, step 3, is the one that would have saved the log.Write the runbook down before the night you need it. At 3 a.m. nobody reasons well, so the steps should not need reasoning: run thecommand, read the line it points at, move to the next step. The same shape works for memory, a dead service or a full inode table.

df, du, lsof +L1, free it safely, restart and verify, then stop it recurring. Each step links to its lesson.

A real case

At GitLab in 2017, the wrong host at night, and backups nobody had restored

[db2]$ sudo rm -rf …/postgresql/data ← meant[db1]$ sudo rm -rf …/postgresql/data ← ranabout 23:00 UTC, 31 January 2017: stopped after about 300 GB were gonepg_dump to S3ran the wrong version, failedFailure emailsrejected, nobody saw themDisk snapshotsnot enabled on database hostsLVM snapshotmade for staging,6 hours old: usedLostabout 6 hours of data: at least 5,000 projects, 5,000 comments and about 700 new users. Down about 18 hours.Changedhourly snapshots, snapshots for databases, automated restore tests, an owner for data durability.At GitLab in 2017, one command on the wrong host at night, and backups nobody had restored.Rebuilding a lagging database replica late at night, a GitLab engineer ran the delete on the primary, db1, instead of the replica, db2.Of the backups, pg_dump had been failing silently, disk snapshots were off for databases, and recovery came from a 6-hour-old copy madefor staging. Same habits as our night: check where you are before deleting, and only trust a backup you have restored.

GitLab deleted the primary database instead of the replica, and found most backups had failed silently.

Lesson complete

You finished “The disk filled up at 3 a.m.”

Nicely done.

Open in a new tab