Triage
Seven symptoms, the first command for each, and what it shows
A triage map: from each symptom, one first command, and what its output says about the cause.
When something is wrong, start from the symptom. Slow: check the load and whether the CPU is waiting on disk. Programs killed: check available memory and the kernel’s out-of-memory messages. Writes failing: check space and inodes. A service down: its status and its log. Nobody can connect: what is listening, and the firewall. Denied even as root: look for an SELinux denial. It will not boot: stop the boot early and repair it. The rest of this lesson takes them in turn.
Logs
One journal, filtered by unit, priority, boot and time
journald collects every log; journalctl filters it by unit (-u), priority (-p), boot (-b) and time (--since).
Programs and the kernel send their messages to the journal. journald stores them, and rsyslog copies them into the familiar text files in var log. journalctl filters: one service in this boot, errors in the previous boot, the last hour as it happens, or only the kernel. Make the journal persistent so the previous boot is still there, and let logrotate keep the text files in check.
A service will not start
A service will not start. Where do you look?
systemctl status shows the failure and its last log lines; journalctl -u has the rest. Read, fix one thing, restart.
The most common call of all: the service will not start. Where do you look? Status first. It says failed, and under that it prints the last lines the service logged. Often that is the whole answer. If not, the journal has everything the service has said since boot. What you find is nearly always one of four things: a mistake in the settings file, a port another process already holds, a file it is not allowed to read, or a program that is not where the unit says it is. Each has a quick way to confirm it. Then fix one thing, restart, and read status again.
Disk pressure
df finds the full disk, du the folder, lsof the hidden space
df shows how full each file system is, du where the space went, and lsof +L1 deleted files that are still held open.
df says var is full. df dash i says inodes are fine, so it is space, not a million tiny files. Either one stops writes. du adds up the folders and finds less than five gigabytes. The rest is a deleted log file that a running program still holds open. The space only comes back when the program closes it.
Memory pressure
Read "available", not "free", and watch for the OOM killer
Linux fills spare memory with cache, so judge by available; when it runs out, swap fills and the OOM killer ends a process.
free dash h on a server in trouble. Programs use most of the memory and only a little cache is left. Free is always small on Linux, because idle memory is used as cache; available is the number that matters, and here it is low. Swap is nearly full and busy. When nothing is left, the kernel kills the process with the worst score, and logs it.
Memory pressure · Going deeper · 1 of 2
sysctl changes the kernel now; a file in sysctl.d keeps it
sysctl reads and writes kernel parameters under /proc/sys; files in /etc/sysctl.d apply them at every boot.
The kernel’s settings appear as files under proc sys, and sysctl reads them by name. sysctl dash w changes one straight away, until the next reboot. A file in etc sysctl dot d makes it permanent, and sysctl dash dash system loads every file now. A few you will meet: swappiness, IP forwarding, the open-file limit and the highest process number.
Kernel modules · Going deeper · 2 of 2
Drivers load into the running kernel, by hand or at boot
lsmod lists loaded modules, modprobe loads and removes them, and modules-load.d and modprobe.d configure boot.
SELinux
Every file and process has a label, and labels decide access
SELinux labels processes and files with types; the policy allows httpd_t to read httpd_sys_content_t, and restorecon fixes wrong labels.
nginx runs with an SELinux type, httpd t. The policy lets it read files labelled as web content. A file moved from a home directory keeps its home label, and nginx is denied, even though the file permissions allow it. ls dash capital Z shows the labels, and restorecon resets them to what the policy expects for that folder. Moving keeps labels; copying makes new ones.
SELinux toolbox · Going deeper · 1 of 2
Read the denial, then fix the label, boolean or port
An AVC denial names process, target and action; fixes are restorecon, semanage fcontext, setsebool -P and semanage port.
AppArmor · Going deeper · 2 of 2
Ubuntu’s alternative: a list of allowed paths per program
AppArmor confines programs to the paths in their profile, in enforce or complain mode.
Rescue and emergency
Stop the boot early enough to repair what breaks it
From GRUB you can boot into rescue.target, emergency.target or stop in the initramfs with rd.break to repair a system.
The boot passes several stages, and you can stop at most of them. At the GRUB menu, press e to edit the entry for one boot. rd break stops in the initramfs, before the real root is mounted. The emergency target mounts only the root file system, read only. The rescue target adds local disks and basic services. Normally the boot carries on to the full system. Use emergency for a broken fstab, rescue for a service that breaks the boot, and rd break when the root password is lost. Add the target to the kernel line, then press Control X.
Rescue and emergency · Going deeper
Seven steps to reset a lost root password on RHEL
Boot with rd.break, remount /sysroot read-write, chroot, passwd, touch /.autorelabel, and exit twice.
Reboot and, at the menu, press e. Add rd break to the kernel line and boot. The real root file system is mounted read only at sysroot: remount it read-write. chroot into it, so the commands act on the real system. Set the new password. Then the step people forget: touch dot autorelabel, so SELinux relabels the system at the next boot. Exit twice; the system relabels and reboots, and the new password works.
Lab
Break it, then fix it from the symptom
A persistent journal, a hidden full disk, memory pressure, sysctl and modules, SELinux labels and ports, an emergency boot and a root password reset.
This lab is the course in miniature. You break a server in each of the ways on the triage map, then diagnose each fault from its symptom, fix it, and prove the fix survives a reboot. Press the down arrow for the steps.
Lab · Going deeper
Break it, then fix it from the symptom
Use a RHEL-family VM (AlmaLinux or Rocky) with nginx installed and a console you can reach. Take a snapshot first. For each fault, start from the triage map, not from what you know you broke.
Use a RHEL-family VM (AlmaLinux or Rocky) with nginx installed and a console you can reach. Take a snapshot first. For each fault, start from the triage map, not from what you know you broke. Steps. Logs. Make the journal persistent (mkdir -p /var/log/journal; systemctl restart systemd-journald). Reboot, then journalctl -b -1 -p warning. Disk. fallocate a 1 GiB file, hold it open with tail -f in the background, delete it. Compare df -h and du -sh; find it with lsof +L1; free the space by stopping tail. Memory. Run stress-ng --vm 1 --vm-bytes 90% --timeout 60s (from EPEL) and watch free -h and vmstat 1. Read any OOM lines with journalctl -k. Kernel settings. Read vm.swappiness, set it to 10 for good in /etc/sysctl.d, sysctl --system, reboot and check. Load a module with modprobe and list it with lsmod. SELinux label. Create a page in your home, mv it into /var/www/html, curl it (403). Find the AVC with ausearch, fix it with restorecon, and explain why cp would not have broken it. SELinux port and boolean. Make nginx listen on 8081 and restart it (it fails); fix it with semanage port. Check getsebool httpd_can_network_connect. Emergency boot. Add a bad line to /etc/fstab (a UUID that does not exist, no nofail). Reboot, land in emergency mode, fix fstab, systemctl daemon-reload, reboot. Lost root password. Reset it with rd.break, following the seven steps exactly, including touch /.autorelabel. Write it down. For each fault, write one line: symptom, first command, what it showed, fix. That is your own triage map. Check: Every fault ends with the system healthy after a reboot: df and du agree, getenforce says Enforcing, nginx serves on 80 and 8081, and you can log in as root with the new password.
Steps
- Logs. Make the journal persistent (mkdir -p /var/log/journal; systemctl restart systemd-journald). Reboot, then journalctl -b -1 -p warning.
- Disk. fallocate a 1 GiB file, hold it open with tail -f in the background, delete it. Compare df -h and du -sh; find it with lsof +L1; free the space by stopping tail.
- Memory. Run stress-ng --vm 1 --vm-bytes 90% --timeout 60s (from EPEL) and watch free -h and vmstat 1. Read any OOM lines with journalctl -k.
- Kernel settings. Read vm.swappiness, set it to 10 for good in /etc/sysctl.d, sysctl --system, reboot and check. Load a module with modprobe and list it with lsmod.
- SELinux label. Create a page in your home, mv it into /var/www/html, curl it (403). Find the AVC with ausearch, fix it with restorecon, and explain why cp would not have broken it.
- SELinux port and boolean. Make nginx listen on 8081 and restart it (it fails); fix it with semanage port. Check getsebool httpd_can_network_connect.
- Emergency boot. Add a bad line to /etc/fstab (a UUID that does not exist, no nofail). Reboot, land in emergency mode, fix fstab, systemctl daemon-reload, reboot.
- Lost root password. Reset it with rd.break, following the seven steps exactly, including touch /.autorelabel.
- Write it down. For each fault, write one line: symptom, first command, what it showed, fix. That is your own triage map.
Lesson complete
You finished “Troubleshooting a server”
Nicely done.