Triage

Seven symptoms, the first command for each, and what it shows

SymptomFirst commandWhat the output tells youEverything is slowuptime; topload above the CPU count; wa high = waiting on diskPrograms get killedfree -h; journalctl -k | grep -i oomavailable near 0; "Out of memory: Killed process"Writes faildf -h; df -i100% of space, or 100% of inodes (IUse%)A service is downsystemctl status app; journalctl -u app -bits exit code and its last log linesNobody can connectss -tlnp; firewall-cmd --list-allnot listening, listening on 127.0.0.1, or blockedDenied, even as rootausearch -m avc -ts recentan SELinux denial: wrong label, boolean or portIt will not bootGRUB menu: rescue, emergency, rd.breakusually fstab, a full disk, or a lost passwordStart from the symptom: each one has a first command, and its output points to the cause.Most incidents are one of these seven. Run the first command before you change anything, and write down what it said: it is yourevidence, and the fix is often obvious from it. The rest of this lesson takes each row in turn, and the lab makes you break each one.

A triage map: from each symptom, one first command, and what its output says about the cause.

Logs

One journal, filtered by unit, priority, boot and time

programsand the kernelsystemd-journaldindexed, with metadata/var/log/journalkept across reboots (persistent)rsyslog→ /var/log/messages, secure (RHEL)→ /var/log/syslog, auth.log (Ubuntu)$ journalctl -u nginx -b$ journalctl -p err -b -1$ journalctl --since "1 hour ago" -f$ journalctl -kone unit, this booterrors and worse, previous bootthe last hour, then followkernel messages onlyMake the journal persistent: mkdir -p /var/log/journal (or Storage=persistent in journald.conf), then systemctl restart systemd-journald.logrotate renames and compresses the text logs on a schedule (/etc/logrotate.d/), so they cannot fill the disk.Everything logs to the journal; filter it by unit, priority, boot and time instead of scrolling.journalctl -b -1 needs a persistent journal: without /var/log/journal the history is lost at every reboot, often the one you need.Text files in /var/log still matter for tools and habits: rsyslog writes them from the same messages.

journald collects every log; journalctl filters it by unit (-u), priority (-p), boot (-b) and time (--since).

A service will not start

A service will not start. Where do you look?

1 systemctl status appfailed, and the last lines it logged2 journalctl -u app -b -eeverything it said since boot3 fix, restart, statusone change, then look againWhat the log usually saysHow to confirm itThe settings file has a mistakerun the service’s own check: nginx -t, sshd -tThe port is already takenss -tlnp shows which process holds itIt may not read a file or directoryls -l as the service’s user; on Red Hat systems, SELinuxThe program is missing or not executablestatus=203/EXEC: check the path in ExecStartA service that will not start has already said why. Status shows the last lines; the journal has the rest.Read before you restart: starting it again without a change gives the same failure. Fix one thing, restart, and readstatus again. A service that failed too often in a row is held back by systemd; systemctl reset-failed app clears that.The exit status is in the status output too: 203/EXEC means the program named in ExecStart could not be run.

systemctl status shows the failure and its last log lines; journalctl -u has the rest. Read, fix one thing, restart.

Disk pressure

df finds the full disk, du the folder, lsof the hidden space

example$ df -h /varFilesystem Size Used Avail Use% Mounted on/dev/mapper/rhel-var 20G 20G 0 100% /var$ df -i /varFilesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/rhel-var 10M 41K 10M 1% /varSpace or inodes?Full space: big files. Full inodes:millions of tiny files (a mail queue,session files). Either stops writes.# du -xh --max-depth=1 /var | sort -h | tail -31.2G /var/cache3.1G /var/log4.6G /vardu finds 4.6G, but df says 20G is used:the rest belongs to deleted files still held open# lsof +L1java 2314 app 3w REG 253,2 16106127360 0 /var/log/app/app.log (deleted)Someone deleted a 15 GiB log that java still writes to. The space comes back when java closes it: restart the service.df says the disk is full; du says where; lsof +L1 finds the space that du cannot see.A deleted file is only removed once nothing has it open. Truncate a live log instead of deleting it (: > app.log), and let logrotatemanage logs. -x keeps du on one file system. Example numbers; the lsof format is real.

df shows how full each file system is, du where the space went, and lsof +L1 deleted files that are still held open.

Memory pressure

Read "available", not "free", and watch for the OOM killer

example: a server in trouble$ free -h total used free shared buff/cache availableMem: 15Gi 12Gi 205Mi 96Mi 2.3Gi 1.1Gi1Swap: 4.0Gi 3.6Gi 0.4Gi2used by programscacheavailable 1.1 Gi:what could still be given out1Ignore "free". Read "available": free memory plus the cache the kernel can drop. Low available is the real shortage.2Swap almost full, and vmstat 1 shows si and so busy every second: the machine is thrashing.kernel: Out of memory: Killed process 2314 (java) total-vm:14092340kB, anon-rss:11823436kBWhen nothing is left, the OOM killer ends the process with the worst score, usually the biggest. journalctl -k | grep -i oomJudge memory by "available", not "free"; when it nears zero, swapping starts and then the OOM killer.Linux uses idle memory as file cache, so "free" is always small on a healthy server. Find the cause with top sorted by memory (M) orps aux --sort=-rss | head. The fix is the program, its limits (MemoryMax= in its unit), or more RAM. Example numbers.

Linux fills spare memory with cache, so judge by available; when it runs out, swap fills and the OOM killer ends a process.

Memory pressure · Going deeper · 1 of 2

sysctl changes the kernel now; a file in sysctl.d keeps it

Kernel settings live as files under /proc/sys$ sysctl vm.swappinessvm.swappiness = 60$ cat /proc/sys/vm/swappiness60now, until reboot# sysctl -w vm.swappiness=10the same as writing 10 into/proc/sys/vm/swappinessevery boot/etc/sysctl.d/90-local.confvm.swappiness = 10# sysctl --systemloads all the files nowSettingWhat it changesvm.swappinesshow readily idle pages go to swap (0-200)net.ipv4.ip_forward1 makes the host route packetsfs.file-maxopen files allowed system-widekernel.pid_maxthe highest process IDsysctl reads and sets kernel parameters; a file in /etc/sysctl.d makes the setting survive a reboot.sysctl -a lists them all, about a thousand. Per-process limits, such as open files for one user or service, are ulimits instead: ulimit -n,/etc/security/limits.conf, or LimitNOFILE= in a systemd unit. Kernel modules (drivers) have their own tools: see Go deeper.

sysctl reads and writes kernel parameters under /proc/sys; files in /etc/sysctl.d apply them at every boot.

Kernel modules · Going deeper · 2 of 2

Drivers load into the running kernel, by hand or at boot

example$ lsmod | head -4Module Size Used byxfs 2621440 2virtio_net 77824 0nft_ct 28672 4$ modinfo -F description virtio_netVirtio network driverLoad and remove# modprobe br_netfilter# modprobe -r br_netfiltermodprobe also loads dependenciesAt every boot/etc/modules-load.d/k8s.confbr_netfilterNever load it/etc/modprobe.d/no-usb.confblacklist usb_storageOptions for a moduleoptions kvm_intel nested=1 in /etc/modprobe.d/kvm.confuname -r gives the running kernel; modules live in /lib/modules/$(uname -r)/.Drivers and features load into the running kernel as modules, by hand or from a list at boot.lsmod shows what is loaded and what uses it; modinfo describes a module and its options. A file in /etc/modules-load.d loads one atboot; blacklist in /etc/modprobe.d stops automatic loading (install name /bin/false stops it completely). Example sizes.

lsmod lists loaded modules, modprobe loads and removes them, and modules-load.d and modprobe.d configure boot.

SELinux

Every file and process has a label, and labels decide access

nginxruns as httpd_t/var/www/html/index.htmlhttpd_sys_content_tallowed: policy lets httpd_t read it/var/www/html/report.htmluser_home_t (moved from a home)denied, even though the mode is 644$ ls -Z /var/www/htmlunconfined_u:object_r:httpd_sys_content_t:s0 index.htmlunconfined_u:object_r:user_home_t:s0 report.html# restorecon -Rv /var/www/html1Relabeled /var/www/html/report.html from ...user_home_t:s0 to ...mv keeps a file’s label; cp makes anew file with the folder’s label.restorecon resets labels to whatthe policy says for that path.SELinux labels every process and file; the policy says which process types may touch which file types.File permissions are checked first, then SELinux, so even root can be denied. The usual cause is a wrong label, not a wrong policy:fix the label (restorecon), never switch SELinux off. getenforce shows the mode: Enforcing, Permissive (logs only) or Disabled.

SELinux labels processes and files with types; the policy allows httpd_t to read httpd_sys_content_t, and restorecon fixes wrong labels.

SELinux toolbox · Going deeper · 1 of 2

Read the denial, then fix the label, boolean or port

# ausearch -m avc -ts recenttype=AVC msg=audit(...): avc: denied { read } for pid=1204 comm="nginx" name="report.html" scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0who (scontext), what (tcontext), which action ({ read }): sealert -a /var/log/audit/audit.log explains it in wordsThe causeThe fixa file in a standard place, wrong labelrestorecon -Rv /pathcontent in a new place, /srv/websemanage fcontext -a -t httpd_sys_content_t '/srv/web(/.*)?', then restorecona feature switched offgetsebool -a | grep httpd; setsebool -P httpd_can_network_connect ona service on a new portsemanage port -a -t http_port_t -p tcp 8081; semanage port -l | grep httpModes: setenforce 0 is Permissive until reboot, for testing only; SELINUX=enforcing in /etc/selinux/config is the boot mode.An AVC denial names the process, the file or port and the action; the fix is a label, a boolean or a port type.semanage fcontext changes the rule for a path, so the label survives relabelling; chcon changes one file only and is undone by restorecon.-P makes a boolean permanent. These tools come from policycoreutils-python-utils and setroubleshoot-server.

An AVC denial names process, target and action; fixes are restorecon, semanage fcontext, setsebool -P and semanage port.

AppArmor · Going deeper · 2 of 2

Ubuntu’s alternative: a list of allowed paths per program

example$ sudo aa-status | head -6 (Ubuntu)apparmor module is loaded.118 profiles are loaded.29 profiles are in enforce mode. /usr/bin/man /usr/sbin/tcpdumpA profile is a list of paths/etc/apparmor.d/usr.sbin.tcpdump /etc/protocols r, /dev/bpf* rw, @{HOME}/** r,Anything not listed is denied.CommandEffectaa-complain /usr/sbin/tcpdumplog violations, allow themaa-enforce /usr/sbin/tcpdumpdeny violations againjournalctl -k | grep apparmorDENIED lines name the path and the programAppArmor, on Ubuntu and SUSE, confines a program to the paths its profile lists.SELinux labels files; AppArmor names paths. Both are mandatory access control: they apply on top of permissions, even to root.Profiles are per program, so unprofiled programs run unconfined. Complain mode is AppArmor’s equivalent of SELinux permissive.

AppArmor confines programs to the paths in their profile, in enforce or complain mode.

Rescue and emergency

Stop the boot early enough to repair what breaks it

firmware, GRUBpress e to editinitramfsrd.break stops hereroot mountedemergency.targetlocal disks, basicrescue.targetfull systemmulti-user.targetEntry pointYou getUse it whenrd.breaka shell before the real root is mountedyou have no root password (RHEL)emergencyroot file system read-only, nothing elsefstab or a disk stops the bootrescuelocal file systems, no network, one usera service breaks the bootAdd systemd.unit=emergency.target (or rescue.target, or rd.break) to the line starting linux, then Ctrl+X to boot once.You can stop the boot at several points to repair what stops it from finishing.Rescue and emergency both ask for the root password; rd.break does not, which is why physical or console access is itself asecurity boundary. Booting is changed once only: editing in GRUB is never saved. On a VM or cloud host, use its console.

From GRUB you can boot into rescue.target, emergency.target or stop in the initramfs with rd.break to repair a system.

Rescue and emergency · Going deeper

Seven steps to reset a lost root password on RHEL

1Reboot; at the GRUB menu press e2On the line starting linux, add rd.break; press Ctrl+X3switch_root:/# mount -o remount,rw /sysrootthe real root is mounted read-only at /sysroot4switch_root:/# chroot /sysrootnow / is the real system5sh-5.2# passwd rootset the new password6sh-5.2# touch /.autorelabelSELinux relabels at next boot: passwd made /etc/shadow unlabelled7exit, exitthe system relabels (a few minutes), then rebootsWithout the root password, stop the boot in the initramfs, remount the real root and set a new one.Step 6 is the one people forget: without it SELinux refuses logins, because the new /etc/shadow has no proper label. This is theRHEL procedure (and RHCSA task); on Ubuntu, choose Advanced options, recovery mode, root shell, then mount -o remount,rw / and passwd.

Boot with rd.break, remount /sysroot read-write, chroot, passwd, touch /.autorelabel, and exit twice.

Lab

Break it, then fix it from the symptom

# fallocate -l 1G /var/tmp/big; tail -f /var/tmp/big &1# rm /var/tmp/big; lsof +L1 | grep bigtail 5120 root 3r REG 253,0 1073741824 0 /var/tmp/big (deleted)# mv ~/report.html /var/www/html/; curl -I localhost/report.html2HTTP/1.1 403 Forbidden# ausearch -m avc -ts recent | grep -c denied1# restorecon -Rv /var/www/html; curl -sI localhost/report.html3HTTP/1.1 200 OK# echo vm.swappiness=10 > /etc/sysctl.d/90-local.conf4(reboot, e, rd.break ... touch /.autorelabel)51A hidden full diskfound with lsof +L12An SELinux denialmode 644 and still 4033Fixed by its labelnot by setenforce 04A tuned kernelthat survives reboot5A lost root passwordreset from the bootLab: break a server five ways, then diagnose each from the symptom with the triage map.Use a RHEL-family VM with a console you can reach. Snapshot it first. Press the down arrow for the steps.

A persistent journal, a hidden full disk, memory pressure, sysctl and modules, SELinux labels and ports, an emergency boot and a root password reset.

Lab · Going deeper

Break it, then fix it from the symptom

Use a RHEL-family VM (AlmaLinux or Rocky) with nginx installed and a console you can reach. Take a snapshot first. For each fault, start from the triage map, not from what you know you broke.

Steps

  1. Logs. Make the journal persistent (mkdir -p /var/log/journal; systemctl restart systemd-journald). Reboot, then journalctl -b -1 -p warning.
  2. Disk. fallocate a 1 GiB file, hold it open with tail -f in the background, delete it. Compare df -h and du -sh; find it with lsof +L1; free the space by stopping tail.
  3. Memory. Run stress-ng --vm 1 --vm-bytes 90% --timeout 60s (from EPEL) and watch free -h and vmstat 1. Read any OOM lines with journalctl -k.
  4. Kernel settings. Read vm.swappiness, set it to 10 for good in /etc/sysctl.d, sysctl --system, reboot and check. Load a module with modprobe and list it with lsmod.
  5. SELinux label. Create a page in your home, mv it into /var/www/html, curl it (403). Find the AVC with ausearch, fix it with restorecon, and explain why cp would not have broken it.
  6. SELinux port and boolean. Make nginx listen on 8081 and restart it (it fails); fix it with semanage port. Check getsebool httpd_can_network_connect.
  7. Emergency boot. Add a bad line to /etc/fstab (a UUID that does not exist, no nofail). Reboot, land in emergency mode, fix fstab, systemctl daemon-reload, reboot.
  8. Lost root password. Reset it with rd.break, following the seven steps exactly, including touch /.autorelabel.
  9. Write it down. For each fault, write one line: symptom, first command, what it showed, fix. That is your own triage map.
Check: Every fault ends with the system healthy after a reboot: df and du agree, getenforce says Enforcing, nginx serves on 80 and 8081, and you can log in as root with the new password.

Lesson complete

You finished “Troubleshooting a server”

Nicely done.

Open in a new tab