Programs and processes
What exactly is running when a program runs?
A process is a running copy of a program, identified by its PID; ps lists processes and kill sends one a request to end.
A program is only a file until something runs it. What is it then? Start sleep, a program that just waits, and put an ampersand after it so the prompt comes straight back. The shell prints a number. ps lists the processes of this terminal: your shell, the sleep, and ps itself. Each has a P I D, a process I D, and that number is the handle for everything that follows. kill with the P I D asks that one process to end.
top and ps
The header is the machine; each row is a process
top shows load, CPU split by kind, memory and swap, then processes sorted by use; ps aux prints the same once.
top is the first command on a slow server. The first line gives the load averages. Then how many processes exist, and in which states. Then where the CPU time goes: your programs, the kernel, idle, waiting on disk, or stolen by the hypervisor on a cloud machine. Then memory, where available is the number that matters, and swap. Below, one row per process. Press capital P to sort by CPU and capital M by memory.
Processes
Every process has a parent, back to PID 1
A process is a running program with a PID, a parent and an owner; every chain leads to systemd, PID 1.
A process is a program that is running. The kernel starts exactly one itself: process one, which on modern Linux is systemd. Everything else descends from it: the SSH server, the web server, cron. When you log in, SSH starts a session, the session starts your shell, and the shell starts each command you type. Some programs start their own children, like these web server workers. ps and pstree draw the family tree for you.
Processes · Going deeper · 1 of 2
A copy of the parent, which then becomes the new program
The shell forks a copy of itself, the copy execs the program, and the parent waits for its exit status.
How does the shell start ls? It calls fork, which makes a copy of the shell, with a new process ID. The copy calls exec, which swaps in the ls program but keeps the ID, the open files and the environment. ls runs and exits. Its exit status waits as a zombie, until the shell collects it with wait. That number is what you read as dollar question mark. Builtins like cd skip all this and run inside the shell itself.
Processes · Going deeper · 2 of 2
Running, sleeping, stuck on disk, stopped, or finished
ps STAT shows R running, S sleeping, D waiting on I/O and unkillable, T stopped and Z zombie.
A process is always in one of a few states. R means it is running, or ready to run. S means asleep, waiting for something, which is where most processes spend most of their time. D is special: the process is inside the kernel, waiting on a disk or a network file system, and it cannot be interrupted, not even by kill. T is stopped, paused by Control Z. And Z is a zombie: finished, waiting for its parent to read its exit status. These letters appear in the STAT column of ps.
Signals
Ask a process to stop before you force it
SIGTERM lets a process clean up and exit; SIGKILL is carried out by the kernel and cannot be caught.
Processes are controlled with signals, which are numbered messages. The ones to know are hang-up, which many services treat as reload; interrupt, which is Control C; terminate, which is the polite default; and kill, number nine. Terminate goes to the process, which can save its work, close its files and exit cleanly. Kill never reaches the process at all. The kernel simply removes it, and whatever it was in the middle of is left broken. So send terminate first, wait, and only then use kill dash nine.
Job control
One job has the terminal; the others wait or run behind
Ctrl-Z stops a job, bg resumes it in the background, fg brings it back; jobs end when the session hangs up.
Your shell can juggle several commands, called jobs. Control Z pauses the one in front. bg lets it carry on in the background, and the prompt comes back. jobs lists them, and fg brings one to the front again. A trailing ampersand starts a command in the background straight away. But background jobs still belong to your session, and when you disconnect they are sent a hang-up and end. Anything that must keep running belongs in tmux, or better, a service.
Load average
Read the load against the number of CPUs
The load average counts processes running or waiting, over 1, 5 and 15 minutes; compare it with the CPU count.
The three load numbers are averages over one, five and fifteen minutes of how many processes were running or waiting to run. Here they are three point nine, two point one and one. On its own a number means nothing: compare it with the CPUs. This machine has four, so a load of four means every CPU is busy and nothing waits. Read the three together for the trend: here, load is rising. On Linux, processes waiting on disk count too.
Load average · Going deeper · 1 of 2
nice decides who wins only when the CPUs are contended
nice values run from -20 to 19; each step gives about 10% less CPU under contention. renice changes a running process.
When two busy processes share a CPU, the scheduler divides it by weight, and nice sets the weight. Equal nice values share equally. Give one a nice of five, and it gets a quarter. At ten, about a tenth. At nineteen, about one per cent. Going below zero, which only root may do, tips the balance the other way. Start long batch jobs with nice so interactive work stays fast, and remember that on an idle machine, nice changes nothing at all.
Tuning profiles · Going deeper · 2 of 2
One command applies a whole set of performance settings
tuned profiles bundle kernel, CPU and disk settings; tuned-adm recommend and profile set them persistently.
Lab
Start, inspect, reprioritise and end processes
pgrep, ps, renice, kill and jobs on processes you start yourself, then real load and real CPU shares in top.
In this lab you start processes and follow them through the tree, move jobs between the foreground and background, send signals, create real load, and watch nice divide a CPU. Press the down arrow for the steps.
Lab · Going deeper
Watch, steer and stop processes
Any Linux machine will do. A second terminal makes the load steps easier to watch.
Any Linux machine will do. A second terminal makes the load steps easier to watch. Steps. See the tree. ps -ef --forest | less, or pstree -p $$. Find your shell and trace its parents back to PID 1. Fork and exec. Run echo $$; bash; echo $$; exit. Then exec bash and explain why the PID did not change. Jobs. sleep 300, then Ctrl-Z, bg, jobs, fg, Ctrl-C. Start sleep 300 & and end it with kill %1. Signals. Run trap 'echo got TERM' TERM; kill $$ in a subshell: bash -c "trap 'echo got TERM' TERM; kill \$\$; echo still here". Then try the same with -9. Make load. Start one busy loop per CPU: for i in $(seq $(nproc)); do (while :; do :; done) & done. Watch top for a minute, then kill %1 %2 ... Share a CPU. taskset -c 0 sh -c "while :; do :; done" & taskset -c 0 nice -n 10 sh -c "while :; do :; done" & and compare %CPU in top. Kill both. Find the hungry ones. ps aux --sort=-%cpu | head -5 and ps aux --sort=-%mem | head -5. Tune the machine. On RHEL or AlmaLinux: tuned-adm active; tuned-adm recommend; set the recommended profile and check it after a reboot. Check: The niced loop gets about a tenth of the CPU the other one gets, and every process you started is gone at the end (pgrep -a sleep is empty).
Steps
- See the tree. ps -ef --forest | less, or pstree -p $$. Find your shell and trace its parents back to PID 1.
- Fork and exec. Run echo $$; bash; echo $$; exit. Then exec bash and explain why the PID did not change.
- Jobs. sleep 300, then Ctrl-Z, bg, jobs, fg, Ctrl-C. Start sleep 300 & and end it with kill %1.
- Signals. Run trap 'echo got TERM' TERM; kill $$ in a subshell: bash -c "trap 'echo got TERM' TERM; kill \$\$; echo still here". Then try the same with -9.
- Make load. Start one busy loop per CPU: for i in $(seq $(nproc)); do (while :; do :; done) & done. Watch top for a minute, then kill %1 %2 ...
- Share a CPU. taskset -c 0 sh -c "while :; do :; done" & taskset -c 0 nice -n 10 sh -c "while :; do :; done" & and compare %CPU in top. Kill both.
- Find the hungry ones. ps aux --sort=-%cpu | head -5 and ps aux --sort=-%mem | head -5.
- Tune the machine. On RHEL or AlmaLinux: tuned-adm active; tuned-adm recommend; set the recommended profile and check it after a reboot.
Lesson complete
You finished “Processes”
Nicely done.