Programs and processes

What exactly is running when a program runs?

$ sleep 300 &1[1] 5338$ ps2 PID TTY TIME CMD 5337 pts/0 00:00:00 bash 5338 pts/0 00:00:00 sleep 5339 pts/0 00:00:00 ps$ kill 53383[1]+ Terminated sleep 3001Start a program& leaves it running behind the prompt2ps lists this terminal’s processesthe shell, sleep, and ps itself3kill asks a process to endnamed by its PIDA program is a file on disk. A process is one running copy of it, with its own number: the PID./usr/bin/sleep is one file. Start it three times and there are three processes, each with its own PID, its own memoryand an owner, the user who started it. Everything you do to running software, inspect it, pause it, give it less CPU,end it, you do to a process, by its PID. The next page widens the view from this terminal to the whole machine.

A process is a running copy of a program, identified by its PID; ps lists processes and kill sends one a request to end.

top and ps

The header is the machine; each row is a process

top - 10:03:54 up 17 days, 2 users, load average: 3.92, 2.10, 1.05load: 1, 5 and 15 minutesTasks: 182 total, 4 running, 178 sleeping, 0 stopped, 0 zombie%Cpu(s): 71.2 us, 6.4 sy, 0.0 ni, 19.8 id, 2.1 wa, 0.0 hi, 0.5 si, 0.0 styour programsthe kernelidlewaiting on diskstolen by the hypervisorMiB Mem : 15987.4 total, 1204.6 free, 9843.2 used, 4939.6 buff/cacheMiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 5812.3 avail Memavail is what matters PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4120 postgres 20 0 4318.2m 2.1g 1.9g R 187.3 13.4 92:14.08 postgresRES is real memory in use; %CPU over 100 means more than one core; P sorts by CPU, M by memory, k kills, q quitstop refreshes every 3 seconds: the header is the whole machine, the table one line per process.Read it top down: is load high for the number of CPUs; is the CPU busy with programs (us), the kernel (sy) or waiting on disk (wa);is available memory low and swap in use. Then the table says which process. ps aux gives the same columns once, for scripts:ps aux --sort=-%mem | head finds the biggest memory users.

top shows load, CPU split by kind, memory and swap, then processes sorted by use; ps aux prints the same once.

Processes

Every process has a parent, back to PID 1

systemdPID 1sshdPID 812nginxmaster, PID 901cronPID 640sshd: anaPID 2131bashPID 2140vim notes.txtPID 2201worker, PID 902worker, PID 903worker, PID 904ps -ef --forestpstree -pshow this treeEvery process has a parent, and every chain leads back to PID 1, which systemd runs.A process is a running program with its own ID, memory, open files and the user it runs as. Your shell is a child of the SSHsession that started it; the commands you type are children of the shell. If a parent dies first, its children are adoptedby PID 1. Kill a parent like sshd: ana and everything below it goes with the session.

A process is a running program with a PID, a parent and an owner; every chain leads to systemd, PID 1.

Processes · Going deeper · 1 of 2

A copy of the parent, which then becomes the new program

bashPID 2140fork()bash (a copy)PID 2250execve("ls")lsstill PID 2250exit(0)zombieuntil collectedwait(): bash collects the exit status into $?, and the zombie is goneA builtin such as cd runs inside bash itself, with no fork: a child could not change its parent’s directory.exec ls replaces the shell itself: when ls ends, the session ends.A new program starts as a copy of its parent, then the copy replaces itself with the program.fork makes a near-free copy, sharing memory until either side writes. execve loads the new program into that process, keepingits PID, open files and environment, which is how redirections set up by the shell reach the program. When it exits, its statuswaits as a zombie, Z in ps, until the parent reads it. Many zombies point to a parent that never calls wait.

The shell forks a copy of itself, the copy execs the program, and the parent waits for its exit status.

Processes · Going deeper · 2 of 2

Running, sleeping, stuck on disk, stopped, or finished

R runningon a CPU, or ready for oneS sleepingwaiting for input, a timer, a lockD uninterruptiblewaiting on disk or NFS: unkillableT stoppedby Ctrl-Z or SIGSTOP; CONT resumesZ zombiefinished, status not yet collectedexitSTAT Ss session leader, sleeping R+ running, foreground Sl sleeping, multi-threadedMost processes are asleep most of the time; the letter in ps STAT says what each one is waiting for.R means wanting the CPU, S waiting for something to happen. D is the one to notice: the process is inside the kernel waiting onstorage, ignores every signal including KILL, and counts toward load. Many processes in D usually mean a slow or hung disk orNFS server, not a CPU problem. A zombie uses no memory, only a slot in the process table.

ps STAT shows R running, S sleeping, D waiting on I/O and unkillable, T stopped and Z zombie.

Signals

Ask a process to stop before you force it

1SIGHUPreload configuration (daemons)2SIGINTinterrupt: Ctrl-C9SIGKILLend now: cannot be caught15SIGTERMplease end: the default of kill18SIGCONTcarry on after a stop19SIGSTOPpause: cannot be caught20SIGTSTPpause request: Ctrl-Zkill 4120 (TERM)process 4120its handler runssaves, closes, exitscleankill -9 4120 (KILL)the kernelremoves it at oncehalf-written files,stale locksA signal is a numbered message to a process. It can handle most of them, but never KILL or STOP.kill sends TERM unless told otherwise, and a well-written program cleans up and exits. Use kill -9 only when TERM has beenignored for a while: the kernel ends the process without asking, and anything it had not saved is lost. Many daemons reloadtheir configuration on HUP. pkill nginx and killall nginx find processes by name; pgrep -a shows what would be matched first.

SIGTERM lets a process clean up and exit; SIGKILL is carried out by the kernel and cannot be caught.

Job control

One job has the terminal; the others wait or run behind

$ tar -czf big.tar.gz /srv/data1^Z[1]+ Stopped tar -czf big.tar.gz /srv/data$ bg2[1]+ tar -czf big.tar.gz /srv/data &$ jobs3[1]+ Running tar -czf big.tar.gz /srv/data &$ fg %14tar -czf big.tar.gz /srv/data1Ctrl-Z stops itstate T: paused, not ended2bg resumes it in the backgroundthe prompt is yours again3jobs lists this shell’s jobsnumbers, not PIDs4fg brings one backCtrl-C now reaches itStart in the background at once with &. To survive logging out: nohup cmd &, disown, or better, tmux or a systemd unit.A job is a command your shell started. Only one has the terminal; the rest wait or run behind.When you log out, the terminal hangs up and the shell sends SIGHUP to its jobs, which ends them. That is why a long copy startedover SSH dies with the connection. tmux keeps a session alive on the server so you can reattach; anything that must run forgood belongs in a systemd service, the subject of the next lesson.

Ctrl-Z stops a job, bg resumes it in the background, fg brings it back; jobs end when the session hangs up.

Load average

Read the load against the number of CPUs

load average: 3.92, 2.10, 1.05 on a machine with 4 CPUs1 min: 3.925 min: 2.1015 min: 1.054.0 = every CPU busy123rising:the last minute isbusier than the lastquarter of an hourLoad is the average number of processes running or waiting to run, read against the number of CPUs.Below the CPU count, nothing waits. Above it, processes queue. On Linux the count also includes processes in state D, waiting ondisk, so a high load with idle CPUs points at storage, not computation. Compare the three numbers to see the trend: here theload is climbing, but still under the 4 CPUs that nproc reports. Bars are to scale.

The load average counts processes running or waiting, over 1, 5 and 15 minutes; compare it with the CPU count.

Load average · Going deeper · 1 of 2

nice decides who wins only when the CPUs are contended

Two busy processes sharing one CPU: the share each getsnice 0 and nice 050% and 50%nice 0 and nice 575% and 25%nice 0 and nice 1090% and 10%nice 0 and nice 1999% and 1%nice -5 and nice 075% and 25%$ nice -n 10 ./backup.sh # start lower# renice -n -5 -p 4120 # raise: root onlynice sets a weight from -20 to 19: each step is about 10% less CPU when processes compete.It changes nothing on an idle machine: a niced job still gets every spare cycle. Only root may lower the number. Computed from thekernel’s scheduler weights (nice 0 = 1024, 10 = 110, 19 = 15); bars to scale.

nice values run from -20 to 19; each step gives about 10% less CPU under contention. renice changes a running process.

Tuning profiles · Going deeper · 2 of 2

One command applies a whole set of performance settings

# tuned-adm listAvailable profiles:- balanced - General non-specialized tuned profile- latency-performance - Optimize for deterministic performance- powersave - Optimize for low power consumption- throughput-performance - Broadly applicable tuning ...- virtual-guest - Optimize for running inside a virtual guestCurrent active profile: virtual-guest# tuned-adm recommendvirtual-guest# tuned-adm profile throughput-performance1Profiles are bundlesof sysctl, CPU and disk settings2recommendpicks one for this hardware3profile NAMEapplies it now and at boot4tuned-adm activeshows what is in forcetuned applies a named bundle of kernel settings for a kind of workload, and keeps it after reboot.It is installed and running on RHEL by default, and the Red Hat exam asks you to set the recommended profile. On Ubuntu installthe tuned package. The profiles live in /usr/lib/tuned/; copy one into /etc/tuned/ to customise it.

tuned profiles bundle kernel, CPU and disk settings; tuned-adm recommend and profile set them persistently.

Lab

Start, inspect, reprioritise and end processes

$ sleep 600 & pgrep -a sleep1[1] 44104410 sleep 600$ ps -o pid,ppid,stat,ni,cmd -p 44102 PID PPID STAT NI CMD 4410 2140 S 0 sleep 600$ renice -n 10 -p 441034410 (process ID) old priority 0, new priority 10$ kill 4410; sleep 1; jobs4[1]+ Terminated sleep 600$ uptime5 10:31:02 up 17 days, 2 users, load average: 0.08, 0.05, 0.011Start and find itpgrep -a shows the command2Its parent is your shellstate S: sleeping3Lower its priorityany user may raise nice4TERM ends it cleanlythe shell reports it5Load, against nprochere, an idle machineLab: start, inspect, reprioritise and end processes, then read a busy machine.Press the down arrow for the steps.

pgrep, ps, renice, kill and jobs on processes you start yourself, then real load and real CPU shares in top.

Lab · Going deeper

Watch, steer and stop processes

Any Linux machine will do. A second terminal makes the load steps easier to watch.

Steps

  1. See the tree. ps -ef --forest | less, or pstree -p $$. Find your shell and trace its parents back to PID 1.
  2. Fork and exec. Run echo $$; bash; echo $$; exit. Then exec bash and explain why the PID did not change.
  3. Jobs. sleep 300, then Ctrl-Z, bg, jobs, fg, Ctrl-C. Start sleep 300 & and end it with kill %1.
  4. Signals. Run trap 'echo got TERM' TERM; kill $$ in a subshell: bash -c "trap 'echo got TERM' TERM; kill \$\$; echo still here". Then try the same with -9.
  5. Make load. Start one busy loop per CPU: for i in $(seq $(nproc)); do (while :; do :; done) & done. Watch top for a minute, then kill %1 %2 ...
  6. Share a CPU. taskset -c 0 sh -c "while :; do :; done" & taskset -c 0 nice -n 10 sh -c "while :; do :; done" & and compare %CPU in top. Kill both.
  7. Find the hungry ones. ps aux --sort=-%cpu | head -5 and ps aux --sort=-%mem | head -5.
  8. Tune the machine. On RHEL or AlmaLinux: tuned-adm active; tuned-adm recommend; set the recommended profile and check it after a reboot.
Check: The niced loop gets about a tenth of the CPU the other one gets, and every process you started is gone at the end (pgrep -a sleep is empty).

Lesson complete

You finished “Processes”

Nicely done.

Open in a new tab